# nLab relative entropy

Contents

### Context

#### Measure and probability theory

measure theory

probability theory

# Contents

## Idea

The notion of relative entropy of states is a generalization of the notion of entropy to a situation where the entropy of one state is measured “relative to” another state.

is also called

• Kullback-Leibler divergence

• information divergence

• information gain .

## Definition

### For states on finite probability spaces

For two finite probability distributions $(p_i)$ and $(q_i)$, their relative entropy is

$S(p/q) := \sum_{k = 1}^n p_k(log p_k - log q_k) \,.$

Alternatively, for $\rho, \phi$ two density matrices, their relative entropy is

$S(\rho/\phi) := tr \rho(log \rho - log \phi) \,.$

### For states on classical probability spaces

###### Definition

For $X$ a measurable space and $P$ and $Q$ two probability measures on $X$, such that $Q$ is absolutely continuous with respect to $P$, their relative entropy is the integral

$S(Q|P) = \int_X log \frac{d Q}{d P} d P \,,$

where $d Q / d P$ is the Radon-Nikodym derivative of $Q$ with respect to $P$.

### For states on quantum probability spaces (von Neumann algebras)

Let $A$ be a von Neumann algebra and let $\phi$, $\psi : A \to \mathbb{C}$ be two states on it (faithful, positive linear functionals).

###### Definition

The relative entropy $S(\phi/\psi)$ of $\psi$ relative to $\phi$ is

$S(\phi/\psi) := - (\Psi, (log \Delta_{\Phi,\Psi}) \Psi) \,,$

where $\Delta_{\Phi,\Psi}$ is the relative modular operator? of any cyclic and separating vector representatives $\Phi$ and $\Psi$ of $\phi$ and $\psi$.

This is due to (Araki).

###### Proposition
• This definition is independent of the choice of these representatives.

• In the case that $A$ is finite dimensional and $\rho_\phi$ and $\rho_\psi$ are density matrices of $\phi$ and $\psi$, respectively, this reduces to the above definition.

## Relation to machine learning

The machine learning process has been characterized as a minimization of relative entropy (Ackley, Hinton and Sejnowski 1985).

Relative entropy of states on von Neumann algebras was introduced in:

A characterization of relative entropy on finite-dimensional C-star algebras is given in

• D. Petz, Characterization of the relative entropy of states of matrix algebras, Acta Mathematica Hungarica, 59 (1992), 3-4. (doi:10.1007/bf00050907)

A survey of entropy in operator algebras is in

• Erling Størmer, Entropy in operator algebras (pdf)

A characterization of machine learning as a process minimizing relative entropy is proposed in

• {AckleyHintonSejnowski} David H. Ackley, Geoffrey E. Hilton, Terrence J. Sejnowski. A learning algorithm for Boltzmann machines, Cognitive Science, 9 (1985), 147–169. (web)