# nLab relative entropy

### Context

#### Measure and probability theory

measure theory

probability theory

# Contents

## Idea

The notion of relative entropy of states is a generalization of the notion of entropy to a situation where the entropy of one state is measured “relative to” another state.

is also called

• Kullback-Leibler divergence

• information divergence

• information gain .

## Definition

### For states on finite probability spaces

For two finite probability distributions $\left({p}_{i}\right)$ and $\left({q}_{i}\right)$, their relative entropy is

$S\left(p/q\right):=\sum _{k=1}^{n}{p}_{k}\left(\mathrm{log}{p}_{k}-\mathrm{log}{q}_{k}\right)\phantom{\rule{thinmathspace}{0ex}}.$S(p/q) := \sum_{k = 1}^n p_k(log p_k - log q_k) \,.

Alternatively, for $\rho ,\varphi$ two density matrices, their relative entropy is

$S\left(\rho /\varphi \right):=\mathrm{tr}\rho \left(\mathrm{log}\rho -\mathrm{log}\varphi \right)\phantom{\rule{thinmathspace}{0ex}}.$S(\rho/\phi) := tr \rho(log \rho - log \phi) \,.

### For states on classical probability spaces

###### Definition

For $X$ a measurable space and $P$ and $Q$ two probability measures on $X$, such that $Q$ is absolutely continuous with respect to $P$, their relative entropy is the integral

$S\left(Q\mid P\right)={\int }_{X}\mathrm{log}\frac{dQ}{dP}dP\phantom{\rule{thinmathspace}{0ex}},$S(Q|P) = \int_X log \frac{d Q}{d P} d P \,,

where $dQ/dP$ is the Radon-Nikodym derivative of $Q$ with respect to $P$.

### For states on quantum probability spaces (von Neumann algebras)

Let $A$ be a von Neumann algebra and let $\varphi$, $\psi :A\to ℂ$ be two states on it (faithful, positive linear functionals).

###### Definition

The relative entropy $S\left(\varphi /\psi \right)$ of $\psi$ relative to $\varphi$ is

$S\left(\varphi /\psi \right):=-\left(\Psi ,\left(\mathrm{log}{\Delta }_{\Phi ,\Psi }\right)\Psi \right)\phantom{\rule{thinmathspace}{0ex}},$S(\phi/\psi) := - (\Psi, (log \Delta_{\Phi,\Psi}) \Psi) \,,

where ${\Delta }_{\Phi ,\Psi }$ is the relative modular operator? of any cyclic and separating vector representatives $\Phi$ and $\Psi$ of $\varphi$ and $\psi$.

This is due to (Araki).

###### Proposition
• This definition is independent of the choice of these representatives.

• In the case that $A$ is finite dimensional and ${\rho }_{\varphi }$ and ${\rho }_{\psi }$ are density matrices of $\varphi$ and $\psi$, respectively, this reduces to the above definition.

## References

Relative entropy of states on von Neumann algebras was introduced in

A characterization of relative entropy on finite-dimensional C-star algebras is given in

• D. Petz, Characterization of the relative entropy of states of matrix algebras (pdf)

A survey of entropy in operator algebras is in

• Erling Størmer, Entropy in operator algebras (pdf)

Revised on July 15, 2011 22:12:00 by Urs Schreiber (81.156.13.230)