nLab
relative entropy

Idea

The notion of relative entropy of states is a generalization of the notion of entropy to a situation where the entropy of one state is measured “relative to” another state.

is also called

• Kullback-Leibler divergence

• information divergence

• information gain .

Definition

For states on finite probability spaces

For two finite probability distributions $(p_i)$ and $(q_i)$, their relative entropy is

$$S(p/q):=\sum _{k=1}^n{p}_k(\log p_k-\log q_k).$$S(p/q) := \sum_{k = 1}^n p_k(log p_k - log q_k) \,.

Alternatively, for $\rho ,\varphi$ two density matrices, their relative entropy is

$$S(\rho /\varphi ):=\mathrm{tr}\rho (\log \rho -\log \varphi ).$$S(\rho/\phi) := tr \rho(log \rho - log \phi) \,.

For states on classical probability spaces

Definition

For $X$ a measurable space and $P$ and $Q$ two probability measures on $X$, such that $Q$ is absolutely continuous with respect to $P$, their relative entropy is the integral

$$S(Q\mid P)={\int }_X\log \frac{dQ}{dP}dP,$$S(Q|P) = \int_X log \frac{d Q}{d P} d P \,,

where $dQ/dP$ is the Radon-Nikodym derivative of $Q$ with respect to $P$.

For states on quantum probability spaces (von Neumann algebras)

Let $A$ be a von Neumann algebra and let $\varphi$, $\psi :A\to ℂ$ be two states on it (faithful, positive linear functionals).

Definition

The relative entropy $S(\varphi /\psi )$ of $\psi$ relative to $\varphi$ is

$$S(\varphi /\psi ):=-(\Psi ,(\log {\Delta }_{\Phi ,\Psi })\Psi ),$$S(\phi/\psi) := - (\Psi, (log \Delta_{\Phi,\Psi}) \Psi) \,,

where ${\Delta }_{\Phi ,\Psi }$ is the relative modular operator? of any cyclic and separating vector representatives $\Phi$ and $\Psi$ of $\varphi$ and $\psi$.

This is due to (Araki).

Proposition

• This definition is independent of the choice of these representatives.

• In the case that $A$ is finite dimensional and ${\rho }_{\varphi }$ and ${\rho }_{\psi }$ are density matrices of $\varphi$ and $\psi$, respectively, this reduces to the above definition.

References

Relative entropy of states on von Neumann algebras was introduced in

• Huzihiro Araki, Relative Entropy of States of von Neumann Algebras (pdf)

A characterization of relative entropy on finite-dimensional C-star algebras is given in

• D. Petz, Characterization of the relative entropy of states of matrix algebras (pdf)

A survey of entropy in operator algebras is in

• Erling Størmer, Entropy in operator algebras (pdf)