nLab Rényi entropy

Contents

Contents

Idea

In information theory, Rényi entropy refers to a class of measures, of entropy that are essentially logarithms of diversity indices.

For special values of its parameter, the notion of Rényy entropy reproduces all of: Shannon entropy, Hartley entropy/max-entropy and min-entropy.

Definition

Let pp be a probability distribution over nn \in \mathbb{N} elements, and let α\alpha be a non-negative real number not equal to 1:

α 0{1}. \alpha \;\in\; \mathbb{R}_{\geq 0} \setminus \{1\} \,.

The Rényi entropy of pp at order α\alpha is:

H α(p)11αlog( i=1 n(p i) α). H_\alpha(p) \;\coloneqq\; \frac{1}{1-\alpha} \log \left( \sum_{i=1}^n (p_i)^\alpha \right) \,.

Properties

Relation to other notions of entropy

For various (limiting) values of α\alpha the Rényi entropy reduces to notions of entropy that are known by their own names:

order0\phantom{\to} 01\to 12\phantom{\to}2\to \infty
Rényi entropyHartley entropy\geqShannon entropy\geqcollision entropy\geqmin-entropy

Monotonicity

The Rényi entropy is an anti-monotone function in the order-parameter α\alpha:

α 1α 2H α 1(p)H α 2(p). \alpha_1 \;\leq\; \alpha_2 \;\;\;\;\;\;\; \Rightarrow \;\;\;\;\;\;\; H_{\alpha_1}(p) \;\geq\; H_{\alpha_2}(p) \,.

(e.g. Ram & Sason 16, Fact 1)

In particular, in terms of the above special cases, this means that

HartlyEntropyShannonEntropyCollisionEntropyMinEntropy. HartlyEntropy \;\geq\; ShannonEntropy \;\geq\; CollisionEntropy \;\geq\; MinEntropy \,.

References

Due to:

  • Alfréd Rényi, On Measures of Entropy and Information, Berkeley Symposium on Mathematical Statistics and Probability, 1961: 547-561 (1961) (euclid)

Textbook account:

  • J. Aczél, Z. Daróczy, Chapter 5 of: On Measures of Information and their Characterizations, Mathematics in Science and Engineering 115, Academic Press 1975 (ISBN:978-0-12-043760-3)

See also

On holographic Renyi entropy in relation to holographic entanglement entropy and quantum error correcting codes:

Last revised on May 28, 2021 at 12:14:23. See the history of this page for a list of all contributions to it.