Rényi entropy




In information theory, Rényi entropy refers to a class of measures, of entropy that are essentially logarithms of diversity indices.

For special values of its parameter, the notion of Rényy entropy reproduces all of: Shannon entropy, Hartley entropy/max-entropy and min-entropy.


Let pp be a probability distribution over nn \in \mathbb{N} elements, and let α\alpha be a non-negative real number not equal to 1:

α 0{1}. \alpha \;\in\; \mathbb{R}_{\geq 0} \setminus \{1\} \,.

The Rényi entropy of pp at order α\alpha is:

H α(p)11αlog( i=1 n(p i) α). H_\alpha(p) \;\coloneqq\; \frac{1}{1-\alpha} \log \left( \sum_{i=1}^n (p_i)^\alpha \right) \,.


Relation to other notions of entropy

For various (limiting) values of α\alpha the Rényi entropy reduces to notions of entropy that are known by their own names:

order0\phantom{\to} 01\to 12\phantom{\to}2\to \infty
Rényi entropyHartley entropy\geqShannon entropy\geqcollision entropy\geqmin-entropy


The Rényi entropy is an anti-monotone function in the order-parameter α\alpha:

α 1α 2H α 1(p)H α 2(p). \alpha_1 \;\leq\; \alpha_2 \;\;\;\;\;\;\; \Rightarrow \;\;\;\;\;\;\; H_{\alpha_1}(p) \;\geq\; H_{\alpha_2}(p) \,.

(e.g. Ram & Sason 16, Fact 1)

In particular, in terms of the above special cases, this means that

HartlyEntropyShannonEntropyCollisionEntropyMinEntropy. HartlyEntropy \;\geq\; ShannonEntropy \;\geq\; CollisionEntropy \;\geq\; MinEntropy \,.


Due to:

  • Alfréd Rényi, On Measures of Entropy and Information, Berkeley Symposium on Mathematical Statistics and Probability, 1961: 547-561 (1961) (euclid)

Textbook account:

  • J. Aczél, Z. Daróczy, Chapter 5 of: On Measures of Information and their Characterizations, Mathematics in Science and Engineering 115, Academic Press 1975 (ISBN:978-0-12-043760-3)

See also

On holographic Renyi entropy in relation to holographic entanglement entropy and quantum error correcting codes:

Last revised on May 28, 2021 at 08:14:23. See the history of this page for a list of all contributions to it.