Verified formula topic
Information Theory Formulas
Browse 0 reviewed information theory formulas with concrete inputs, outputs, calculations, applicability checks, and LaTeX.
Browse 3 supporting references
Information Theory
Cross-Entropy
H(p,q)=-\sum_x p(x)\log q(x)Measures the expected coding cost when q is used for data distributed as p.
Open formulaInformation Theory
Kullback–Leibler Divergence
D_{\mathrm{KL}}(p\|q)=\sum_x p(x)\log\frac{p(x)}{q(x)}Measures directed discrepancy from a reference distribution q to a target distribution p.
Open formulaInformation Theory
Mutual Information
I(X;Y)=\sum_{x,y}p(x,y)\log\frac{p(x,y)}{p(x)p(y)}Measures how much knowing one random variable reduces uncertainty about another.
Open formula