Information Theory: Notation Reference explains entropy, conditional entropy, mutual information, divergence, and logarithm bases.
Core notation for Information Theory
In information theory, the notation usually represents random variables, alphabets, distributions, code lengths, channels, and logarithm bases. The table below gives a compact starting set for information theory; define any local variation before the first calculation.
| Concept | Notation | How to read it |
|---|---|---|
| KL divergence | D_{\mathrm{KL}}(P\Vert Q)=\sum_x p(x)\log\frac{p(x)}{q(x)} | directed divergence from Q to P |
| Channel capacity | C=\max_{p(x)}I(X;Y) | maximum mutual information over input distributions |
| Chain rule | H(X,Y)=H(X)+H(Y\mid X) | joint entropy decomposition |
| Entropy | H(X)=-\sum_x p(x)\log p(x) | average uncertainty of X |
How to read a complete expression
Read D_{\mathrm{KL}}(P\Vert Q)=\sum_x p(x)\log\frac{p(x)}{q(x)} as “directed divergence from Q to P.” Identify the outer operation first, then its inputs, index set, conditions, and result type. This prevents a superscript, bar, vertical line, or styled letter in information theory from being interpreted by appearance alone.
Conventions that belong in the notation table
- State the logarithm base and resulting unit.
- Distinguish entropy, cross-entropy, and kl divergence.
- Specify discrete sums versus continuous integrals.
Cross-checks before reuse
- Verify probabilities normalize.
- Test deterministic and uniform distributions.
- Confirm the order of p and q in every divergence.
Accessibility and portability
Keep the information theory source selectable and editable. For an isolated character in information theory, Unicode text may be sufficient; for structured expressions, preserve LaTeX, MathML, or a native equation object. When an image of information theory is unavoidable, describe the operation, inputs, conditions, and conclusion rather than listing glyph names.
Verification checklist
- Verify probabilities normalize.
- Test deterministic and uniform distributions.
- Confirm the order of p and q in every divergence.
- Confirm every symbol used in information theory has one defined meaning in the local context.
- Reopen the exported file for Information Theory: Notation Reference and compare it with the editable source.
How this guide was checked
Page purpose: information theory notation reference — Look up notation, definitions, and usage conventions
Automated quality check: Kept noindex until critical findings are resolved.
Verification references
These primary standards and official documentation pages were used to check character identity, syntax, or platform behavior described above.
- The Unicode StandardUnicode Consortium — Character identity, encoding, names, and conformance.
- Unicode Technical Report #25: Unicode Support for MathematicsUnicode Consortium — Mathematical character usage, variants, and notation support.
- LaTeX Project DocumentationThe LaTeX Project — LaTeX syntax, authoring model, and official documentation links.
- MathML CoreW3C — Semantic web mathematics elements and browser behavior.