Shannon's Entropy Formula
Measure the information content and uncertainty in data.
The foundation of information theory and data compression.
The Formula
H = -Σ p(x) × log₂(p(x))
Shannon's entropy measures the average amount of information (in bits) per symbol in a message. Higher entropy means more unpredictability and more bits needed to encode the data.
Variables
| Symbol | Meaning |
|---|---|
| H | Entropy (measured in bits when using log base 2) |
| p(x) | Probability of each possible symbol or outcome |
| Σ | Sum over all possible symbols |
| log₂ | Logarithm base 2 |
Example 1
Find the entropy of a fair coin flip
Two outcomes: Heads (p = 0.5), Tails (p = 0.5)
H = -(0.5 × log₂(0.5) + 0.5 × log₂(0.5))
H = -(0.5 × (-1) + 0.5 × (-1))
H = 1 bit (maximum entropy for two outcomes)
Example 2
A source emits A (70%), B (20%), C (10%). Find the entropy.
H = -(0.7 × log₂(0.7) + 0.2 × log₂(0.2) + 0.1 × log₂(0.1))
H = -(0.7 × (-0.515) + 0.2 × (-2.322) + 0.1 × (-3.322))
H = -(−0.360 − 0.464 − 0.332)
H ≈ 1.157 bits per symbol
When to Use It
Use Shannon's entropy when:
- Measuring the information content of a data source
- Designing efficient data compression algorithms
- Evaluating the randomness or predictability of data
- Building decision trees in machine learning (information gain)