Grinstead 4.2: Entropy of a Biased Coin

Problem Statement

A biased coin has probability $p$ of heads and $1-p$ of tails.

(a) Find the entropy $H(p)$ as a function of $p$ . (b) For what value of $p$ is the entropy maximized? (c) Sketch the graph of $H(p)$ for $p \in [0, 1]$ .

Given

Two outcomes: Heads (probability $p$ ), Tails (probability $1-p$ )
$p \in [0, 1]$

Find

$H(p)$ — the entropy as a function of bias
$p^*$ — the value that maximizes entropy
Shape of the entropy curve

Solution

Approach

Apply the Shannon entropy formula directly, then use calculus to find the maximum.

Part (a): Find $H(p)$

By definition of Shannon entropy:

H(p) = -p \log_2 p - (1-p) \log_2 (1-p)

This is called the binary entropy function, often denoted $H_b(p)$ or $h(p)$ .

Part (b): Maximize Entropy

Take the derivative:

\frac{dH}{dp} = -\log_2 p - \frac{p}{p \ln 2} + \log_2(1-p) + \frac{1-p}{(1-p) \ln 2}

= -\log_2 p + \log_2(1-p) - \frac{1}{\ln 2} + \frac{1}{\ln 2}

= \log_2 \frac{1-p}{p}

Setting $\frac{dH}{dp} = 0$ :

\log_2 \frac{1-p}{p} = 0 \implies \frac{1-p}{p} = 1 \implies p = \frac{1}{2}

Verification: The second derivative is:

\frac{d^2H}{dp^2} = -\frac{1}{p \ln 2} - \frac{1}{(1-p) \ln 2} < 0

Since the second derivative is negative everywhere, $p = 1/2$ is indeed a maximum.

Part (c): Sketch

Key points:

$H(0) = 0$ (certain tails)
$H(1/2) = 1$ bit (maximum uncertainty)
$H(1) = 0$ (certain heads)
Symmetric about $p = 1/2$
Concave (bowl-shaped, opening down)

H(p)
  1 |        ___
    |      /     \
    |    /         \
    |  /             \
  0 |/_________________\___
    0       0.5        1    p

Final Answer

[!success] Answer (a) $H(p) = -p \log_2 p - (1-p) \log_2(1-p)$

(b) Maximum at $p^* = 1/2$ , where $H(1/2) = 1$ bit

(c) Symmetric, concave curve with max at center, zeros at endpoints

Discussion

The binary entropy function is fundamental—it appears everywhere:

Channel capacity of binary symmetric channel: $C = 1 - H_b(p)$
Bounds on error-correcting codes
Rate-distortion theory for binary sources

The symmetry $H(p) = H(1-p)$ reflects that a coin biased toward heads has the same entropy as one equally biased toward tails. What matters is the degree of bias, not its direction.

Variations

What if we use natural log? Same shape, but maximum is $\ln 2 \approx 0.693$ nats.
What about a three-sided die? See [[Ternary Entropy]].

[[Conditional Entropy of Binary Channel]]
[[Mutual Information of BSC]]

Mistakes I Made

First attempt: forgot to apply chain rule when differentiating $\log_2(1-p)$ . The base-2 log introduces a factor of $1/\ln 2$ that’s easy to miss.