L'INFORMATION — Entropia II

Shannon

The same sum as Boltzmann's, written in 1948 for an entirely different trade.

Shannon wanted to know how many wires a message needs, and arrived at $H=-\sum p_i\log p_i$ — which, up to the factor $k$, is Boltzmann's formula. This is not an analogy: in both cases you are counting the possibilities you have not yet ruled out. Drag the bars. Spread evenly, uncertainty is greatest and each symbol costs $\log_2 n$ bits. Pile all the probability onto one symbol and $H$ falls to zero: a message whose content you already know carries nothing.

The distribution

symbols6

H (bits)—

H max = log₂ n—

redundancy—

most likely—

its surprise—

surprise of one symbol: −log₂ p bits

drag a bar up or down — the rest renormalises itself

Printed English

Shannon measured the entropy of English by having his wife guess. Your turn.

In 1951 Shannon published an experiment of disarming simplicity. Show someone the text up to a point, ask for the next letter, count the tries before they get it. If they guess first time, that letter carried almost nothing; if it takes ten tries, it was genuinely informative. The number of tries is itself a message that can be decoded — so its entropy bounds the entropy of the text. Shannon found roughly $1.3$ bits per character, where twenty-seven equally likely symbols would demand $4.755$. Put another way: three quarters of written English is foreseeable. Play below; the page counts for you.

The game

characters played0

tries in all0

first try—

your bound, bits/char—

27 equal symbols4.755

frequencies of this passage—

Shannon, 1951: ≈ 1.3 bits per character

how many tries it took you — far left: right first time

L'expérience originale, avec un seul sujet et une lucidité qui n'a pas vieilli : C. E. Shannon, Prediction and Entropy of Printed English, Bell System Technical Journal 30 (1951) 50–64.

Compression

Entropy is not a metaphor here: it is the floor, in bytes, that nothing gets under.

Give frequent symbols short codes and rare ones long codes: that is the whole of Huffman's algorithm, and it is optimal among codes that assign one binary word per symbol. Write anything below and watch the tree assemble itself, two smallest nodes at a time. The average code length always lands between $H$ and $H+1$ bits — never under. Type the same letter a hundred times: $H$ collapses and so does the code. Type well-mixed nonsense: $H$ climbs toward $\log_2 n$ and there is nothing left to win. Compression does not make room, it recovers the room redundancy was wasting.

The text

distinct symbols—

characters—

H (bits/symbol)—

mean L (bits/symbol)—

Huffman total—

against 8 fixed bits—

the theorem: H ≤ L < H + 1, always

the Huffman tree — a leaf's depth is the length of its code

Maxwell’s demon

A tiny being, a frictionless door, and the second law in danger for eighty years.

Maxwell imagined him in 1867: a doorkeeper fine enough to see every molecule, who opens a hatch only for the fast ones heading right and the slow ones heading left. No work is done — the door is massless and frictionless. Yet the right side heats, the left cools, and a temperature difference has been made out of nothing. Switch him on and watch the two temperatures part. The counter below tracks what the demon had to do to get there: record one decision per molecule. Remember that number — it is what pays, in the next chapter.

The hatch

T on the left—

T on the right—

ratio T right / T left—

decisions recorded0

ΔS of the gas—

verdict—

camellia: fast · marine: slow · the two temperatures over time below

Landauer

Thinking is free. Forgetting costs kT ln 2 per bit, and not one joule less.

The answer came in 1961, from Rolf Landauer, and it was not about measuring: it was about forgetting. The demon may look, decide and act — none of that carries a compulsory thermodynamic price. But his memory is finite; to keep going he must clear it, and clearing is irreversible. Two possible states reduced to one is a phase space halved, hence at least $k\ln2$ of entropy poured into the bath: $kT\ln2$ of heat, about $3\cdot10^{-21}$ joules at room temperature. The bead below lives in a double well; the protocol lowers the barrier, tilts, restores — and it ends on the right whatever side it started. That is an erasure. Two things to watch. First: hurry the protocol and the erasure fails — the bead is sometimes left on the wrong side, and an erasure that fails is no erasure. Second: the dissipated heat sits well above $\ln2$. That is expected, and it is honest. The bound is a floor reached only by protocols engineered for it; this one is crude, and it pays for being crude. What you can check here is the direction of travel: slow down and the cost falls — and the floor is never crossed.

The erasure

τ protocol duration600

erasures0

successful—

⟨Q⟩ / kT—

the bound, ln 20.693

excess over the bound—

verdict—

the double well through the protocol, and ⟨Q⟩ against Landauer's bound

Mesuré, et pas seulement déduit : A. Bérut, A. Arakelyan, A. Petrosyan, S. Ciliberto, R. Dillenschneider, E. Lutz, Experimental verification of Landauer's principle linking information and thermodynamics, Nature 483 (2012) 187 — une bille colloïdale dans un double piège optique, exactement la figure ci-dessus.

Szilard’s engine

One molecule, one bit, and a ledger that closes at exactly zero.

Szilard, in 1929, reduced the demon to its barest form: one molecule in a box. Insert a partition, look at which side it is on — that is one bit — then let the partition act as a piston and be pushed to the end. The isothermal expansion of one molecule across half the volume yields exactly $kT\ln2$ of work. You have just turned a bit into energy. But your memory now holds that bit; to go round again you must erase it, and Landauer presents the bill: $kT\ln2$. Run the cycle and watch the right-hand column. The work extracted is real, the debt is real, and their sum is zero to the last decimal. The second law was not saved by a technicality: it was saved by bookkeeping.

The cycle

step—

complete cycles0

work extracted / kT0

erasure paid / kT0

memory—

net balance / kT0

per cycle: +ln 2 of work, −ln 2 of erasure

the box, the partition, the memory — and the ledger that will not move

Réalisé sur un seul électron : J. V. Koski, V. F. Maisi, T. Sagawa, J. P. Pekola, Experimental realization of a Szilard engine with a single electron, PNAS 111 (2014) 13786 — environ $kT\ln2$ de travail extrait par bit, comme annoncé.

Épreuves the page checks by itself—

Six commissions at the telegraph office — and an endless examen. The page measures; the bits are counted, not estimated.

Pitfalls

Six ways information catches the careless.

One formula, two quantities$H$ is counted in bits, $S$ in joules per kelvin. They become the same thing only once a temperature is named: $S=k\ln2\cdot H$.

Measuring is freethis is the mistake that stood for thirty years after Szilard. It is not the looking that pays, it is the forgetting.

Landauer's bound is not a chip's power drawit is over a billion times smaller. It is a physical floor, not an engineering forecast.

A prefix code is not magicHuffman is optimal symbol by symbol. Over blocks, or with a model, you go lower — down to $H$, never under.

Entropy depends on the model“the entropy of English” does not exist absolutely: it is 4.7 bits to someone who knows nothing, 4.1 with letter frequencies, 1.3 for a human. The text did not change.

Compressing twice wins nothingan already-compressed file is roughly incompressible: its entropy per symbol is at the ceiling. That is the very definition of the job being done.

Continued in LE MONDE, where we leave beads and bits for the coffee cup, the Earth, the black holes and the entire budget of the universe.