We built the simplest physics-discovery experiment we could think of. A mass on a spring slides on a frictionless table. We sample its position at discrete instants, with a little measurement noise. The true law is known in advance, Hooke's law ẍ = −(k/m)·x, which makes the setup a perfect test bench.

The machine's job is to estimate velocity and acceleration from the samples, take a library of eight candidate terms {1, x, ẋ, x², xẋ, ẋ², x³, x²ẋ}, and find the combination that best explains the measured accelerations. Fit the data, prefer simple equations. What could go wrong?

Two things, it turns out, and both are the alignment problem in miniature:

  • Fit and simplicity have to live in one loss. That means choosing an exchange rate between them, and the right rate depends on things you don't know.
  • Noise picks the winner. The "most likely model" is a property of the particular noise you happened to get. Rerun the same experiment and a different law wins.

Try it. Panel 1 shows which law wins at every exchange rate; panel 2 reruns the identical experiment under fresh noise.

Fit alone is the wrong objective

With nothing but "minimize the error", the machine never returns Hooke's law. It returns all eight terms, and it is right to: every extra coefficient soaks up a little noise, so the bloated equation genuinely fits better. The favourite impostor, once you charge a little for terms, is a cubic spring ẍ = a·x + b·x³. On a single orbit x and x³ are about 95% correlated (cos³θ is mostly cos θ), and differentiation amplifies noise (acceleration error grows like σ/Δt²), so the cubic has plenty of noise to explain.

We had an intent, "find the true law", and a proxy, "minimize error on these derivative estimates". The optimizer satisfied the proxy and missed the intent. That is Goodhart's law on a frictionless table, a cousin of the specification gaming examples the alignment community collects. The system did not fail its specification. The specification failed us.

Difficulty 1: one loss, two currencies

The fix is to make simplicity part of the objective. We fit all 256 subsets of the library and score each one as n·ln(RSS/n) + γ·cost. Every term pays rent γ, and survives only if it buys more error reduction than it costs. With flat rent of 2 per term this is AIC. With graded rent, priced by description length, x costs 2 and x³ costs 4: a nudge towards low powers that real cubic data can still override.

But error and complexity are different currencies, and γ is the exchange rate. Nothing in the data tells you what it should be. Panel 1 shows what is at stake. Rent too low and extra terms buy their way in. Rent too high and the spring itself gets priced out: the machine concludes that no force acts at all. Hooke's law wins only inside a window between the two, and the window shrinks as noise grows. In one run, Hooke's law won for γ from 0.2 upward at low noise, for 1.2 to 10.6 at σ = 0.3, and only for 1.7 to 6.4 at σ = 0.4. The right exchange rate depends on the noise level, which is exactly what you don't know.

The exchange rate is not even the deepest choice. "Cost" depends on the symbols. In our polynomial library Hooke's law is cheap. Give the library an energy term and the conservation law ẋ² + ω²x² = const is the one-liner; give it cos(ωt) and the solution itself is atomic. Goodman's new riddle of induction made this point in 1955 with "grue". Kolmogorov complexity only fixes simplicity up to a constant that depends on the choice of machine. The constant washes out with infinite data, so simplicity is quasi-objective in the limit. It bites with finite data, which is the only kind anyone has.

This is the same move alignment makes everywhere. RLHF maximises reward minus a β-weighted KL penalty to the base model: two currencies, one exchange rate, chosen by hand. And the penalty is measured in the model's own representation, not ours. Carlsmith's Scheming AIs report notes that simplicity arguments inherit the choice of language. Mapping a model's concepts onto ours is the "ontology identification" problem at the centre of ARC's Eliciting Latent Knowledge report, with an ancestor in de Blanc's Ontological Crises (2011). Whoever chooses the library chooses what the optimizer finds natural.

Difficulty 2: noise picks the winner

Fix the exchange rate and the symbols, and the answer is still not determined. Panel 2 reruns the same experiment, same spring, same method, under 20 fresh noise draws. At σ = 0.3 with flat AIC rent, one run of 20 gave six different winning laws, and Hooke's law won only 8 of them. The rest were cubic springs and stranger things, each the honest best fit to its own data.

A real experiment gets one draw and never sees the others. The "most likely model" is a random variable, and you observe one sample of it. That is uncomfortable in a physics toy. It is worse for a trained model, where each run is one draw of data, initialisation and order, and the objective it ends up pursuing is one sample from a distribution we never see.

Two defenses help, and neither is part of the original objective. A second experiment, releasing the mass from a smaller swing, breaks the x/x³ degeneracy, because x³ scales differently with amplitude. And a bootstrap vote resamples the data thirty times, reruns the whole selection on each, and keeps only terms that win at least 80% of the time. Truth, operationally, becomes what is stable under perturbation rather than what fits best once. At extreme noise the vote does the most aligned thing in the whole experiment: it declares that no law is supported, instead of inventing one.

Counting the patches

To recover one of the simplest laws in physics from one of the cleanest datasets imaginable, we needed an error term, a complexity rent, an exchange rate between them, a price list over symbols, a second experiment, and an ensemble vote with a refusal mode. We never stated the real objective, "find the true law", directly. We stacked proxies, each one an admission that the previous one leaked.

None of the machinery is new. Schmidt and Lipson distilled free-form laws from pendulum data in 2009, trading accuracy against parsimony. Our library is SINDy (Brunton, Proctor and Kutz, 2016) and the vote is Ensemble-SINDy (Fasel et al., 2022), which we reinvented before finding it. The live objection to the pessimism is Wentworth's natural abstraction hypothesis: the world may privilege certain concepts, so human and machine ontologies could converge. Our toy offers a little support, since Hooke's law stays cheap in any library built from position and its derivatives.

Still, that is the cost for ẍ = −(k/m)·x, where ground truth exists and checking takes a second. Aligning optimizers to human values means an exchange rate nobody can derive, one noisy draw of data, contested symbols, no ground truth, and a system far more capable of finding the cubic term we didn't mean.

Michael Paris writes about things that keep him up at night.