Why can eight nonzero updates leave a value unchanged?

The articles on JavaScript and Python floating-point numbers and number formats in model memory explain that a format’s spacing depends on magnitude. This experiment isolates rounding when a value is repeatedly stored. You need addition, powers of two, and the idea of rounding to a representable number.

We add an increment to a stored value eight times. Float64 uses JavaScript’s usual number arithmetic. Float32 rounds the value after each addition. The ideal column uses exact arithmetic in quarter-units, so it can show increments that the stored format loses.

Establish a reference

Start at $2^{24}=16{,}777{,}216$, increment by 1, use eight updates, and select float64. Predict the final value and total change. Run and record every stored value alongside the ideal value.

Change only the storage format

Keep the start, increment, and number of updates fixed. Predict the outcome when Stored number format becomes Float32 (round after each update). Run the comparison. If one update disappears, will repeating it eight times necessarily make it accumulate?

Experiment bench

Load reference settings

Repeatedly add the same positive increment. Float32 rounds the stored value after every addition using Math.fround; float64 uses JavaScript Number. The ideal total is computed exactly in quarter-units with BigInt. This is a storage-rounding experiment, not a model-quality or GPU-speed benchmark.

Load the reference, write a prediction, and run it first. For each comparison, change only the condition named in the lesson. Each run is calculated here in your browser.

Stored number formatFloat64 (JavaScript Number)Float32 (round after each update)Starting value11048576167772169007199254740992Increment per update0.25124Number of updates481632

Prediction and reason for this run

Run experimentReference settings selected · 0/16 runs

Loading reference settings preserves your evidence. Work is saved in this tab across navigation and reloads. Download your report to keep it after closing the tab.

No run recorded yet.

Build your explanation from the evidence

Observations: which runs and values are you comparing?Interpretation: what explains the result?Limitations: what does this experiment leave unresolved?Next test: what will you change, and what result would challenge your explanation?

Download reportClear runs

Explain the stored sequence, rather than looking only at the requested total change. The ideal total is not a substitute for inspecting the actual updates.

Compare the stored values

Float64 reaches 16,777,224, a change of 8. Float32 stays at 16,777,216 throughout. At this magnitude, the next larger float32 value is 16,777,218. Adding 1 lands halfway between the two; ties-to-even rounds back to 16,777,216. Every subsequent update repeats that same situation because the lost part was not stored anywhere.

This is a spacing and rounding effect. It is not evidence that a gradient was zero, or that all low-precision training fails. Real optimizers can use different formats for stored weights, accumulators, and computation.

Design your own recovery test

Keep float32, the starting value, and the update count fixed. Choose a different increment that you think will produce an observable change, then predict and run it. Explain why it is representable at this magnitude.

In a separate comparison, restore increment 1 and choose a smaller starting value. Predict whether float32 will now preserve the updates. Change only the start for this comparison.

Finally, use float64 and compare the reference start with $2^{53}$, holding increment 1 and eight updates fixed. Does using more bits eliminate the phenomenon or move the boundary?

For your independent comparisons, use Comparison baseline to select a recorded run that differs in only the condition you are testing.

Report and transfer

Record the format, start, increment, count, and observed total change for each run. State why this arithmetic experiment cannot establish model accuracy, memory consumption for a whole training job, or device speed.

On paper, compare eight separately rounded additions of 1 with one addition of 8 at the float32 reference start. Are these the same computational procedure?

Check the recovery and transfer

At the float32 reference start, increments of 2 are representable and eight such updates give a total change of 16. At the smaller start $2^{20}$, increments of 1 are also retained. Float64 has its analogous spacing boundary at $2^{53}$, where repeated increments of 1 round back under the same tie rule.

One addition of 8 reaches 16,777,224 in float32. Eight separately rounded additions of 1 stay at the starting value. Accumulating an update in a wider format before storing it changes where rounding occurs; it is not the same sequence of operations.

Continue experimenting

Previous: Can a model become more confident without getting more answers right? · All machine learning experiments