Can a model become more confident without getting more answers right?

The MNIST article introduces logits, softmax, and cross-entropy. The hyperparameter article compares loss with accuracy as training continues. This experiment isolates why these measurements can move differently. You need probabilities, class labels, and the idea that $-\log(p)$ penalizes a low probability assigned to the true class.

We freeze three examples instead of training a model:

Example	Logits	True class
A	$(2,0,0)$	0
B	$(2,0,0)$	1
C	$(0,0,2)$	2

Temperature $T>0$ divides every logit before softmax. The predicted class is the largest logit. A common offset adds the same number to every logit; leave it at zero for the first comparison.

Establish a reference

Use $T=1$ and offset 0. Predict the three class labels and identify which example is wrong. Then predict which example contributes most to cross-entropy. Run the reference and record accuracy, mean predicted-class probability, and mean loss, as well as the per-example losses.

Change only temperature

Keep every logit and true label fixed. Predict the effect of $T=0.25$ before running it. Will sharper probabilities fix the incorrect example, or make that mistake more costly?

Then test $T=2$ under the same fixed conditions. Compare the mean loss with the reference and inspect the per-example values to explain the difference.

Experiment bench

Load reference settings

Three fixed examples: logits (2,0,0), (2,0,0), (0,0,2); true classes 0,1,2. Apply the same positive temperature and common offset to each row. No training; cross-entropy uses natural logarithms and a stable log-sum-exp calculation.

Load the reference, write a prediction, and run it first. For each comparison, change only the condition named in the lesson. Each run is calculated here in your browser.

Temperature T0.250.5124Common logit offset-1000-5051000

Prediction and reason for this run

Run experimentReference settings selected · 0/16 runs

Loading reference settings preserves your evidence. Work is saved in this tab across navigation and reloads. Download your report to keep it after closing the tab.

No run recorded yet.

Build your explanation from the evidence

Observations: which runs and values are you comparing?Interpretation: what explains the result?Limitations: what does this experiment leave unresolved?Next test: what will you change, and what result would challenge your explanation?

Download reportClear runs

Before opening the explanation, describe separately what happened to the class decisions, confidence, and loss. A mean can hide opposing changes among examples.

Compare the three measurements

All tested positive temperatures preserve the winning class, so accuracy stays at $2/3$. At $T=1$, the winning probability is about 0.787 and mean loss about 0.9062. At $T=0.25$, confidence rises above 0.999, but mean loss rises to about 2.6673 because B is now confidently wrong. At $T=2$, mean loss is about 0.8848.

This small constructed batch is enough to show that confidence and accuracy are different measurements. It does not establish an optimal temperature for MNIST, calibration on a population, or the cause of a real model’s validation-loss increase.

Design your own comparison

Choose another available temperature, predict the direction of the loss change, and compare it with the reference. Keep the examples and offset fixed. Use the individual losses to explain whether improved correct-example losses outweigh the incorrect-example loss.

For a separate test, restore $T=1$ and choose a nonzero common offset. Predict whether probabilities change even though all raw scores change. Test a small offset and then 1000, keeping temperature fixed. The bench uses a stable calculation, so it does not exponentiate the large raw logits directly.

For your independent comparisons, use Comparison baseline to select a recorded run that differs in only the condition you are testing.

Report and transfer

Specify that these are forward evaluations of a frozen batch, not retraining or independent validation measurements. Report the sample where the loss change is concentrated.

Would correcting B’s label change how temperature affects the mean loss? Work this case on paper; the bench intentionally keeps labels fixed.

Check the reasoning

A common offset cancels from softmax because the same exponential factor appears in numerator and denominator. Subtracting the largest scaled logit before exponentiating uses this identity to avoid overflow.

If B’s true class were 0, all three predictions would be correct. Sharpening the probabilities toward their unique winning classes would then decrease all three cross-entropies. The conclusion about the batch depends on its labels, not just its confidence scores.

Continue experimenting

Previous: Which shifts does pooling actually ignore? · Next: Why can eight nonzero updates leave a value unchanged? · All machine learning experiments