What does a random seed reproduce in PyTorch?
In sampling from distributions, probabilities specified how likely each character was, while uniform draws selected actual characters. Where do those random-looking numbers come from?
A pseudorandom number generator, or PRNG, updates an internal state using a deterministic algorithm. A seed initializes that state. Repeating the same initialization and sequence of operations can reproduce the same outputs.
Predict a tiny generator
Our toy generator uses one integer as its entire state:
The remainder operation keeps the state between 0 and 15. Starting from seed , calculate the next four states before running it. The seed is the starting state, not the first emitted output. Divide each emitted state by 16 to get a number in .
Record the reference with 7, 0, and 4. Predict what will happen if you record exactly the same settings again, then test it. Each recorded run restarts from the selected seed.
Toy recurrence x ← (5x + 3) mod 16; u = x / 16. Each recorded run restarts at the chosen seed, consumes the selected number of earlier outputs, then shows the next outputs. Fixed category probabilities are 60%, 30%, 10%. This tiny 16-state generator illustrates state and repetition; it is not PyTorch’s generator and cannot represent those probabilities exactly.
0/16 runs. Each run keeps its own inputs and results. Work stays in this tab across reloads; download a report before closing it.
Write a prediction to record your reference.
Open and compare the two sequences. Explain their agreement before opening the arithmetic.
Check the recurrence
For example, . The next outputs are obtained by feeding each new state back into the same rule. Their divided values are .
Using probabilities , these map to a, a, a, b. Repeating the seed and operations repeats the states and outcomes. No new physical randomness enters this toy recurrence.
Change the seed, then change consumption
Choose another seed. Predict whether it changes the probability intervals or the generated values. Change only the seed, keeping the recurrence, probabilities, earlier consumption, and output count fixed. Record and compare with the reference.
Next return to . Predict the next four inspected outputs if one output was already consumed by an earlier operation. Change only to 1, then record the run. Compare with the original seed-7 reference.
Explain why the same seed can now accompany different inspected outputs.
Separate initialization from current state
Changing the seed changes where the sequence starts. It does not change our fixed probability intervals. Different random values can still map to the same character, so different seeds need not produce different short character strings.
With seed 7 and one earlier output consumed, 6 has already been used. The four inspected states are now . The generator did not forget its seed or change its rule: it advanced.
The seed describes initialization. The current state describes where the generator is now. The order and number of state-consuming operations therefore matter for reproduction.
This tiny generator is for tracing, not realistic sampling. It has only 16 possible states and repeats after 16 transitions. Its 16 uniform-grid values allocate 10 points to a, 5 to b, and 1 to c, giving full-cycle proportions 62.5%, 31.25%, and 6.25%. That coarse grid cannot represent 60%, 30%, and 10% exactly. Real generators use much larger state spaces and better statistical properties; these are not PyTorch’s seed outputs.
Reproduce sampling with a PyTorch generator
An explicit generator lets you keep this sampling stream separate from unrelated operations. Predict which pairs below must agree in the same environment before running the code:
import torch
p = torch.tensor([0.6, 0.3, 0.1])
g = torch.Generator(device="cpu").manual_seed(42)
first = torch.multinomial(p, 5, replacement=True, generator=g)
saved = g.get_state().clone()
second = torch.multinomial(p, 5, replacement=True, generator=g)
g.set_state(saved)
second_again = torch.multinomial(p, 5, replacement=True, generator=g)
assert torch.equal(second, second_again)
g.manual_seed(42)
first_again = torch.multinomial(p, 5, replacement=True, generator=g)
assert torch.equal(first, first_again)Write an explanation before opening the feedback. Do not assert that first and second must differ: categorical outcomes can coincide by chance.
Explain the reset and resume
The first sampling call advances g. Saving its state captures that position. Restoring it repeats the second call. Reseeding with 42 returns to the initial position, repeating the first call instead.
The algorithm, probability tensor, call shapes, device, and operation order are unchanged within each comparison. The example compares whole sampling calls; do not assume every sampler consumes exactly one uniform number per output or that splitting a batch preserves a sequence.
torch.Generator exposes manual_seed, get_state, and set_state. Pass the generator to operations that support it, using a device compatible with the tensors. A generator for one stream does not automatically govern every random operation in a program.
Default generators and experiment reproducibility
torch.manual_seed(42) seeds PyTorch’s default generators. It is convenient when initializing a complete experiment. An explicit g controls calls that receive generator=g; merely creating it does not seed ordinary module initialization or dropout. Python and NumPy also maintain their own random state.
Random weight initialization, shuffling, and dropout can all affect training. A reproducible run also depends on data, operation order, software, and hardware. Some operations are nondeterministic. PyTorch does not promise identical results across all releases, platforms, or CPU/GPU execution; deterministic-algorithm settings address some additional sources of variation. See PyTorch’s reproducibility guidance.
Repeating one seed helps debug and compare controlled changes. It does not establish that a result holds across random initializations; that requires a comparison over several seeds and reporting the variation.
Where does an unspecified seed come from?
The library chooses how to initialize its generator when you do not provide a seed. Systems commonly obtain initialization entropy from operating-system facilities, which can incorporate hardware and system events. Do not infer a particular initialization rule for every library.
Pseudorandomness describes deterministic generation from state. Physical randomness comes from measured physical processes. A seeded ML generator can produce many values without fresh physical entropy for every draw; it is not intended for security-sensitive secrets. PyTorch also offers Generator.seed() to initialize from a nondeterministic source and return the chosen seed.
Design a state-consumption test
Choose a seed and record eight outputs with no earlier consumption. Choose an available nonzero consumption count, and predict the next four inspected outputs by referring to your first trace. First record a four-output reference at the same seed and zero consumption; then change only consumption for the matched test. Check against both the reference and the longer trace.
Record what would contradict your state-advancement explanation. Then test a full cycle of 16 outputs. Does your chosen seed return? Explain why switching seeds in this toy can choose a different position on the same cycle rather than an unrelated sequence.
Check your trace
If you consumed outputs, the next four should be entries through of the unchanged longer sequence. Any mismatch under identical settings would contradict this deterministic recurrence.
All 16 seed choices lie on the same 16-state cycle in this particular toy. This is a property of its rule, not a claim about every real PRNG or every pair of PyTorch seeds.
Transfer: spot the reseeding mistake
Someone places g.manual_seed(42) inside a loop immediately before each identical sampling call. Predict whether this produces successive draws from the stream or repeatedly uses its beginning. Then explain whether changing the probabilities while keeping a seed fixed must preserve the sampled characters.
Check the reasoning
Reseeding before every identical call restarts the stream each time, so it repeats the same output block. Initialize once before the loop to advance naturally. Reset deliberately when reproducing a comparison.
The seed does not fix the probability distribution. Changing probabilities can change which outcomes are selected even with identical random inputs. Conversely, changing the seed leaves probabilities unchanged but changes the random stream. The sampling note lets you isolate both interventions.
Download your report with the seed, recurrence, consumed outputs, and actual trace. For PyTorch experiments, also record the version, device, inputs, relevant settings, and sequence of random operations.
Continue
Previous: Sampling from distributions · Next: Missing values · All statistics notes