Understanding BERT — masked language modeling and next-sentence prediction

If word2vec is the first piece of the modern stack — words turned into vectors whose geometry encodes meaning — BERT is the piece where the rest of the recipe snaps into place. Take a deep stack of Transformer blocks; let self-attention rewrite every word’s vector in light of all the others; pretrain the whole thing on a mountain of raw, unlabeled text with an objective that forces it to actually model language; only then bolt on a small task-specific head and fine-tune. That’s BERT in 2018 — and, give or take a couple of orders of magnitude of scale and a swap of “read bidirectionally to understand” for “read left to right to generate,” that’s GPT, Claude, Llama, and every retrieval system shipping today. Frontier models look intimidating because they’re a deep pile of ideas accumulated over decades, but almost every one of those ideas is smaller and far easier to see clearly on its own — and BERT is the one where you can watch the modern pretrain-then-adapt recipe work end to end in a model small enough to load on a laptop. That’s where the modern stack took its current shape, and it’s where this article starts too.

Before BERT, NLP looked like this: pick a task (sentiment, named-entity recognition, question answering), gather labeled data for that task, train a model from scratch — usually some flavor of LSTM. Repeat for every new task. Each model started from random weights and learned everything — vocabulary, syntax, world knowledge — from however many labeled examples you happened to have. Tasks with little labeled data did poorly. Tasks with lots of data did okay but never benefited from each other’s knowledge.

BERT (2018) replaced that shape with a different one: train one large model once on raw unlabeled text in a way that forces it to learn language, then attach a small task-specific head and fine-tune. Suddenly every NLP task got better at the same time, including ones with very little labeled data.

It’s worth naming what made this possible, because it wasn’t only BERT. Through the 2010s, NLP had worked through the deep-learning toolkit one architecture at a time — feed-forward nets, CNNs borrowed from vision, then RNNs and LSTMs — each an improvement on the last, and each stuck on something: a fixed input window, no real grip on long-range structure, or training that couldn’t be parallelized across the sequence. The Transformer was the first general enough to drop all three limitations at once, and general enough to stop being a single-monolingual-task tool — it carried over to multilingual and multi-task settings too. That generality is what turned transfer learning (TL) — pretrain one model, then adapt it to a new task or language by fine-tuning it on a small labeled dataset instead of training from scratch — from a nice idea into the default way NLP gets done. The idea itself predates Transformers (ULMFiT and ELMo were already pretraining LSTMs), but Transformers are what let it scale; BERT is the first of the reusable backbones, with GPT and T5 right behind it.

One piece of that story deserves its own mention, because it’s the hinge the whole architecture turns on: attention arrived before the Transformer, not with it. Plain RNN seq2seq models (Sutskever et al., 2014) compressed an entire input sentence into a single fixed-size context vector and handed that to the decoder — everything the output needed had to survive one bottleneck, and long sentences didn’t. Bahdanau et al. (2015) fixed this by letting the decoder look back over every input position at each step and learn which ones to focus on, associating parts of the output with the relevant parts of the input instead of leaning on a frozen summary. But that attention still rode on top of a sequential RNN, so the can’t-parallelize problem stayed. The Transformer (2017) made the radical move — keep the attention, throw the recurrence out entirely: every position attends directly to every other, a mechanism called self-attention, which is what finally made training parallel and great depth affordable.

The thing to keep in mind throughout this article is what that one model actually produces: context-aware embeddings. word2vec had already shown that representing words as dense vectors was powerful, but its vectors were static — bank got the same vector in “river bank” and “bank account.” BERT’s encoder reads the whole sentence and hands back a different vector for bank each time, shaped by the words around it. That one upgrade — static to contextual — is most of why a fine-tuned BERT beat hand-built task-specific architectures across the board, and it’s still why BERT-family encoders power search, retrieval, and clustering today.

One expectation to set straight up front, because it trips people up: BERT does not generate text — it only understands it. You feed it a sentence and it hands back those context-aware vectors, which a small task head then reads for understanding jobs — sentiment / topic / intent classification, named-entity recognition and other token tagging, extractive question answering (finding the answer span inside a passage), and pooled embeddings for semantic search, retrieval, clustering, and deduplication. None of these produce new text; they read the meaning BERT extracted. (The fine-tuning section near the end of this article walks through each of these head shapes concretely.) The masked-token prediction this whole article is built around is how BERT is pretrained — a device for learning good representations — not how it gets used afterward. Writing text, one next token at a time, is the job of the decoder (GPT) lineage, which is built from this same block but flips the attention mask to causal. The honest one-liner: BERT only understands; a decoder understands and generates. (Where that fork lives — the single bit of attention masking that separates the two — is the encoder–decoder article.)

The architecture that does this was familiar — the encoder half of the Transformer from the year before. The original Transformer was an encoder–decoder built for machine translation: the encoder read the source sentence, the decoder generated the target. BERT kept a deep stack of those encoder layers, threw the decoder away, and repointed them — from turning one language into another toward building a representation of language general enough that almost any understanding task can be read straight off it. What was new were the two pretraining objectives that made the “train once, fine-tune anywhere” recipe work — the objectives that actually taught the encoder to produce good context-aware embeddings: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). This article walks through both — what they do, why they were chosen, what they teach the model — with running Python code and live predictions from a real BERT.

Two related articles bracket this one: word2vec for the static embeddings BERT builds past, and RAG for where the contextual ones get used downstream — it pools BERT’s per-token output into one vector per passage and runs cosine similarity for retrieval.

Why a new training objective?

Up to 2018, the standard way to train a sequence model was next-token prediction: feed the model a sequence, ask it to predict the next token, compare the prediction to the actual next token, backpropagate. This works perfectly for an LSTM — it processes tokens left-to-right one at a time, and the architecture matches the task.

But left-to-right is a limitation. Consider this sentence with one word hidden:

The river ___ overflowed after the storm.

Reading left-to-right and stopping at the blank, several words look plausible: water, level, bank, flow. The word that disambiguates is overflowed, which appears after the blank. To get the right answer, the model needs to look at the right-hand context too — not just the left.

A river bank overflowing makes sense; a river level overflowing is awkward; a river flow overflowing is wrong. The clue isn’t in the past — it’s in the future, and a unidirectional model can’t reach it.

LSTMs got a partial fix for this: a bidirectional LSTM runs two separate LSTMs over the sequence, one left-to-right and one right-to-left, and concatenates their outputs at each position. That gives every position access to both directions, but in a shallow way — the two directions are independent until the very last layer, so neither half sees the other’s contextualized representations. The model learns “what does the left context look like” and “what does the right context look like” but never “what does the combined context look like” at depth.

The 2017 Transformer encoder fixed this in a much cleaner way: every token attends to every other token in the sequence in a single bidirectional self-attention layer, and that layer is stacked many times. Each position has deep bidirectional context — every layer sees the previous layer’s full bidirectional representation.

But there’s a catch: you can’t train a bidirectional encoder with next-token prediction. To predict token N from the tokens around it, you’d need the model not to see token N itself. With bidirectional attention, every other token in the sequence attends to N, so the answer trivially leaks back to the position trying to predict it. The model learns the identity function and nothing else.

Inside a self-attention layer

Before getting to the mask, it’s worth seeing what the attention layer actually computes. Each input token is projected into three vectors — a query, a key, and a value. Every position scores its query against every other position’s key, scales the scores and softmaxes them into attention weights, then mixes the values by those weights. Self-attention just means the queries, keys, and values all come from the same sequence — each token’s new representation is a weighted blend of every token’s value, itself included. BERT runs several of these in parallel — multiple heads, each with its own Q/K/V projections — so different heads can lock onto different relationships in the same sentence; their outputs are concatenated and projected back to one vector per token.

This is the layer, stacked many times, that the encoder is built from. We keep it at sketch depth here, because BERT’s encoder layers are ordinary Transformer blocks — the full mechanism (scaled dot-product attention, why we project into Q/K/V, how multi-head splits the work, the feed-forward sublayer, positional encodings) is its own article: Inside a transformer block. Everything in the rest of this section — encoder vs decoder, which objective each can be trained on — comes down to one small modification to the score step.

Encoder vs decoder — one bit of architecture

The whole encoder-vs-decoder distinction in modern transformer language models comes down to a single design choice: the attention mask. Mechanically the two halves are the same multi-head self-attention layer — same query/key/value projections, same dot products, same softmax. The only difference is whether a triangular mask is applied to the attention scores before the softmax. The mask sets certain positions to negative infinity, which the softmax pushes to zero, removing those positions from what each token can see. (For the encoder/decoder split as its own topic — not just BERT’s slice of it — see Understanding transformers: encoder, decoder, and the split.)

In the encoder grid every cell is filled — token N can attend to every other token in both directions. That’s the bidirectional context BERT relies on. In the decoder grid the upper-right triangle is masked out — token N attends only to tokens 1…N. That’s what makes a decoder usable for autoregressive generation: at inference time you generate token N+1 from the tokens already produced, and the same forward pass that worked at training time keeps working at inference because no position ever depended on a future token.

Once you frame the architecture as just a choice of mask, it becomes obvious which training objective fits which architecture:

There are exactly two viable cells, and they pin down the two model families that defined the next several years of NLP:

• Decoder + next-token prediction = GPT. The causal mask already hides the future, so asking the model to predict the next token is the natural objective. Position N produces a prediction for token N+1 using only positions 1…N as context. Same forward pass works for training and for inference; you just feed back the previous output. The whole GPT line — GPT-2, GPT-3, GPT-4, Llama, Claude — uses this combination.
• Encoder + masked-token prediction = BERT. Bidirectional attention means next-token prediction would leak (the answer attends to itself), but if the target is replaced with [MASK] instead of merely hidden, the model has to use full bidirectional context to recover it. That’s MLM, and it’s the entire reason BERT works.

The other two cells don’t lead anywhere useful. Encoder + next-token prediction leaks. Decoder + masked-token prediction is redundant — future tokens are already invisible to past positions under the causal mask, so replacing them with [MASK] adds no new signal; the only positions that would benefit from MLM-style targets are exactly the ones the causal mask already hid. Each architecture has exactly one objective that fits. The rule in one line: the objective follows from the mask — bidirectional attention (no mask) forces masked-token prediction; causal attention (the triangular mask) invites next-token prediction.

So the question “why does BERT need MLM?” has a precise answer: it’s the only training objective that can teach a bidirectional encoder anything. The bidirectional encoder is what gives BERT its understanding-task superpowers. The choice of objective follows from the choice of architecture, not the other way around.

BERT’s contribution is the two objectives that make the encoder approach work at scale:

• MLM swaps “predict the next token” for “predict tokens that have been hidden somewhere in the middle of the sequence.” The hidden tokens are replaced with a special [MASK] symbol. The model uses full bidirectional context to recover them, and nothing leaks because the original token is gone from the input.
• NSP adds a sentence-level objective on top: given two sentences, predict whether B actually followed A in the source text or was randomly sampled from somewhere else. The intent was to teach BERT something about discourse and inter-sentence relationships, on top of the within-sentence understanding that MLM provides.

Both objectives are trained jointly, in a single forward pass, on the same input. Every training example is a pair of sentences A and B with ~15% of the tokens masked. BERT runs once through the encoder, then two heads read off the same final-layer representations: one head predicts the masked tokens (MLM loss), the other head predicts whether B follows A (NSP loss). The two losses are summed and backpropagated together — there is no separate “MLM phase” and “NSP phase,” and the same encoder weights have to satisfy both objectives at once. The key thing to see is that the two heads are parallel readers of one shared output, not sequential stages: the sentence does not pass “through MLM and then into NSP.” The encoder produces one context-aware vector per token, and each head grabs the rows it cares about — MLM the masked positions, NSP the [CLS] position — at the same time.

The next sections walk through each objective in detail. Both are conceptually simple; the interesting parts are the design choices around them — particularly the masking trick that prevents the model from learning to depend on [MASK] itself.

BERT’s input format

Both objectives operate on the same input: a pair of token sequences A and B packed into a single fixed-length input. Three things are worth knowing about how that input is constructed, because they recur in every code sample below.

WordPiece tokenization. BERT doesn’t operate on whitespace-separated words. It uses a subword vocabulary of about 30,000 pieces, learned from the training corpus. Common words map to a single piece (paris, river); rare words split into prefix + continuation pieces (tokenization → token, ##ization). The ## prefix marks a continuation. This keeps the vocabulary small enough to embed (~30k × 768 = 23M parameters for bert-base) while still handling any word — even ones the model never saw at training — by falling back to character-level splits. (WordPiece is one of a family of subword schemes — BPE, SentencePiece, byte-level BPE — that differ in their merge strategy and unit; the tokenization article walks through all of them and shows how each splits the same sentence.) Throughout the article you’ll see lowercase tokens because we use bert-base-uncased, which lowercases everything during tokenization.

Special tokens. Every input starts with [CLS] (classifier) and uses [SEP] (separator) to mark sentence boundaries. A typical input looks like:

[CLS] the river bank overflowed [SEP] it caused major flooding [SEP]

[CLS] is the position whose final-layer representation is read by the NSP head (and, later, by classification fine-tuning heads). [SEP] simply tells the model where one sentence ends and the next begins.

Three summed embeddings. Each input position gets three embeddings that are summed before entering the encoder stack:

• Token embedding — looked up from the WordPiece vocabulary.
• Segment embedding — one of two learned vectors (E_A or E_B) marking which sentence the token belongs to. Used by NSP.
• Position embedding — a learned vector per absolute position (0 to 511). Self-attention is permutation-invariant on its own, so position information has to be added explicitly.

The sum at each position is what the encoder actually sees. The encoder itself — twelve stacked self-attention + feed-forward blocks for bert-base, twenty-four for bert-large — is exactly the encoder half of the original Transformer. The mechanics of those blocks live in that paper — and we open them up in Inside a transformer block; here we focus on what BERT does with the encoder, not what’s inside it. The shape of one block is worth seeing, though — it fixes which design choices BERT actually made (the ones that later models would change):

BERT uses the original encoder layer unchanged: full bidirectional attention, post-norm LayerNorm, and a non-gated GELU feed-forward. Modern blocks have since moved most of these — pre-norm, RMSNorm, gated MLPs (GeGLU/SwiGLU), and cheaper attention — but BERT is the baseline they’re measured against. (The diagram shows the arrangement; for how each box works, that’s the transformer-block article.)

Where do the token embeddings come from? From scratch, jointly with the rest of the model. The token embedding table is just another set of learnable parameters, initialized randomly and updated by backprop during pretraining. BERT does not load word2vec or GloVe at the input, and neither does GPT, Llama, or any modern transformer LM, for two reasons. First, the keys don’t line up — word2vec gives one vector per whitespace word; BERT indexes by WordPiece subword tokens (##ization), and GPT/Llama use BPE pieces, so the vocabularies are incompatible. Second, joint optimization wins — a frozen word2vec table is locally optimal for predicting context with a tiny shallow network, not necessarily the best input for a 24-layer transformer. Letting the embeddings co-adapt with the encoder yields an input representation tuned for the layers that actually consume it. (Pre-2018 it was common to initialize an LSTM’s first layer with word2vec and fine-tune from there. After BERT, basically no foundation model does that.)

From word2vec to context-aware embeddings

The intro promised that BERT’s output is a “context-aware embedding.” Now that we’ve seen the input format, here’s the mechanism behind that phrase — what actually changes between the vectors that enter the encoder and the vectors that come out.

The input side is still a lookup. The token embedding for bank is one fixed vector, pulled from a 30k-row table, identical in “river bank” and “bank account.” The segment and position embeddings get added on top, but those don’t depend on the neighbouring words either — position 3 is position 3 regardless of what’s sitting there. So the vector that enters the encoder stack is, in spirit, the same kind of object word2vec produced back in 2013: a static embedding — one vector per token, context-free. And it isn’t just BERT — every modern transformer LM, GPT through Claude through Llama, starts the same way: layer 0 is a static token-embedding table, structurally identical to word2vec’s E (just much wider). The transformer stack above is what turns those static vectors into context-aware ones.

Everything interesting happens inside the stack. Each of bert-base’s twelve layers is a round of self-attention: every token’s vector is rewritten as a weighted blend of every other token’s vector in the sequence, with the weights set by how relevant each pair is. The position embeddings supply word order — so “‘Squatch eats pizza” and “pizza eats ‘Squatch” don’t collapse to the same thing — and self-attention does the mixing. By the top of the stack the vector at the bank position has folded in river and flooded from its neighbours; it is no longer the dictionary entry for “bank,” it’s “bank, in this sentence.” Run the encoder on “the bank approved my loan” and the same input vector comes out somewhere completely different.

That output — one vector per token, each a function of the whole sentence — is a contextual (or context-aware, or contextualized) embedding. Static in, contextual out. word2vec showed that dense vectors are a powerful representation; BERT made the vector depend on context.

This is why BERT-family encoders are still everywhere even though the headline models today are decoders (more on that in the last section). Anywhere a system needs to compare pieces of text — semantic search, retrieval-augmented generation, deduplication, clustering, “find similar tickets” — you run the text through an encoder, pool the per-token vectors into one vector per sentence or passage, and measure cosine similarity. Passages about the same thing land near each other in that space; unrelated ones don’t. word2vec could already do a crude version of this for single words; BERT extended it to whole sentences, and to the meaning a word takes on from the company it keeps. (This is exactly the pipeline the RAG article picks up.)

The same “embed and compare” idea also powers evaluation metrics like BERTScore: embed a generated text and a reference with BERT, match their token vectors, and score how close they are in meaning — so “the meeting was postponed until Friday” and “they delayed it to the end of the week” rate as similar even though they share almost no words, which a word-overlap metric like ROUGE misses. Note what BERT is doing here: it is the judge, not what’s being judged — still an understanding task (measure similarity), never generation, even when the thing it’s scoring was generated by some other model.

One thing that paragraph glosses over: mean-pooling the per-token vectors out of a pretrained BERT doesn’t actually give you good sentence embeddings on its own. The model was trained to predict masked tokens, not to make similar sentences land near each other, and out of the box its pooled vectors are surprisingly weak for similarity tasks — the original Sentence-BERT paper (Reimers & Gurevych, 2019) showed they were sometimes worse than averaged GloVe. Their fix is the recipe almost every sentence embedding you’ll actually use is built on: fine-tune BERT in a Siamese setup — feed two sentences through the same encoder, pool each into one vector, and train with a contrastive loss so paraphrases end up close and unrelated pairs end up far apart. Models like all-MiniLM-L6-v2, mpnet-base, and the embedding endpoints from OpenAI and Cohere are all descendants of that idea. The BERT encoder produces the contextual representations; Sentence-BERT-style fine-tuning is what makes them comparable across whole sentences.

You’ll see it concretely in the next section: mask France in “the capital of France is Paris” and BERT’s top guesses are brittany, algeria, reunion — French regions and territories, not random countries — because the vector at that slot has already absorbed Paris, capital, and is from both sides before the MLM head ever reads it. The head is just reading a context-aware embedding.

One bit of lineage: BERT wasn’t the first contextual-embedding model — ELMo (early 2018) got there first using a bidirectional LSTM, and the shallow-concatenation limits of that approach are exactly what we walked through in the first section. BERT’s contribution was producing these embeddings with a deep bidirectional transformer — and finding the training objective that makes such a thing trainable at all. That objective is MLM, which is where we go next.

Masked Language Modeling

The architecture supports bidirectional context. The standard objective doesn’t. MLM is the swap.

For each training example, BERT picks 15% of the tokens at random and “corrupts” them — most often by replacing with [MASK]. The encoder runs over the corrupted input. A small classifier head reads the final-layer representation at each corrupted position and predicts the original token from the 30k-piece vocabulary. More precisely: at each masked position the model predicts a distribution over the whole vocabulary; we know which token was actually there before we masked it; and the loss measures how far the prediction was from that target. Formally that is cross-entropy, summed over the corrupted positions only — uncorrupted positions don’t contribute to the loss.

A quick disambiguation, because the word “mask” is overloaded and these two get conflated constantly: this [MASK] — corrupting input tokens to manufacture a fill-in-the-blank task — is not the same thing as the attention mask from the encoder-vs-decoder section above. That one hides positions from attention (the causal triangular mask that makes a decoder), and it acts on the attention scores inside the block; this one hides words from the input so the model has something to predict, and it acts on the input tokens before the model even runs. Different mechanisms, different jobs — and they’re not even paired: BERT uses this input [MASK] (MLM) while, being bidirectional, having no causal attention mask at all; GPT is the reverse (causal mask, no [MASK] token). The only thing the two share is the word.

Why this teaches anything useful is the whole bet behind BERT: there’s no way to fill those blanks reliably without actually understanding the sentence — its syntax, its semantics, the facts it refers to. A model that can pass this fill-in-the-blank test at scale has had to build that understanding into its representations, and that understanding is exactly what a downstream task like classification inherits when you fine-tune on top of it. The cloze test is never the real goal; it’s a cheap, label-free way to force the encoder to learn language.

If “predict a hidden word from its context” sounds familiar, it should: it’s the same bet word2vec’s CBOW objective made back in 2013 — predict the center word from the words around it. MLM is that idea with the limits removed. CBOW saw a tiny symmetric window (a handful of neighbours, order thrown away) and ran one shallow projection over their average; MLM sees the whole sentence, keeps word order via position embeddings, and the “context” each prediction leans on is the output of a deep stack of self-attention layers, not a bag of nearby words. And the payoff is different: CBOW’s reason for existing is the static lookup table that falls out of training; MLM’s is the contextual per-token vectors the encoder produces along the way. That makes them different kinds of artifact, and it’s the distinction worth holding onto: word2vec hands you a table you look up — one fixed vector per word, kept after the network is thrown away — while BERT hands you a model you run — feed it a sentence and it computes a fresh vector for every token, with no word→vector dictionary to keep around. word2vec learns words; BERT learns to read. (The BERT paper doesn’t credit word2vec for this — it calls MLM a “Cloze task,” after a 1953 reading-comprehension test. Same fill-in-the-blank shape, much older name.)

That sounds simple, and the basic shape is. But two design choices in the corruption procedure matter a lot, and both look strange the first time you see them.

Why 15%?

Empirical. The original BERT paper tried other rates and 15% worked best.

The two failure modes bracket the right answer. Too low (say 5%) and most positions in any given example contribute zero gradient — training is wasteful, the model only learns from a small fraction of each batch. Too high (say 50%) and the model loses too much context to make confident predictions — half the input is [MASK], so the surrounding evidence the model is supposed to lean on is itself missing. 15% is a sweet spot: enough signal per example, but the unmasked 85% still carries enough context to disambiguate.

The 80/10/10 split, and why it exists

Of the 15% selected tokens, BERT doesn’t always replace them with [MASK]. The actual rule is:

• 80% are replaced with [MASK]
• 10% are replaced with a random token from the vocabulary
• 10% are kept as-is (no change to the input — but the model is still asked to predict the original at this position)

The 80% case is what you’d expect. The 10% + 10% looks weird until you think about what happens at fine-tune and inference time: [MASK] never appears. Downstream tasks feed clean text into the same encoder. If the model had only ever seen [MASK] at training time, it would learn a representation specialized to that symbol — at every other position it would essentially pass the input through, since it never had to predict anything there. That representation would be useless for fine-tuning.

The two extra cases break that lazy strategy:

• 10% random token. The model sees a wrong token in a position and has to predict what the right one was. This teaches the model that an input token is a hint, not a guarantee — every position needs to be evaluated against context, not trusted.
• 10% unchanged. The model sees the original token but is still being scored for predicting the original. Crucially, the model doesn’t know which positions are being scored. So it has to produce a useful prediction at every position regardless of what symbol is sitting there.

The combined effect is that the encoder learns to produce meaningful contextualized representations at every token, not just at [MASK] positions. That’s what makes the same encoder usable for downstream tasks where every input is clean text.

What this looks like in code

Loading a pretrained BERT and doing the masking yourself takes about thirty lines. The script below is the same one we used to generate the predictions for the demo further down — for each sentence it walks through every position, replaces it with [MASK], runs the encoder, and reads the top-5 vocabulary predictions from the MLM head.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

MODEL_NAME = "bert-base-uncased"
TOP_K = 5

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForMaskedLM.from_pretrained(MODEL_NAME).eval()
mask_id = tokenizer.mask_token_id

text = "The capital of France is Paris."
encoded = tokenizer(text, return_tensors="pt")
input_ids = encoded["input_ids"][0]

# Walk every position (skipping [CLS] and [SEP]).
for i in range(1, input_ids.size(0) - 1):
    masked = input_ids.clone()
    masked[i] = mask_id
    with torch.no_grad():
        logits = model(masked.unsqueeze(0)).logits[0, i]
    probs = torch.softmax(logits, dim=-1)
    top_probs, top_ids = probs.topk(TOP_K)

    original = tokenizer.convert_ids_to_tokens(int(input_ids[i]))
    print(f"\n{original!r}:")
    for tid, p in zip(top_ids, top_probs):
        token = tokenizer.convert_ids_to_tokens(int(tid))
        print(f"  {token:>15}  {float(p):.4f}")

AutoModelForMaskedLM loads the encoder plus the MLM prediction head (a linear layer + GELU + LayerNorm + linear-to-vocab, with the final projection weight tied to the input token embeddings). The forward pass returns logits of shape (batch, seq_len, vocab_size); we softmax and read the top-K at the position we masked.

Try it

The widget below is built from the output of that script run over six sentences. Pick a sentence, click any token, see the top-5 predictions BERT produces when that token is replaced with [MASK].

A few things worth looking for:

• In “The capital of France is Paris,” mask France and the top alternatives are brittany, algeria, reunion — all French regions or territories. Mask Paris and you get lille, lyon, marseille, tours — other major French cities. BERT didn’t memorize “France’s capital is Paris” as a fact; it learned a contextual cluster — the slot accepts “things that play this role in French geography.”
• In “I went to the store yesterday,” mask went and was is the strongest competitor at ~1%. The presence of yesterday later in the sentence pushes the prediction toward past tense, even though “go” would be syntactically valid.
• In “The river bank overflowed after the storm,” mask bank and predictions include water-related and structural terms — exactly the disambiguation we set up in section 1. The bidirectional encoder uses overflowed to constrain what fits at bank.
• For frequent function words (the, of, is), BERT is essentially certain of the original token (>99%). For content words, the distribution is wider — and looking at the runners-up tells you what BERT thinks the local context allows.

Next Sentence Prediction

MLM teaches BERT what individual tokens mean in context, and how words constrain each other within a sentence. But many downstream tasks operate at a higher level than that — question answering, natural language inference, paraphrase detection, retrieval. All of them need the model to understand relationships between sentences, not just within them.

NSP is the second objective the original BERT used to inject that signal. Same encoder, same forward pass, second head, second loss.

The training task

For each training example BERT samples two sentences A and B from the corpus:

• 50% of the time — B is the actual next sentence after A in the source document.
• 50% of the time — B is a random sentence sampled from anywhere in the corpus.

The two are packed into a single input as [CLS] A [SEP] B [SEP] (the format we showed in the input section), with segment embedding E_A on every token of A and E_B on every token of B. The encoder runs once. A small 2-class classifier reads the final-layer representation at the [CLS] position and predicts one of {IsNext, NotNext}. Loss is cross-entropy against the true label.

The diagram makes the asymmetry worth pausing on: the encoder still produces a vector for every position — all 512 of them — but NSP reads only the first one. The [CLS] vector is routed through a small feed-forward layer and a softmax into the two-class {IsNext, NotNext} probability; the other 511 token vectors are simply ignored by this head (they’re what the MLM head reads instead). That’s the whole reason [CLS] exists as a dedicated slot: bidirectional attention lets it pull in information from the entire pair, so a single position can carry the pair-level summary the classifier needs — and the same slot is what classification heads reuse at fine-tuning time.

The [CLS] choice is deliberate. Because attention is bidirectional, the [CLS] representation can in principle aggregate information from every token in both sentences. The pretraining loss pressures the encoder to put a summary of the pair’s relationship into that one position, which is exactly what fine-tuning heads will later read for sentence-level classification tasks.

What this looks like in code

import torch
from transformers import AutoModelForNextSentencePrediction, AutoTokenizer

MODEL_NAME = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForNextSentencePrediction.from_pretrained(MODEL_NAME).eval()

a = "Paris is the capital of France."
b = "It has a population of about two million people."

encoded = tokenizer(a, b, return_tensors="pt")
with torch.no_grad():
    logits = model(**encoded).logits[0]

# Class 0 = IsNext, class 1 = NotNext (HuggingFace convention).
probs = torch.softmax(logits, dim=-1)
print(f"P(IsNext) = {float(probs[0]):.4f}")

tokenizer(a, b, ...) packs both sentences into the [CLS] A [SEP] B [SEP] format and produces the right token_type_ids for the segment embeddings — you don’t have to assemble it manually.

Try it

The widget below scores six curated sentence pairs. Some are real continuations, some are completely unrelated, and one is a deliberate edge case (we’ll come back to it).

The clearly-paired and clearly-unrelated cases come out as you’d expect — 1.000 and 0.000 (or near it). The interesting pair is the third: “Paris is the capital of France.” / “France is a country in western Europe.” These two sentences are not adjacent in any document — they’re an arbitrary topical pairing — but BERT scores them at 1.000 IsNext anyway. Why?

What NSP actually learns (the RoBERTa caveat)

The negative samples in NSP training are random sentences pulled from anywhere in the corpus. Random sentences are almost always about a different topic than the source sentence. The model can solve the task easily by just asking “do these two sentences share a topic?” — without needing to learn anything about discourse structure, sentence ordering, or anaphora.

That’s exactly what our edge case exposes. “Paris is the capital of France” and “France is a country in western Europe” share heavy topical overlap (Paris, France, Europe), so BERT’s learned topic-matching heuristic fires IsNext. It hasn’t actually checked whether one would naturally follow the other.

RoBERTa (2019) tested removing NSP entirely. They trained an otherwise-identical model on MLM only and found it matched or beat the original on every downstream benchmark. ALBERT replaced NSP with Sentence Order Prediction (SOP) — given two adjacent sentences, predict which came first. SOP can’t be solved by topic-matching (both sentences are from the same passage), and it added back the discourse signal NSP was supposed to provide.

So NSP was a real idea pointed in the right direction, but the training procedure made it solvable by a shortcut. Most modern BERT-family models drop it entirely or replace it with a harder sentence-pair objective. We include it here because it was part of the original BERT release, the [CLS] token and the sentence-pair input format come directly from it, and the failure mode it illustrates — “the model solved the task, just not the way we hoped” — is one you’ll meet again every time you design a self-supervised objective.

Pretrain, then fine-tune

MLM and NSP aren’t ends in themselves. Nobody actually wants a model that fills in [MASK]. The point is what the encoder weights look like after pretraining: they encode a general-purpose understanding of English, distilled from billions of tokens of unlabeled text.

The original BERT was pretrained on Wikipedia (~2.5B words) plus BookCorpus (~800M words) — a few days of TPU compute, expensive but a one-time cost. Then for any downstream task, the recipe is the same:

1. Load the pretrained encoder.
2. Attach a small task-specific head — usually a single linear layer.
3. Fine-tune the whole thing for a few epochs on the task’s labeled data, with a tiny learning rate (~2e-5 to 5e-5) so the encoder gets gently nudged rather than reset.

The encoder went into pretraining as random weights, came out understanding language, and now gets adapted — not retrained — for whatever task you have.

The four common head shapes cover most of NLP:

• Sentence classification (sentiment, NLI, spam detection, topic classification) — a linear layer reads the final-layer representation at the [CLS] position and projects to N classes. The same [CLS] slot that NSP read during pretraining is now the input to a different classifier.
• Sentence pair classification (paraphrase detection, entailment, similarity scoring) — same as above, but the input is [CLS] A [SEP] B [SEP] and the head reads [CLS]. This is structurally identical to NSP and is one of the reasons NSP was included in pretraining.
• Token classification (named-entity recognition, part-of-speech tagging) — a linear layer at every token position projects to per-token labels. Each token’s contextualized representation already encodes “what role does this word play here,” so a linear projection is enough.
• Question answering (SQuAD-style extractive QA) — two linear layers, one predicting the start token of the answer span and one predicting the end token. Input is [CLS] question [SEP] passage [SEP]; the heads read every position of the passage.

In every case, the head is a few thousand parameters on top of a hundred-million-parameter pretrained encoder. The heavy lifting was done once, during pretraining, and gets reused indefinitely.

This is the part that changed NLP economics. Before BERT, training a competitive model on a small task — say, classifying support tickets, with 5,000 labeled examples — meant either training a model from scratch on those 5,000 examples (which wouldn’t learn enough about language to generalize) or building elaborate task-specific architectures with hand-crafted features. After BERT, the same task became “fine-tune bert-base-uncased for three epochs and ship it,” with state-of-the-art results, on a single GPU, in under an hour. Every downstream task inherited the language understanding that pretraining had already produced.

The pretrain → fine-tune pattern is so dominant now that it’s easy to forget how recent it is. BERT was the proof. Everything since — RoBERTa, DeBERTa, T5, GPT-3 onwards, instruction-tuning, RLHF — builds on the same shape: do the expensive thing once on lots of unlabeled data, then adapt cheaply.

Where BERT lives today

The headline AI of 2025 is generative — ChatGPT, Claude, Gemini — and those are all decoder-only models in the GPT lineage, not encoder-only models in the BERT lineage. So a fair question is whether BERT still matters.

It does. Encoders and decoders are good at different things, and encoders never went away — they just stopped being the loud part. Anywhere a system needs to read text rather than produce it, a BERT-family model is probably involved:

• Search and retrieval — Google moved BERT into its search ranking pipeline in 2019 to better understand query intent. Modern semantic search and retrieval-augmented generation (RAG) systems use BERT-family encoders to produce dense embeddings for documents and queries; vector similarity over those embeddings is how the system finds relevant context to feed into a generative model.
• Classification at scale — sentiment analysis, content moderation, intent classification, spam detection, toxicity scoring. All of these are sentence-level classification tasks, and a fine-tuned bert-base or distilbert does them faster, cheaper, and often more accurately than a large generative model.
• Token-level tasks — named-entity recognition, part-of-speech tagging, span extraction. Bidirectional context per position is exactly the right shape for these.
• Embeddings as a building block — every “semantically similar,” “find duplicates,” or “cluster these documents” feature in modern apps is built on encoder embeddings.

The lineage filled out quickly: DistilBERT (2019) compressed BERT to 60% the size with 97% the performance via knowledge distillation; RoBERTa (2019) kept the architecture but trained longer on more data without NSP and beat the original; ALBERT (2019) shared parameters across layers; DeBERTa (2020) refined the attention mechanism with disentangled positional encodings; ELECTRA (2020) replaced MLM with a more sample-efficient objective (predict which tokens were swapped, not which were masked). Each one is a tweak to one of the design choices we walked through in this article — the masking procedure, the pretraining objective, the parameter budget, the position encoding.

Decoder-only models (GPT, Llama, Claude) overtook encoders for one specific axis: generating text. A causal mask is required for autoregressive generation, and once you’ve made that architectural commitment, you can scale and instruction-tune your way into a model that can also be coaxed into doing classification, retrieval, and everything else. That’s where the headline numbers and the venture capital went.

But encoders are still the right tool for understanding-only tasks where generation isn’t needed and per-call latency and cost matter. They run on CPUs, they fit on phones, they classify a million documents an hour for less than the cost of a single GPT-4 call. The pretrain → fine-tune recipe BERT introduced is also exactly what made the GPT-style instruction-tuned chat models possible — same shape, different objective.

If you’ve made it this far, you have the pieces to read the original paper directly: “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. The MLM and NSP setups described in section 3.1 will look familiar.

For a follow-up that builds on this article, see “BERT in the browser” — the same bert-base-uncased model loaded with Transformers.js and run client-side, so you can mask any sentence you type rather than picking from our six pre-computed examples.