Understanding transformers: encoder, decoder, and the architectural split

The previous articles in this series covered RNNs and LSTM/GRU — sequence models that maintain a hidden state and process tokens one at a time. They work, but they have two structural problems. First, the hidden state has to compress everything seen so far into a fixed-size vector — long sequences lose information. Second, training is sequential by construction; you can’t parallelize across the time axis because each step depends on the previous.

The 2017 paper “Attention Is All You Need” replaced recurrence with attention — a mechanism where every token can directly look at every other token in the sequence. No bottleneck through hidden state, no sequential dependency at training time, much longer effective context. The architecture they introduced is the Transformer, and within a year it had split into three families — encoder-only (BERT), decoder-only (GPT), and encoder-decoder (T5) — that dominate NLP today. Those three labels are the field’s main way of characterizing any transformer: you classify a model by which stack(s) of the original architecture it kept. So a phrase like “encoder-only transformer” is not a generic description — it pins the model to one specific branch (BERT-style: bidirectional, understanding-focused), as opposed to decoder-only (GPT, generation) or encoder-decoder (T5, sequence-to-sequence).

This article walks through that split: where Transformers came from, what BERT and GPT each took from the original design, and why the choice between them is really a choice between two ways of using attention.

The original Transformer

The 2017 paper — Attention Is All You Need, from eight researchers at Google — targeted a specific problem: machine translation. Given a sentence in English, produce the equivalent sentence in French. One thing the title is easy to misread: the paper introduced the Transformer, not attention itself. The attention mechanism came from Bahdanau et al. (2015), as a helper bolted onto RNN seq2seq. What the Transformer added was the claim that attention alone suffices — keep it, throw the recurrence away, and build the whole model from self-attention (plus multi-head, positional encodings, and feed-forward blocks). So: Google introduced the architecture; attention the mechanism predates it. The architecture has two halves designed for that translation shape:

The encoder reads the whole source sentence at once. Every token in “I love you” can attend to every other token — bidirectional attention, no constraints. The output is a sequence of contextualized representations: one per input token.

The decoder generates the target sentence one token at a time. At each step, it does two things:

• Self-attention over the target tokens generated so far (“Je t”), with a causal mask — token N can only attend to tokens 1…N-1, never the future. This matters because at inference the model is generating left-to-right; it can’t peek at tokens it hasn’t produced yet.
• Cross-attention from the decoder back to the encoder output, letting each generated token consult the source sentence representation.

It’s worth being explicit about why the decoder needs both, because a natural objection is “if the encoder already passed its understanding over, why does the decoder attend at all?” The two attentions do different jobs. Cross-attention is how the encoder’s information is used — it isn’t injected automatically; the decoder has to attend to the encoder’s output to read it, so removing cross-attention would leave the encoder’s work unused. And the decoder also needs self-attention for something the encoder cannot help with: the encoder only ever saw the source (English), but the decoder is building the target (French) one token at a time — so it must attend to the target words it has already produced to keep agreement, word order, and fluency right. The encoder knows nothing about the target; the target does not even exist when the encoder runs. So: cross-attention pulls from the source, self-attention tracks the output as it is generated.

Two attention mechanisms, two stacks. This shape — read everything in one stack, generate one-by-one with reference back to the first stack — is what translation needs and what the original Transformer was built for.

Zoomed out to the full architecture, end to end, that is this — tokenizer and embeddings at the bottom, the positional signal added in, each stack repeated six times, and a final linear-plus-softmax projecting the decoder’s output to token probabilities:

The block-level detail — what each Add and LayerNorm is actually doing — is the subject of Inside a transformer block. For a book-length walkthrough of the encoder- and decoder-style variants, Sebastian Raschka’s Encoder- and Decoder-Style Transformers (ML Q and AI, Ch. 17) is a good companion.

Two kinds of attention masks

The fork that gave us BERT and GPT is really about the attention mask — which tokens each token is allowed to look at.

Mechanically, both are the same multi-head self-attention layer — the same query/key/value matrices, the same softmax, the same dot products. The only difference is whether you apply a triangular mask before the softmax. Set the masked positions to -infinity, the softmax pushes them to zero, and tokens can no longer attend across the mask.

So the fork between bidirectional and causal attention is one bit of architecture, but it has huge consequences:

• Bidirectional attention lets every token see the whole sequence in one pass. Great for understanding tasks (classification, NER, embeddings) where you want full context for each token. Cannot be used for autoregressive generation — the token at position N would see token N+1, which doesn’t exist yet.
• Causal attention restricts each token to seeing only its predecessors. Required for autoregressive generation: you can train the model to predict token N+1 from tokens 1…N at every position simultaneously, then at inference generate one token at a time using the same forward pass.

Once you’ve made that choice, you’ve decided most of what the model is good at — and which training objective fits it. The rule in one line: the objective follows from the mask — masked-token prediction (MLM) for the bidirectional encoder, next-token prediction for the causal decoder. Then BERT and GPT happened.

BERT: keep the encoder

BERT (2018) took the encoder stack from the Transformer and dropped the decoder entirely. No causal mask, no cross-attention, no autoregressive generation. Just bidirectional self-attention layers.

Original Transformer encoder:
[bidirectional self-attention] → [feed-forward] → repeat × N → output

BERT:
[bidirectional self-attention] → [feed-forward] → repeat × 12 → [CLS] vector

The pretraining objective, masked language modeling (MLM), fits the bidirectional shape: hide ~15% of input tokens, and train the model to predict them given the full surrounding context — left and right of each masked token. This is something a causal model fundamentally cannot do; it would be cheating.

For downstream tasks, BERT outputs a sequence of contextualized vectors — one per input token — plus a special [CLS] token that aggregates information from the whole sequence. Classification: take the [CLS] vector, run it through a small linear head. NER: take each token’s vector, run it through a per-token classifier. Embeddings: pool the token vectors into a single sentence vector.

What BERT can’t do: autoregressive generation. The bidirectional mask means it can’t be used for left-to-right generation in the standard way.

The BERT family today: BERT itself, RoBERTa (better-trained BERT), ALBERT (parameter-shared), DistilBERT (distilled smaller version), DeBERTa-v3 (disentangled attention + ELECTRA-style training), ModernBERT (2024 redesign). All encoder-only.

GPT: keep the decoder, drop the cross-attention

GPT (2018) took the decoder stack — but with one important modification.

The original Transformer decoder has two attention layers per block: causal self-attention over the target sequence, then cross-attention to the encoder. Without an encoder, the cross-attention has nothing to attend to. So GPT removed it:

Original Transformer decoder block:
[causal self-attention] → [cross-attention to encoder] → [feed-forward]

GPT decoder block:
[causal self-attention] → [feed-forward]

What’s left is causal self-attention plus a feed-forward layer — the same shape as a BERT encoder block, except for the attention mask. This is why people sometimes get confused by the naming: GPT really is “just the decoder stack,” but with cross-attention removed it looks structurally similar to BERT, differing mainly in the mask.

The pretraining objective, next-token prediction, fits the causal shape exactly: at every position, predict the next token given everything before. The causal mask makes this efficient — one forward pass over the whole sequence trains the model to predict every position simultaneously.

For downstream tasks, GPT generates text. Classification by GPT works in two ways:

• Generative: prompt the model with the input + task description (“Is this confidential? Yes or No: …”) and read the generated label as text.
• Discriminative head: take the final token’s hidden state, attach a linear classifier head — same shape as BERT’s [CLS] head, but using the last position rather than a special prefix token.

What GPT does well: text generation, dialogue, code, anything autoregressive. What it does less well architecturally: classification with clear lexical signal. The bidirectional context BERT gets for free is something a causal model has to compress as it reads.

The GPT family today: GPT-3/4/5, Llama 1/2/3, Qwen, Mistral, Gemma, DeepSeek, Phi. All decoder-only.

T5 and BART: keep both halves

A smaller third family kept the original encoder-decoder shape, where the encoder reads input bidirectionally and the decoder generates output autoregressively with cross-attention back to the encoder.

T5 (2019) recast every NLP task as text-to-text: classification, translation, summarization, even regression — all “input text → output text.” The pretraining objective is span corruption (a masked-language variant), and the architecture is the original Transformer with both stacks.

BART (2019) is the same shape, with a denoising autoencoder objective: corrupt the input (mask spans, shuffle sentences, etc.), then train the encoder-decoder to reconstruct it.

Whisper (OpenAI’s speech-to-text) follows the same pattern: encoder reads audio features bidirectionally, decoder generates the transcript autoregressively.

These are particularly good at structured generation tasks where you have a clear input-to-output mapping: translation, summarization, speech recognition. The bidirectional encoder gives full input understanding, the causal decoder generates the output, and cross-attention lets the decoder consult the encoded input at every generation step. This is the original Transformer’s shape, and for translation-shaped tasks it remains genuinely strong.

The encoder-decoder family today: T5, FLAN-T5, mT5, BART, mBART, Whisper, NLLB, Pegasus.

Cross-attention: the same head, two streams

The one-head toy computed queries, keys, and values from one sequence. Cross-attention changes their sources. The decoder’s current representations supply queries; the encoder’s output supplies keys and values:

Q = decoder_states @ Wq             # (n_target, d_k)
K = encoder_states @ Wk             # (n_source, d_k)
V = encoder_states @ Wv             # (n_source, d_v)
A = softmax(Q @ K.T / sqrt(d_k), axis=-1)
H = A @ V                           # (n_target, d_v)

The attention matrix is rectangular: (n_target, n_source). Each target position receives a mixture of source values. After combining heads and projecting to the decoder width, that contribution enters a residual addition to the decoder stream. It does not replace the encoder representations or append source tokens to the target sequence.

These weights are hand-constructed to illustrate possible alignment, not measured from a translation model. The diagram puts target queries in columns, unlike the row-query convention in the code. Each column distributes weight over source keys. A query can gather from several source positions (not from ne and pas), and several queries can gather from one source position (to, pet from caresser).

The decoder’s self-attention is causal over target positions. Its cross-attention can inspect the full encoded source, which is already available; padding may still be masked. This is the additional sublayer in the original encoder–decoder. A decoder-only model has no separate encoder stream to consult.

Three families, side by side

The naming is asymmetric and confusing — “decoder-only” implies “the part that decodes,” which suggests there must be an encoder somewhere. There isn’t, in GPT. The model processes its input prompt by running the prompt tokens through the same causal-attention stack that will later generate output tokens; there’s no separate input-encoding step. “Decoder-only” really means “we kept the decoder stack as the only stack, with cross-attention removed.”

Why the architectural fork matters in practice

The fork between bidirectional and causal attention has two practical consequences worth keeping front of mind.

For classification tasks, encoder models are doing more of the work architecturally. A BERT classifier on a 100M-parameter model can match or beat a 7B causal LLM with a discriminative head, because bidirectional attention lets every token see every other in one pass. The MLM pretraining objective is also closer to “given context, fill in” than next-token prediction is to “given text, what label.” Encoder classifiers reach high accuracy with two orders of magnitude fewer parameters and ~5–25 ms inference instead of 100–500 ms.

For generation tasks, decoder models are required, not just preferred. You cannot use bidirectional attention for autoregressive generation — the model would see the future tokens it’s supposed to predict. Whatever else you might want from a generative model, the causal mask is non-negotiable.

The encoder-decoder family wins when both halves are doing real work. Translation needs deep input understanding (encoder) AND fluent output generation (decoder) AND a way for each output token to consult the input (cross-attention). The 2017 architecture was built for exactly this shape, and T5 / Whisper / BART still hold up in their domains.

But general-purpose models converged on decoder-only — and not only because of scale. A decoder-only model does sequence-to-sequence without a separate encoder: put the source right in the prompt and generate the target, and ordinary causal self-attention lets the generated tokens attend back to it — you get the cross-attention benefit for free, in one stack. So one architecture, one objective (next-token), and the input simply lives in the context. That simplicity, plus the fact that any task (“translate this”, “summarize this”, “answer this”) becomes “describe it in the prompt and continue”, is why a new general-purpose model is decoder-only — even though dedicated encoder-decoders still win the narrow, translation-shaped tasks they were built for.

The 2024+ trend is decoder-only models taking over tasks they’re architecturally suboptimal for (classification, embedding, ranking) by sheer scale and instruction-tuning capability. It works, but at significant cost — Llama-Guard 3 (8B) doing what DistilBERT (66M) does well, at 100× the inference cost. The architectures haven’t gone away; the fashion has.

Summary

The 2017 Transformer paper introduced one architecture for translation: an encoder that reads input bidirectionally, a decoder that generates output autoregressively, and cross-attention bridging them. Within a year, the field split that architecture three ways:

• BERT kept just the encoder stack, gave us bidirectional context for understanding tasks, and trained with masked language modeling.
• GPT kept just the decoder stack (minus its cross-attention), kept the causal mask required for autoregressive generation, and trained with next-token prediction.
• T5 / BART kept both halves and the cross-attention, for tasks shaped like translation.

The single-bit architectural choice — bidirectional or causal attention — decides almost everything else about what the model is good at. Encoder for understanding, decoder for generation, both for sequence-to-sequence. Modern variants — DeBERTa-v3, Llama-3, Qwen-2.5, Whisper — sit on this same map.

When you pick a model in 2026, you’re really picking which half of the 2017 Transformer you want, plus a few decisions about scale, tokenizer, and training data. The split is the architecture.

Working notes — to be removed before publish.

• Continues the understanding-* series (adaline, linear-models, rnns-sequences-hidden-state, rnns-lstm-gru-sequence-recognition).
• Cross-link from this article: forward to fine-tuning articles (Vast/Kinetic/local-classifier), back to the RNN series, and to the MoE article (which uses transformers as building blocks).
• Cross-link to this article from: Article C’s “How the architectures differ” subsection — currently inline in fine-tuning-safety-classifier-workstation-gpu. Trim that section to a one-paragraph link once this article is up.