Word embeddings — word2vec and vectors that mean something
A word embedding represents a word with a short vector of learned numbers. Word2vec is a useful way to study how those numbers acquire structure: its training task is simple enough to follow from a context window to a gradient update.
The familiar analogy king − man + woman ≈ queen illustrates a possible vector relationship. The diagram below uses hand-placed 2D points, not measured word2vec vectors. Its equal offsets are exact by construction; real analogy results are approximate and depend on the corpus, model, and comparison method.
Click step button to watch the construction step by step.
Schematic: hand-placed 2D points illustrate vector arithmetic.
In a trained model, the coordinates come from optimization rather than named semantic axes. Some relationships can appear as useful directions, but there is no universal “royalty” coordinate or offset that works for every word.
One-hot and TF-IDF representations give each vocabulary item a separate coordinate. Dense embeddings instead use a shared, smaller set of dimensions. Both remain useful representations; embeddings add a learned way to compare words beyond exact identity.
Word2vec, introduced in 2013, made large-scale word-vector training more efficient with shallow prediction models. It built on earlier distributed representations rather than introducing the idea of learned word vectors.
Skip-gram predicts context words from a center word; CBOW predicts a center word from its context. Words with similar usage provide similar prediction evidence, which can lead to related representations. This is a statistical training effect, not a guarantee that every semantic relation becomes a clean vector offset.
From hypothesis to geometry
The distributional hypothesis connects similar contexts with related meanings or grammatical roles. This is useful evidence, not a complete definition of meaning: antonyms such as hot and cold can also share many contexts.
A small embedding dimension constrains the model’s score matrix to have low rank, encouraging shared predictive structure. It does not force words into the same coordinates. In fact, replacing E by E A and E' by A⁻¹ E' for an invertible matrix A preserves every dot-product score while potentially changing distances and angles between input embeddings.
It helps to distinguish three uses of “embedding”:
Static word embeddings assign a word the same vector across contexts. Contextual embeddings, such as BERT’s hidden states, depend on surrounding tokens. Sentence or passage embeddings summarize a longer text for comparison or retrieval, often with additional contrastive training. These representations need not share the same training objective.
An autoregressive transformer also starts with a token-embedding lookup, usually over subword tokens. Its later hidden states depend on the preceding context. Pooling those states does not automatically produce a good retrieval embedding; that is a separate modeling choice.
This article follows static word embeddings from co-occurrence counts to skip-gram and CBOW, then explains what changes when representations become contextual.
The distributional hypothesis
Way back in 1957, the linguist J.R. Firth famously wrote, “You shall know a word by the company it keeps.” That’s the whole idea. Words that appear in similar contexts have similar meanings. To see why, consider three sentences with one word missing:
- The ___ barked at the postman.
- The ___ purred on my lap.
- The ___ flew south for winter.
You don’t need to know the missing words to know they refer to different kinds of animals. The context — the words around the blank — narrows down what fits. That’s one half of the idea: context predicts the word.
Looking across many sentences, words such as dog, puppy, and hound share useful context patterns. A prediction model can reuse parameters across those patterns. Shared context can indicate similarity, but also topical association or grammatical compatibility.
The question is how to compute it efficiently. The classic answer, going back to the early 1990s, was very large sparse co-occurrence matrices — latent semantic analysis and its kin. For the sentence “the cat sat on the mat” with a 2-word window (look up to 2 tokens left and right of each center word), the matrix looks like this:
the cat sat on mat
the [ 0, 1, 2, 1, 1 ]
cat [ 1, 0, 1, 1, 0 ]
sat [ 2, 1, 0, 1, 0 ]
on [ 1, 1, 1, 0, 1 ]
mat [ 1, 0, 0, 1, 0 ]Each M[i,j] counts occurrences of word j in word i’s window. We exclude the center position, not every occurrence of the same word. The diagonal happens to be zero in this example; repeated words within a window can produce nonzero diagonal entries.
The matrix is built by sliding the window across the corpus and tallying.
At each position with center word i, look at the words in its ±W window, and for each neighbour j in that window, increment M[i][j] by one. Scrub the widget from start to end and the matrix fills in cell by cell — one pass through the corpus and every co-occurrence count is recorded.
This is the first step of the count-then-factorise pipeline, and there are two more on top of it. The shape of a word’s vector changes substantially at each stage:
Stage 1 — raw counts. The matrix above, as it is. Each row is V numbers long (“V-long” for short), one slot for every word in the vocabulary — already the word’s vector, just a wildly oversized one. Each entry is literally “how many times did this specific vocab word appear in this word’s ±2 window across the corpus.” The row for cat is [1, 0, 1, 1, 0]. The toy corpus has only six tokens so the numbers are tiny, but the structure is what matters at scale.
Stage 2 — frequency adjustment. PMI compares an observed co-occurrence probability with the product of its marginals. Unobserved pairs have undefined or negative-infinite raw PMI, so a practical matrix often uses positive PMI, PPMI = max(PMI, 0), assigning zero to unobserved entries. Smoothing and weighting choices affect the result.
Stage 3 — compression. Truncated SVD approximates the weighted matrix using its largest singular values and corresponding singular vectors. A common word-vector table is U_d Σ_d. This is a low-rank approximation, not automatically PCA: PCA first centers the data, whereas this matrix factorization need not.
raw cat counts: [1, 0, 1, 1, 0]
PPMI (rounded): [0.182, 0, 0.405, 0.405, 0]
SVD representation: a row of U_d @ Sigma_dThe resulting d coordinates combine information from many context words. Individual coordinates need not have names such as “royalty” or “gender.”
Count-based factorization and word2vec both produce dense tables, but optimize different objectives. The shifted-PMI connection applies specifically to skip-gram with negative sampling under stated assumptions; it does not make every word2vec variant equivalent to SVD. We derive that connection in the negative-sampling article.
A word–context matrix has V×V possible entries, but sparse storage avoids allocating every zero. Sparse counting and factorization can still be costly for large corpora. Word2vec offers another route: update vectors directly from training examples without constructing that matrix.
word2vec
Word2vec learns its embedding table by predicting words in local windows. We will start with a full-softmax model to make the forward pass and gradients explicit, then distinguish the cheaper objectives used in practical implementations.
The lookup itself gives a word one vector regardless of its sentence. A downstream sequence model can still combine these vectors with word order and context. Static embeddings are therefore a starting representation, not a complete sentence model.
Word2vec provides two alternative training architectures:
- Skip-gram. Given a center word, predict the words in the small window around it. We feed the center word as input and predict each of the surrounding context-window words in turn — every
(center, context)pair is a separate training example, so the same center word is reused once per neighbour. - CBOW (continuous bag of words). The reverse: given the words in a window, predict the center word. We feed a set of context words in and predict the single word in the middle. This fill-in-the-blank shape — recover the missing word from its surroundings — is the one BERT’s masked language modeling later inherits and scales up.
Both are doing the same thing, just running the prediction in opposite directions. We’ll work through skip-gram end to end, then come back to CBOW.
The training data and one-hot inputs
The whole skip-gram pipeline splits cleanly into two stages with a sharp boundary: pre-processing turns raw text into a list of training examples, then training runs those examples through a neural network.
Let’s first look at pre-processing — turning the raw text into a long list of integer pairs. No neural network is involved yet:
- Tokenize the corpus — split the text into a list of word tokens (whitespace-separated, usually lowercased; word2vec uses word-level tokens, not subword pieces).
- Build the vocabulary — assign each distinct token an integer ID; that ID will later serve as the token’s index into one-hot vectors and as the row number into the embedding matrix. The total count is
V(the vocabulary size). - Extract
(center, context)pairs — slide a window over the ID stream and emit one training example per neighbour.
We choose a window of two tokens on either side for this example. The window size is a hyperparameter; practical implementations may also sample a smaller effective radius and subsample frequent words.
Using the sentence “the cat sat on the mat” as example, we set the window to ±2 and slide it word by word. With the window centered on sat, the neighbours are the, cat, on, the — this single position emits four pairs: (sat, the), (sat, cat), (sat, on), (sat, the). Step forward to on and the window finds cat, sat, the, mat — four more pairs. Step again, four more, and so on, until a whole corpus collapses into a long list of (center, context) pairs, generated entirely from the text itself, with no human ever labelling anything.
The resulting integer pairs fit in a short Python loop:
# Step 1: tokenize.
corpus = "the cat sat on the mat"
tokens = corpus.split()
# ['the', 'cat', 'sat', 'on', 'the', 'mat']
# Step 2: build vocabulary and convert tokens to integer IDs.
vocab = sorted(set(tokens)) # ['cat', 'mat', 'on', 'sat', 'the']
word2id = {w: i for i, w in enumerate(vocab)} # {'cat': 0, 'mat': 1, 'on': 2, 'sat': 3, 'the': 4}
V = len(vocab) # 5
ids = [word2id[w] for w in tokens] # [4, 0, 3, 2, 4, 1]
# Step 3: slide a ±2 window over the ID stream, emit (center, context) pairs.
window = 2
pairs = []
for i, c in enumerate(ids):
for j in range(max(0, i - window), min(len(ids), i + window + 1)):
if i != j:
pairs.append((c, ids[j]))
len(pairs) # 18 — exactly the (center_id, context_id) pairs the widget above emits.Pre-processing is essentially the same regardless of which word2vec variant you train next — tokenization, vocabulary, and windowing are identical. Only the format of the emitted examples differs: skip-gram packs them as pairs, while CBOW emits one context bag plus its center per window. Training is where the algorithm actually lives, and the rest of this section walks through it in detail.
Each pair is a training example
With pre-processing done, let’s look at how the training algorithm uses these pairs.
Each pair supplies an input word and a target label. The window around sat emits four examples with targets the, cat, on, the — three distinct words, with the occurring twice. In the full-softmax model, the target is used by the loss rather than the forward scoring step.
It’s similar to MNIST: each MNIST example pairs an image with its digit label, and here each skip-gram pair (c, t) pairs the center word c (input) with one context word t (target). For (sat, cat): feed sat in, get back a predicted distribution over the vocabulary, compare it against cat, take an SGD step. Then the next pair.
Before we can feed anything into the network, we need to represent the word as a vector of numbers.
In MNIST that step is mostly free — an image is already a grid of pixel intensities, so we just flatten it into a 784-number vector.
A word has no inherent numeric content, so we invented one back in the build-the-vocabulary step above — every word already has an integer index from 0 to V−1. To feed it into the network we expand that index into a one-hot V-vector — a vector V numbers long, all zeros except a single 1 at the word’s index.
For our running vocabulary of 5 words (V=5), the encoding looks like this:
"cat" → [1, 0, 0, 0, 0]
"mat" → [0, 1, 0, 0, 0]
"on" → [0, 0, 1, 0, 0]
"sat" → [0, 0, 0, 1, 0]
"the" → [0, 0, 0, 0, 1]Luckily, there’s a clever bit of linear algebra that lets us skip building the one-hot vector at all — multiplying a one-hot by a matrix is the same as picking out one row of that matrix (a lookup by the word’s integer ID). For our 5-word vocabulary, with sat at index 3 and some matrix M of shape (5, d):
one-hot for "sat" M (5 rows × 3 cols) result
[ 0 0 0 1 0 ] · [ row 0: 0.21 -0.43 0.15 ] = [ 0.33 -0.27 0.84 ]
[ row 1: 0.07 0.62 -0.31 ] (just row 3)
[ row 2: -0.55 0.18 0.40 ]
[ row 3: 0.33 -0.27 0.84 ]
[ row 4: -0.12 0.49 -0.06 ]Every term that touches a 0 from the one-hot vanishes, leaving only the row 3 contribution — so the answer is simply row 3 of M.
# dot product, column by column:
col 0: 0·0.21 + 0·0.07 + 0·(-0.55) + 1·0.33 + 0·(-0.12) = 0.33
col 1: 0·(-0.43) + 0·0.62 + 0·0.18 + 1·(-0.27) + 0·0.49 = -0.27
col 2: 0·0.15 + 0·(-0.31) + 0·0.40 + 1·0.84 + 0·(-0.06) = 0.84Mental model — what we’re trying to do
For each (center, target) pair, score the target against competing vocabulary words and adjust the two embedding tables to reduce the prediction loss.
Training uses an unnormalized dot product, not cosine similarity. Cosine divides that dot product by both vector norms and is commonly used to compare embeddings after training. Softmax turns the training scores into a distribution while preserving their ordering.
We’re going to have two trainable weight matrices — each between a pair of layers in the network — together giving every word in the vocabulary two d-dim vectors (one embedding per role):
Eof shape(V, d)— each row is one word’s input embedding, used when the word appears as the center of a training pair (the word being conditioned on).E'of shape(d, V)— each column is one word’s output embedding, used when the word appears as the target/context being predicted (the word being scored as a candidate).
E and E' are independent parameter matrices, not transposes of each other. The different layouts simply put input vectors in rows and output vectors in columns. A word has one trainable vector for each role.
Separate tables let a word play different input and output roles. After training, applications commonly use E; using output vectors or combining the two tables is also possible. The downstream task determines which representation is useful.
The widget below makes that step concrete — computing a similarity across dimensions between the center word and every word in the vocabulary, the interaction between E and E'. We focus on sat as the center (highlighted green in E). Step through to watch the matmul score it against every word in the vocabulary — one dot product per word, filling the scores vector entry by entry. (Stages 1 and 2 only — softmax and the loss come a couple of sections later.)
The score is v_c · v'_t. With output vectors stored as columns, scores = v_c @ E' computes all vocabulary scores at once. If they are stored as rows in Ep, the equivalent expression is scores = Ep @ v_c; no Python loop is needed in either layout.
Like a classifier’s output layer, each candidate has a weight vector that scores the hidden representation. The embedding dimensions are learned jointly and generally do not correspond to individually named semantic features.
The architecture
Once we have collected pairs, we’re essentially running a supervised labelled-prediction task — given a center word, predict which word from the vocabulary comes nearby — so the setup is similar to MNIST in key ways: one hidden layer, softmax over output classes, cross-entropy against the label. What differs is the scale (vocabulary size V here vs. 10 digit classes for MNIST), the input format (one-hot vs. dense real-valued pixels), and the goal (we want the trained embeddings, not the prediction itself).
Architecturally, word2vec (2013) is a two-layer feed-forward neural network — a feed-forward net of two fully-connected layers (the two weight matrices E and E'), with data flowing input → hidden → output and no loops. It’s the same family as the MNIST classifier, with one twist: the hidden layer is purely linear, so it learns word encodings rather than acting as a nonlinear feature extractor.
Concretely, the input is a V-dim one-hot for the center word; the hidden layer has d neurons (e.g. 300) and is purely linear (no bias, no nonlinearity); the output has V neurons with softmax across all V producing P(w | c) — the probability of each vocabulary word given the center. The two matrices E and E' we introduced in the previous section live between those layers: E is the input → hidden weight matrix (shape (V, d)), E' is the hidden → output one (shape (d, V)). These are the network’s only learned parameters; both start random, and after training only E ships as the final word-embedding table — E' is discarded.
Here is the full-softmax architecture in Keras, using the toy vocabulary above:
from tensorflow import keras
from tensorflow.keras import layers
V = len(vocab) # five words from the example above
d = 3
model = keras.Sequential([
keras.Input(shape=(1,), dtype='int32'), # integer ID of the center word
layers.Embedding(input_dim=V, output_dim=d), # E: shape (V, d), the lookup
layers.Reshape((d,)), # (1, d) → (d,)
layers.Dense(V, use_bias=False), # E': shape (d, V), the linear layer
layers.Softmax(), # softmax over V vocab scores
])
model.compile(optimizer='sgd', loss='sparse_categorical_crossentropy')Embedding(V, d) looks up input vectors; Dense(V, use_bias=False) stores the output weights. The input has shape (batch, 1), so the embedding output is (batch, 1, d). Reshape removes that one-token sequence axis before scoring. There is no nonlinear activation between the two trainable layers.
Softmax turns the V raw scores into a probability distribution over the vocabulary, and sparse_categorical_crossentropy is the standard classification loss on top of it — the same softmax-plus-cross-entropy MNIST uses, just with V vocabulary classes instead of 10 digits. We’ll walk through the loss and its gradient in detail in the sections that follow.
Embedding dimension d is a hyperparameter controlling capacity, memory, and computation. A few hundred dimensions is common, but the appropriate size depends on the corpus and task; it is not fixed by the vocabulary size alone.
What we’ve just described is the base-case forward pass — a softmax over the entire vocabulary on the output side.
The forward pass — lookup, scoring, softmax
Now that we’ve covered the high-level architecture — input one-hot, hidden lookup, output scores — let’s zoom in and walk through exactly what happens when a single training example flows through the network. For a center word c, the forward pass moves left-to-right in three stages: lookup, scoring, and softmax.
The widget back in the mental-model section already showed the first two stages in isolation — one-hot times matrix collapsing to a row read, then a row-times-matrix producing V scores. Here we name them, add softmax on top, and follow the numbers end to end.
Stage 1: input × E → hidden (the lookup). Mathematically this is the matrix multiplication one_hot(c) @ E, producing a d-dim hidden vector. The input is sparse — V−1 entries are zero — so almost every multiplication evaluates to zero, and the whole (V, d) matmul collapses to a single row read: v_c = E[c].
Stage 2: hidden × E’ → vocabulary scores. scores = v_c @ E' computes one dot product per output word, costing O(Vd). The backward pass can give gradients to every output column and to the selected input row.
The matmul never looks at the target word. It uses only v_c = E[sat] and scores sat against the entire vocabulary, producing all V scores at once. So those exact five scores — cat -0.19, mat 0.26, on 0.39, sat -0.07, the -0.45 — are identical for every training pair that shares the center sat: (sat, cat), (sat, on), and (sat, the) all run the same matmul and land on the same five numbers. The target word enters only later, at the loss; the forward pass never sees it.
Stage 3: softmax → probabilities. The V scores — called logits — are arbitrary real numbers: they could be negative, unbounded, not summing to anything in particular. Softmax turns them into P(w | c) — V non-negative numbers that sum to 1, the model’s predicted probability that word w is in the context of c.
Using the same 5-word vocab, the widget below walks all three stages. Pick a (center, context) training pair, then hit step to fill the scores vector one dot product at a time; once all V scores are in, softmax turns them into probabilities.
In this hand-chosen numerical example, on has the highest score, approximately 0.39. The numbers illustrate the calculation; they are not evidence that this tiny model has learned the sentence.
In skip-gram these scores are the dot products v_c · E'[:, w] for every vocab word w, and the resulting probabilities are P(w | c). Pulling out just the softmax step from the widget above, the five scores [-0.19, 0.26, 0.39, -0.07, -0.45] turn into a proper probability distribution that sums to 1:
Softmax exponentiates scores and divides by their sum. Implementations first subtract the largest score for numerical stability; this leaves the probabilities unchanged. The output sums to one, but normalization alone does not guarantee well-calibrated predictions.
The loss function
The loss for one training example with target word t is the negative log-probability the model assigned to that target:
loss = −log P(t | c)That gives us a single number per training pair (c, t) — small when the softmax has piled probability onto the true target, large when it hasn’t. This is cross-entropy with a one-hot label — the same loss MNIST uses, and the gradient through softmax is derived there.
Concretely, for the pair (sat, cat) using the scores and probabilities from above:
cat mat on sat the
scores = [-0.19, 0.26, 0.39, -0.07, -0.45]
probabilities = [ 0.16, 0.25, 0.29, 0.18, 0.12]
target = cat
P(cat | sat) = 0.16
loss = −log(0.16) ≈ 1.83
# what-ifs — how the loss responds to different P(cat):
P(cat) = 0.90 → loss = −log(0.90) ≈ 0.11 (good prediction)
P(cat) = 0.01 → loss = −log(0.01) ≈ 4.6 (bad prediction)The widget below carries the five softmax probabilities over from above and plots them against the −log curve. Click a different word to designate it as the target — the marker slides along the curve, and you can see directly how a target the model already favours costs almost nothing while a target it underweights pays a sharp price.
The loss grows as the target probability approaches zero. After differentiating through softmax, however, the gradient with respect to a logit is P(w) − 1[w=t], bounded between −1 and 1. Parameter gradients also depend on the vectors involved.
A uniform distribution over five words assigns probability 0.20 and loss log(5) ≈ 1.61 to any target. Our example gives cat about 0.16, so its loss is higher. Uniform output is a useful reference, not a property of every random initialization.
The gradient
The gradient is similar to MNIST: the gradient of the loss with respect to each logit is P(w) − 𝟙[w == t] — predicted probability minus the one-hot target. The target word’s gradient is P(t) − 1 (negative — push its score up); every other word’s is P(w) (positive — push its score down, proportional to how much probability it currently has). Words the model already correctly thinks are unlikely barely move; words it’s wrong about get the most signal.
The widget below makes that subtraction concrete on the same five probabilities from above. Click a different target to see the gradient row redraw — one tall blue bar pulling the target’s score up, four short red bars pushing the others’ down.
Backprop carries those score-gradients into E' and E by the chain rule:
scores = v_c @ E' (the forward step we're differentiating)
∂loss / ∂E'[:, w] = ( P(w) − 𝟙[w == t] ) · v_c ← gradient on column w of E'
∂loss / ∂v_c = E' @ ( P − one_hot_t ) ← gradient into the hidden vector
∂loss / ∂E[c] = ∂loss / ∂v_c ← because v_c = E[c]Two consequences worth keeping in mind, both from the one-hot input:
The lookup gives a gradient only to row E[c]; other input rows have zero gradient for this example. Full softmax involves every column of E'. These gradients describe dot-product scores; they do not guarantee that every pairwise distance changes in the direction suggested by a “pull/push” metaphor.
Those gradients are then used to take a gradient descent step: apply them to the weights as E -= lr × ∂L/∂E and E' -= lr × ∂L/∂E'. This is the optimiser’s job, and the choice of optimiser (vanilla SGD, momentum, Adam, RMSprop, AdaGrad…) only matters at this step — they all consume the same gradients but use them differently.
Mini-batches and epochs
The walkthrough above processed pairs one at a time. Most NN training generalises that into mini-batch SGD — group examples into batches of B and process a whole batch in one forward + backward pass, exactly the same mini-batch SGD as MNIST, just averaged across B examples per step. That’s the recipe BERT, GPT, and pretty much every modern model use, and it’s what the bullets below describe.
The original word2vec implementation and Gensim use optimized CPU training with asynchronous worker updates. The full-softmax mini-batch description below is a general reference implementation, not a description of Gensim’s inner loop.
An epoch is a pass over the training corpus or prepared examples. In implementations that resample context windows or discard frequent tokens, successive epochs need not contain exactly the same word pairs.
Per-batch, the loop looks like this:
- Forward pass on a batch of B examples — each
(c, t)pair flows through the network. Vectorized, the lookup stacks B rows fromEinto a(B, d)matrix, the score step becomes a single(B, d) @ (d, V) → (B, V)matmul, and softmax runs row-wise. - Loss — compute cross-entropy loss per example (same formula as B=1), then average across the batch into a single scalar.
- Backward pass (backprop). Compute the gradients via the chain rule — pure calculus from the loss back to every weight, no weight updates yet. Output: gradient tensors
∂L/∂Eand∂L/∂E', the same shape as the weight tensors. - Gradient descent step — apply the gradients to the weights via the optimiser, exactly as in the single-pair case above. Repeat for the next batch.
Run that loop across many batches per epoch and a handful of epochs over the corpus, and the rows of E and E' settle into the geometry the loss prefers.
CBOW — and the bridge to BERT
CBOW (continuous bag of words) is word2vec’s second algorithm, run as a mirror image of skip-gram. Where skip-gram takes a center word and predicts a target word as its context, CBOW takes several words as context and predicts the center — the missing word.
CBOW and BERT’s masked language modeling both predict a word or token from surrounding context. Their representations differ: CBOW averages a small unordered context bag, while BERT builds position-sensitive contextual states with a transformer. BERT is not simply CBOW with a larger window.
Mechanically, CBOW differs from skip-gram by exactly one averaging step on the input; the rest of the training loop is identical. So most of the skip-gram sections above carry over unchanged — we focus here only on where CBOW differs.
Pre-processing — context bags instead of pairs
The pre-processing pipeline is the same as skip-gram: tokenize, build vocab, slide a window over the token stream. What changes is the shape of what gets emitted per window position:
Position centered on: Skip-gram emits (per position): CBOW emits (per position):
───────────────────── ────────────────────────────── ──────────────────────────
the (pos 0) (the, cat), (the, sat) ({cat, sat}, the)
cat (pos 1) (cat, the), (cat, sat), (cat, on) ({the, sat, on}, cat)
sat (pos 2) (sat, the), (sat, cat), ({the, cat, on, the}, sat)
(sat, on), (sat, the)
on (pos 3) (on, cat), (on, sat), ({cat, sat, the, mat}, on)
(on, the), (on, mat)
the (pos 4) (the, sat), (the, on), (the, mat) ({sat, on, mat}, the)
mat (pos 5) (mat, on), (mat, the) ({on, the}, mat)
───────────────────── ────────────────
total: 18 pairs total: 6 examplesSame corpus, same window — skip-gram produces 18 separate (center, neighbour) training pairs, CBOW produces 6 (context_bag, center) examples. Step through it in the widget:
The architecture
CBOW’s architecture is the skip-gram one with its ends flipped — the inputs are the context words, the output is the center word. Same two weight matrices E and E', same dimensionality d, same softmax over V. The only architectural difference is on the input side: skip-gram looks up one row of E (the center), CBOW looks up C rows (one per context word) and averages them into a single d-dim hidden vector h:
Skip-gram: h = E[c] (one row read)
CBOW: h = mean(E[c_1], E[c_2], ..., E[c_C]) (C rows read, then averaged)Visualized below — the C one-hot inputs, the C corresponding row lookups in E, and the averaging step that produces h:
shape: C × V
shape: V × d
shape: d
The averaging is what gives “bag of words” its name — word order inside the window is discarded, the context becomes a multiset. At each training step, the lookups happen on the current, in-flight values of E — the rows you read in the forward pass are the same rows you update in the backward pass, just like any neural-net SGD training. Once h is computed, everything downstream — scores = h @ E', softmax, cross-entropy loss, backprop, SGD on E and E', and the negative-sampling shortcut — is identical to skip-gram.
Training — three small differences from skip-gram
The forward pass, loss, gradient, mini-batching, and negative-sampling shortcut all carry over from skip-gram unchanged. Three differences worth keeping in mind:
CBOW makes one prediction per window, while skip-gram makes one per context occurrence, so CBOW often needs less output-layer work. With a mean context vector, each input occurrence receives grad_h / C; repeated word IDs must accumulate their contributions. Full softmax still updates all output columns, while negative sampling updates selected output vectors. Relative speed and embedding quality depend on the data and settings.
How word2vec is actually trained in practice — Gensim
Gensim’s Word2Vec handles vocabulary construction, context sampling, negative sampling, and vector lookup. Its sentences input is an iterable of tokenized sentences. This tiny example checks the API; it is far too small to learn useful semantic analogies.
from gensim.models import Word2Vec
sentences = [
"the king sat in the palace".split(),
"the queen sat in the palace".split(),
"the king and the queen ruled".split(),
]
model = Word2Vec(sentences, vector_size=50, window=2, min_count=1,
sg=1, negative=5, hs=0, workers=1, seed=42, epochs=20)
v_king = model.wv["king"]CPU training and memory access
With negative sampling, a single pair touches only a few embedding vectors. Many implementations store both tables by row; with the column layout used earlier, output access is E'[:, w]. The work per pair is approximately:
1. Read the input vector: E[center]
2. Read output vectors: target + k sampled negatives
3. Compute scores: k+1 dot products, each length d
4. Accumulate gradients: one input row and sampled output vectors
5. Apply the SGD update: repeated indices receive summed contributionsTwo float32 embedding tables require 2 × V × d × 4 bytes: about 2.4 GB for one million words and 300 dimensions, before vocabulary and optimizer overhead. Small, irregular updates can suit CPU implementations, but GPU performance depends on batching, memory layout, and implementation. There is no hardware-independent rule that word2vec must be faster on CPU.
Alternatives to full softmax
The explanatory network above scores every vocabulary word, costing O(Vd) per example. The Gensim example uses negative sampling instead. Two established ways to avoid the full vocabulary calculation are:
Negative sampling replaces vocabulary classification with sampled binary labels, costing O((k+1)d) for one positive and k negatives. It learns a different objective; it is not an unbiased shortcut to the full-softmax gradient. The next article derives the loss and gives a working update.
Hierarchical softmax gives words the leaves of a binary tree and assigns a vector to each internal node. A word’s probability multiplies the appropriate left/right branch probabilities along its path, using one dot product per node. A balanced tree gives paths of order log₂V; word2vec’s Huffman tree gives frequent words shorter paths. The leaf probabilities form a normalized distribution.
Hierarchical softmax remains available in Gensim, for example with hs=1, negative=0. The choice depends on training cost and the representations the task needs; it is not an obsolete algorithm that universally lost to negative sampling.
Where static embeddings break
word2vec produces static embeddings: one fixed vector per word, regardless of context. This is exactly the right shape for the distributional hypothesis as originally stated, but it has three failure modes that became increasingly visible as NLP moved to harder tasks.
Polysemy
Consider these two sentences:
- I deposited the cheque at the bank.
- We had a picnic on the river bank.
A static embedding gives bank one vector for both sentences. It mixes evidence from different usages, without identifying which sense is intended in this occurrence. That mixture is not necessarily an arithmetic average of separate sense vectors.
bank with two sense clustersglove-wiki-gigaword-300The widget compares bank with financial and river-related words in one pretrained model. These similarities reflect that model and corpus. A downstream model that also sees the surrounding words can still disambiguate the sentence; the static vector alone does not provide that distinction.
No syntax sensitivity
A bag-of-vectors representation throws away word order. The sentences:
- Dog bites man.
- Man bites dog.
have the same bag of words, but different token orders. Averaging their static vectors gives the same representation. Keeping the vectors in sequence preserves the order, which an RNN, convolution, or transformer can use.
Vocabulary coverage and adaptation
“Static” means context-independent, not permanently frozen. Word vectors can be fine-tuned or retrained. A plain word2vec lookup has no entry for an unseen word, so it needs an unknown-token policy or another representation for vocabulary growth.
fastText addresses part of the unseen-word problem by composing vectors from character n-grams. Its word representations remain context-independent, so handling an unseen spelling and resolving a word’s sense in a sentence are separate problems.
What comes next
ELMo used bidirectional language-model states to represent tokens in context; BERT uses a transformer encoder trained with masked-token prediction. To continue from the training objective here, read negative sampling and contrastive learning, which follows the distinction between word2vec’s sigmoid loss and CLIP’s softmax loss.