The static word-embedding family — word2vec, GloVe, fastText

The word2vec article covers one method in depth. But word2vec was one of three static word-embedding methods that mattered, and the other two — GloVe and fastText — are worth knowing not because they beat word2vec (they mostly tie it) but because each makes a different design decision visible.

They all produce the identical kind of object: a (V, d) lookup table, one fixed d-dimensional vector per vocabulary word, frozen after training. Same output, same downstream use, same blind spots. What differs is how the table gets filled:

• word2vec streams local (center, context) pairs out of the corpus and nudges the vectors one pair at a time with SGD.
• GloVe counts all co-occurrences into a global matrix first, then factorises it directly.
• fastText builds each word vector out of its character n-grams.

This article fills in what the word2vec article deliberately leaves out: the GloVe theory (why ratios of co-occurrence probabilities are the right target, and what the weighting function is for), fastText’s n-gram trick, and the two training knobs — window size and frequent-word subsampling — that shape what all three learn.

word2vec, in one paragraph

word2vec (Mikolov et al., 2013) learns embeddings by prediction. Slide a window over the corpus; for each center word, train a shallow network to predict its context words (skip-gram) or the reverse (CBOW). The full softmax over the vocabulary is too expensive, so training uses negative sampling — turn “predict the context” into “tell true context words from a handful of random impostors.” The vectors that make that classification work end up encoding co-occurrence structure, and Levy & Goldberg (2014) proved word2vec is implicitly factorising a shifted PMI matrix — the same object the other methods target explicitly. The word2vec article walks through all of this; the negative-sampling article covers the loss.

The one thing worth carrying forward: word2vec never builds a matrix. It sees the same word pair many times as it streams the corpus and takes an independent gradient step each time. That redundancy is exactly what GloVe reacts against.

GloVe: factorise the counts head-on

GloVe (Pennington, Socher & Manning, 2014) makes the opposite bet. Instead of streaming pairs, count everything first — build the full co-occurrence matrix X, where X_ij is how often word j appears in word i’s context — then fit the vectors to those aggregate counts in one pass over the nonzero cells. If two words co-occur 347 times, word2vec processes that event 347 times; GloVe records the number 347 once and fits to it once. This is the global in “Global Vectors”: global corpus statistics, not local windows.

But the real content of GloVe is what it fits to. Raw counts are the wrong target, and the paper’s central argument is why.

Ratios of co-occurrence probabilities

Let P(k | w) be the probability that word k shows up near word w. GloVe’s insight is that a single probability tells you little, but a ratio of two probabilities is sharply meaningful.

Take w = ice and w' = steam, and probe them with different words k:

The ratio does the discrimination that the raw numbers can’t. Words relevant to ice but not steam (solid) push the ratio well above 1; words relevant to steam but not ice (gas) push it well below 1; words relevant to both (water) or neither (fashion) sit near 1 and cancel out. Meaning lives in the ratios, not the probabilities.

So GloVe designs its objective so that differences between word vectors map to log-ratios of co-occurrence:

(w_i − w_j) · w_k  ≈  log ( P(k | i) / P(k | j) )

Work that requirement through — a vector difference should encode a ratio, and log(a/b) = log a − log b — and it collapses to a per-cell target that each pair must satisfy independently:

w_i · w_k + b_i + b_k  ≈  log X_ik

The bias terms b_i, b_k absorb each word’s overall frequency, so the dot product is free to carry the association rather than the raw count. This is exactly the factorisation form from the matrix-factorisation article: fit u_i · v_j + b_i + b_j to log X_ij. GloVe is a least-squares factorisation of the log-count matrix, motivated top to bottom by the ratio argument.

The weighting function

One problem remains: not all cells of X deserve equal trust. Very frequent pairs (the–of) would dominate a plain least-squares fit; very rare pairs (a single chance co-occurrence) are mostly noise; and unobserved pairs (X_ij = 0, the overwhelming majority) shouldn’t contribute at all — log 0 is undefined anyway. GloVe handles all three with a weighting function on the squared error:

f(x) = (x / x_max)^α   if x < x_max
     = 1               otherwise

with x_max = 100 and α = 0.75 as the paper’s defaults. Read it off: unobserved pairs (x = 0) get weight 0 and drop out; rare pairs get a small, sublinearly-growing weight so they contribute but don’t get over-trusted; frequent pairs are capped at weight 1 so the can’t drown out everything else. The full loss is

J = Σ_ij  f(X_ij) · ( w_i · w_j + b_i + b_j − log X_ij )²

a weighted sum of squared errors over the nonzero cells only, minimised by gradient descent. That weighting curve — zero at the origin, rising sublinearly, flat above x_max — is the piece of GloVe that has no analogue in word2vec, and it’s why GloVe emphasises the mid-frequency co-occurrences where most of the signal actually lives.

word2vec vs GloVe, side by side

In practice their vectors are hard to tell apart on downstream tasks; the interesting differences are mechanical.

fastText: words are bags of character n-grams

fastText (Bojanowski et al., 2017) keeps word2vec’s training procedure but changes what a “word vector” is. Instead of one atomic vector per word, a word is represented as the sum of its character n-gram vectors. The word where, with n = 3 and boundary markers, becomes:

<where>  →  <wh, whe, her, ere, re>, plus the whole-word token <where>

The vector for where is the sum of the vectors for those pieces. Two consequences fall out:

• Out-of-vocabulary words get real vectors. A word never seen in training still decomposes into n-grams that were seen, so it gets a composed vector instead of nothing. word2vec and GloVe simply have no entry for it.
• Morphology is shared for free. walk, walking, walked, walker overlap heavily in their n-grams, so their vectors are related by construction — no need to relearn the connection per word. This is why fastText shines on morphologically rich languages (Finnish, Turkish, German compounds) where word2vec fragments the vocabulary into thousands of barely-seen forms.

The catch: n-grams capture orthography, not meaning. bank has the same character n-grams whether it’s a riverbank or a financial institution, so fastText inherits the same polysemy problem as the others (see below). It fixes the spelling axis of the OOV problem, not the sense axis.

Two knobs that shape all three

Independent of which method you pick, two training choices change what the embeddings capture.

Window size: syntactic vs topical. How wide you set the context window changes the kind of similarity the vectors encode. Small windows (±1–2) make a word’s neighbours its immediate grammatical companions, so “similar” comes to mean substitutable — same part of speech, same syntactic slot (Boston near Denver, walking near running). Large windows (±5–10) reach across the whole sentence, so “similar” drifts toward topical — words about the same subject regardless of grammar (Boston near Red Sox near baseball). Neither is more correct; they answer different questions, and the window is how you choose.

Subsampling frequent words. In raw text, the, a, and of swamp everything, and they carry almost no distributional signal. word2vec’s fix is to discard frequent words probabilistically before forming training pairs. A word w with corpus frequency f(w) is dropped with probability

P(discard w) = 1 − sqrt( t / f(w) )

with the threshold t ≈ 10⁻⁵. Words below the threshold are essentially always kept; words far above it (the function words) are dropped most of the time. Two things improve at once: training gets faster (fewer near-useless updates), and — subtly — deleting frequent words widens the effective window, letting the remaining content words see each other across the gaps the stop-words used to fill. It’s a small formula with an outsized effect on embedding quality.

Same table, same ceiling

All three methods produce a (V, d) lookup table, and all three hit the same wall: one fixed vector per word. bank gets a single vector that averages its financial and geographic senses into something that’s neither; word order is discarded, so dog bites man and man bites dog are indistinguishable once you pool the vectors; and the table is frozen at training time, blind to any word coined afterward. GloVe’s ratios and fastText’s n-grams change how the table is built, not that it’s a static table — so they inherit the ceiling described in the word2vec article’s where static embeddings break section.

Getting past it means giving up “one vector per word” entirely and computing a fresh vector for each word as it appears in a sentence — which is what the contextual encoders do. That’s the jump to ELMo and BERT, and the start of the next rung in the series.

References

• Mikolov et al., 2013 — word2vec
• Pennington, Socher & Manning, 2014 — GloVe
• Bojanowski et al., 2017 — fastText / subword information
• Levy & Goldberg, 2014 — word2vec as implicit matrix factorisation