Gradient boosting, at its core, is built from residuals. Start with a constant. Compute what’s left over. Fit a small tree to that. Add a fraction of it to the running total. Repeat. Building that loop from scratch turns up the fact that the residuals are the negative gradient of the loss, which is where the “gradient” in gradient boosting comes from — each tree is a step in function space.

That framing has a seam in it. Look closely at what the algorithm actually does:

1. Compute the negative gradient at each point — the direction that would reduce the loss.
2. Fit a tree to those numbers by minimizing squared error.
3. Take a step in the direction of that tree.

Step 2 is where the seam is. The loss you actually care about might be log-loss, or a ranking objective, or Poisson deviance — but the tree is grown by squared error against the gradient, no matter what that loss is. The split criterion has nothing to do with your objective. The tree is fit to a linearization, and then you hope a small learning rate keeps you close enough for the linearization to hold.

XGBoost closes that seam. Instead of computing a gradient and then fitting a tree to it with an unrelated criterion, it writes the tree directly into the loss function, expands the loss to second order, and asks: given this tree structure, what leaf values minimize the actual regularized objective? That question has a closed-form answer. And once you have the answer, you can score any candidate tree structure — which gives you a split criterion derived from your objective rather than borrowed from regression.

Everything else — the regularization, the Hessian weighting, the split gain formula with the mysterious $\gamma$ in it — falls out of that one move.

The objective, with the tree inside it

At boosting round $t$, every training point already has a prediction $\hat{y}_i^{(t-1)}$ accumulated from the trees built so far. We want to add one more tree, $f_t$. The objective is:

$$\text{Obj}^{(t)} = \sum_{i=1}^{n} l\left(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)\right) + \Omega(f_t)$$

The first part is whatever loss you chose, evaluated at the updated prediction. The second part is new, and it’s the first thing that distinguishes XGBoost from classical gradient boosting — an explicit complexity penalty on the tree itself:

$$\Omega(f) = \gamma T + \frac{1}{2} \lambda \sum_{j=1}^{T} w_j^2$$

where $T$ is the number of leaves and $w_j$ is the value at leaf $j$. So $\gamma$ charges a fixed price per leaf, and $\lambda$ is L2 regularization on the leaf values — ridge regression, applied to the outputs of a tree.

In classical GBM, regularization is external: you cap the depth, you require a minimum sample count per leaf, you shrink with a learning rate. Those are constraints imposed on a procedure that doesn’t know about them. Here, $\Omega$ is inside the thing being minimized, so every decision the tree makes — every split, every leaf value — is made with the penalty already priced in.

The problem is that $\text{Obj}^{(t)}$ isn’t something you can optimize directly. $f_t$ is a tree: a discrete structure, not a parameter vector. You can’t take a derivative with respect to a tree.

Second order, then drop the constant

The trick is to stop treating $l$ as a black box and approximate it locally. Taylor-expand the loss around the current prediction $\hat{y}_i^{(t-1)}$, treating the new tree’s output $f_t(x_i)$ as the small displacement:

$$l\left(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)\right) \approx l\left(y_i, \hat{y}_i^{(t-1)}\right) + g_i f_t(x_i) + \frac{1}{2} h_i f_t(x_i)^2$$

where $g_i$ and $h_i$ are the first and second derivatives of the loss with respect to the current prediction:

$$g_i = \frac{\partial l(y_i, \hat{y}_i^{(t-1)})}{\partial \hat{y}_i^{(t-1)}}, \qquad h_i = \frac{\partial^2 l(y_i, \hat{y}_i^{(t-1)})}{\partial (\hat{y}_i^{(t-1)})^2}$$

Classical gradient boosting stops at first order — it uses $g_i$ and nothing else. XGBoost keeps the quadratic term. That’s the whole difference at this stage, and it’s worth being precise about what it means: first order tells you which way to go, second order tells you how far. Gradient descent versus Newton’s method, applied to functions instead of vectors.

Now the useful part. The term $l(y_i, \hat{y}_i^{(t-1)})$ is a constant — it’s the loss you already have, and no choice about $f_t$ can change it. Constants don’t affect argmins, so drop it:

$$\text{Obj}^{(t)} \approx \sum_{i=1}^{n} \left[ g_i f_t(x_i) + \frac{1}{2} h_i f_t(x_i)^2 \right] + \gamma T + \frac{1}{2} \lambda \sum_{j=1}^{T} w_j^2$$

Look at what just happened. The loss function is gone. Log-loss, squared error, Poisson, a custom ranking objective — whatever it was, it has been fully absorbed into two numbers per training point, $g_i$ and $h_i$. Everything downstream of this line is identical regardless of what you’re optimizing. This is why XGBoost can accept an arbitrary objective as a callback that returns a gradient and a Hessian: past this point, the tree-builder never needs to know what loss produced them.

Grouping by leaf

The sum above runs over training points, but a tree doesn’t have a parameter per point — it has one per leaf. So regroup.

A tree is two things: a structure $q$ that maps each point to a leaf index, and a vector $w$ of leaf values. Then $f_t(x) = w_{q(x)}$. Define the set of points landing in leaf $j$:

$$I_j = \{\, i \mid q(x_i) = j \,\}$$

Every point in $I_j$ gets the identical output $w_j$. So rewrite the sum over points as a sum over leaves, with an inner sum over the points in each leaf:

$$\text{Obj}^{(t)} \approx \sum_{j=1}^{T} \left[ \left( \sum_{i \in I_j} g_i \right) w_j + \frac{1}{2} \left( \sum_{i \in I_j} h_i + \lambda \right) w_j^2 \right] + \gamma T$$

The $\lambda$ slid inside the leaf sum because $\frac{1}{2}\lambda \sum_j w_j^2$ contributes exactly $\frac{1}{2}\lambda w_j^2$ to leaf $j$. Give the two inner sums names — they’re the only statistics that will ever matter again:

$$G_j = \sum_{i \in I_j} g_i, \qquad H_j = \sum_{i \in I_j} h_i$$

and the objective collapses to something small enough to read:

$$\text{Obj}^{(t)} = \sum_{j=1}^{T} \left[ G_j w_j + \frac{1}{2} (H_j + \lambda) w_j^2 \right] + \gamma T$$

$n$ training points have become $T$ leaves, each summarized by two numbers.

The closed form

Each leaf’s term is $G_j w_j + \frac{1}{2}(H_j + \lambda) w_j^2$ — a quadratic in a single scalar $w_j$, with no coupling to any other leaf. A quadratic $aw + \frac{1}{2}bw^2$ with $b > 0$ is minimized at $w^* = -a/b$, taking value $-\frac{1}{2}a^2/b$. No iteration, no search. Differentiate, set to zero, read off the answer:

$$w_j^* = -\frac{G_j}{H_j + \lambda}$$

Substitute back to get the value of the objective for a tree with structure $q$:

$$\text{Obj}^*(q) = -\frac{1}{2} \sum_{j=1}^{T} \frac{G_j^2}{H_j + \lambda} + \gamma T$$

This is the structure score — the punchline of the derivation. It is a number you can compute for any proposed tree shape, and it says how good that shape is under your actual objective, with regularization already accounted for. Lower is better. The $\gamma T$ term means every leaf must pay for itself.

Two things fall out of the leaf-value formula immediately. First, $\lambda$ sits in the denominator: it shrinks leaf values toward zero, and it does so most aggressively where $H_j$ is small — leaves supported by little curvature get pulled toward zero hardest, which is exactly the behavior you want from a regularizer. Second, notice what the denominator is not: it’s not the number of points in the leaf. It’s the sum of their Hessians.

The split gain

You still can’t enumerate every possible tree structure — there are exponentially many. So do what decision trees always do: grow greedily, one split at a time, and use the structure score to decide.

Take a leaf with statistics $(G, H)$ and consider splitting it into a left child $(G_L, H_L)$ and a right child $(G_R, H_R)$, where $G = G_L + G_R$ and $H = H_L + H_R$ — gradients and Hessians are sums, so children’s statistics always add up to the parent’s. The gain is the objective before minus the objective after:

$$\text{Gain} = \frac{1}{2} \left[ \frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L + G_R)^2}{H_L + H_R + \lambda} \right] - \gamma$$

The three fractions are the structure scores of the left child, the right child, and the parent they’d replace. The $-\gamma$ is there because the split turns one leaf into two, and each leaf costs $\gamma$.

This replaces Gini impurity and variance reduction. And unlike those, it isn’t a heuristic that happens to correlate with what you want — it’s the exact change in the regularized objective you’re minimizing, to second order.

The $\gamma$ term also gives you pruning for free. If the best available split has $\text{Gain} \leq 0$, the split makes the objective worse, and you don’t take it. Not because a depth cap forbade it — because the arithmetic says it isn’t worth it.

The widget below is a single boosting round on a binary classification problem, caught partway through training. The labels really flip at $x = 5.5$, but the trees built so far have put their boundary at $x = 7$ — confident out at both edges, wrong across the middle. Each point carries its margin from those trees, which is all it takes to give it a $g_i$ (plotted on the y-axis) and an $h_i$ (plotted as marker area — bigger dot, more curvature). Drag split to move the threshold; the dashed lines are the resulting leaf values $w_L^*$ and $w_R^*$, and the lower panel is the gain at every candidate threshold:

Things worth doing here:

• Turn up $\lambda$. Both leaf values shrink toward zero. The gain curve flattens too — with heavy L2, no split looks impressive, because no leaf is allowed to commit.
• Turn up $\gamma$. The whole gain curve slides down by exactly $\gamma$. Push it far enough and the entire curve drops below zero: every candidate split is now pruned, and the leaf stays a leaf. That’s $\gamma$ as a minimum admission price, visible as a rigid vertical shift.
• Watch the marker sizes. Points out at the edges have large margins, so $p_i$ is near 0 or 1, so $h_i = p_i(1-p_i)$ is nearly zero — they’re tiny dots. The model is already confident about them and they barely influence the leaf value, even though their gradients are not zero. The uncertain points in the middle are fat dots and dominate. First-order boosting cannot see this distinction.
• Hit “Snap to best” at a few $\lambda$ values. The optimal threshold moves. Regularization doesn’t just scale the outputs — it changes which split gets chosen.

What the Hessian actually buys

It’s worth grounding the abstraction in two concrete losses.

Squared error, $l = \frac{1}{2}(y_i - \hat{y}_i)^2$. Then $g_i = \hat{y}_i - y_i$ (the negative residual) and $h_i = 1$. So $H_j$ is just the count $|I_j|$, and:

$$w_j^* = -\frac{G_j}{H_j + \lambda} = \frac{\sum_{i \in I_j} (y_i - \hat{y}_i)}{|I_j| + \lambda}$$

which is the mean residual in the leaf, shrunk by $\lambda$. That’s classical gradient boosting — fit the tree to the residuals, predict their average — recovered as a special case, with ridge regularization on top. For squared error, the second-order machinery gains you nothing, because the second order is the whole function: the Taylor expansion of a quadratic is exact.

Log-loss, with $p_i = \sigma(\hat{y}_i)$. Then $g_i = p_i - y_i$ and $h_i = p_i(1 - p_i)$. Now the Hessian varies per point, and it means something specific: $p_i(1-p_i)$ is the variance of a Bernoulli — the model’s uncertainty at that point. A point the model has already nailed ($p_i \approx 0.99$, correct) has $h_i \approx 0.01$. A point it’s torn on ($p_i \approx 0.5$) has $h_i = 0.25$, twenty-five times the weight.

So $H_j + \lambda$ isn’t counting samples, it’s counting how much the loss actually curves in this leaf — which is why the same $\lambda$ behaves sensibly across leaves holding wildly different populations. And here the Taylor expansion is a genuine approximation rather than an identity, which is where the second-order term earns its keep: log-loss is not a quadratic, and a Newton step lands much closer to the minimum than a gradient step of the same nominal size.

The engineering on top

The derivation gives you a better split criterion. It doesn’t, by itself, make anything fast. XGBoost’s 2016 paper is titled “A Scalable Tree Boosting System”, and most of it is systems work. Four pieces matter.

Approximate split finding. The exact greedy algorithm sorts every feature and scans every candidate threshold — optimal, but it requires the sorted data to fit in memory and it doesn’t distribute. The approximate version proposes a limited set of candidate thresholds from feature quantiles and only evaluates those. Roughly $1/\epsilon$ candidates per feature instead of $n$.

The weighted quantile sketch. Which quantiles, though? Here the derivation pays off in an unexpected place. Complete the square on the objective:

$$\sum_{i=1}^{n} \frac{1}{2} h_i \left( f_t(x_i) - \left(-\frac{g_i}{h_i}\right) \right)^2 + \text{const}$$

This is exactly a weighted squared-error problem: each point has target $-g_i/h_i$ and weight $h_i$. So the candidate splits shouldn’t come from ordinary quantiles that treat every row alike — they should come from quantiles weighted by $h_i$, putting more candidates where the loss actually curves. Existing quantile sketches didn’t support weights with provable error bounds, so the paper introduces one that does. The regularizer told them which data structure to invent.

Sparsity-aware split finding. Real tabular data is full of holes: missing values, zeros, and the vast empty space left by one-hot encoding. Rather than imputing, XGBoost gives every split a default direction. It enumerates only the non-missing entries, tries sending all missing values left, then tries sending them all right, and keeps whichever scores better. Missing-ness becomes a learned parameter of each split rather than a preprocessing decision you make in the dark. The paper reports a 50× speedup on sparse data from this alone, because the scan cost drops to the number of present values.

Cache and disk. Pre-sorted feature blocks in compressed column format, so the sort happens once rather than per split. Prefetching gradient statistics into cache, because the row indices arrive in an order that defeats the prefetcher. Block compression and sharding across disks for datasets that exceed RAM. Unglamorous, and the reason it ran on hardware people actually had in 2016.

Two more regularizers ride along, both borrowed rather than derived: shrinkage (the learning rate $\eta$, scaling each tree’s contribution so later trees still have work to do) and column subsampling (each tree sees a random subset of features — lifted straight from random forests, and per the paper’s users, often better at preventing overfitting than row subsampling).

Where it lands in production

The theory says XGBoost should win on tabular data with sparse interactions and meaningful columns. What’s striking about where it actually shows up is how much narrower the pattern is than “tabular data.”

Click through the ten cases below — they’re taken from Andres Vourakis’s 10 Real-World Use Cases of XGBoost in Production (2026 edition), and I’ve added the “why trees” reading for each:

Four things recur across them.

Ranking is the biggest single bucket. Dropbox, Spotify, and Etsy are all ordering a list — search results, artists, ad creatives. Ranking is where boosted trees have been dominant since LambdaMART, and the reason is structural: the features are counts, scores, recencies, and match statistics, one per query–document pair. That’s a table, and the pairwise/listwise objective plugs into the $g_i$/$h_i$ interface like any other loss.

The constraint is often the model, not the accuracy. Instacart needs real-time predictions over hundreds of millions of items. Sophos ships a PDF malware detector to endpoints and got it 25× smaller than the model it replaced. Neither is bragging about beating a neural network on a benchmark — they’re saying a few hundred integer comparisons per prediction fits inside a latency or binary-size budget that a neural network doesn’t.

The hybrid pattern is real, and it runs in both directions. NVIDIA embeds a transaction graph with a GNN and feeds the embeddings to XGBoost, reporting 40% better accuracy. Nubank keeps LightGBM as the production baseline and fuses transformer embeddings of transaction sequences into it. Same architecture, opposite motivations: deep learning learns a representation the table doesn’t contain (a graph, a sequence), trees make the decision once that representation is a column. This is the honest answer to “trees or neural networks” — in production the answer is frequently both, in series, and the boosted model is the part that ships the prediction.

And Etsy migrated away. Etsy powered personalized search ranking with GBDTs and then began moving to deep learning. That’s not a contradiction; it’s the boundary drawn from the other side. At sufficient scale, with rich text on both the query and the listing, the representation stops being tabular — and once the model needs to learn what the features are, the tree’s inductive bias becomes a ceiling rather than a gift. It’s the same lesson as the rest of the list, seen from the far edge.

The takeaway

Strip the systems work away and XGBoost is one idea: put the tree inside the loss, expand to second order, and solve.

Everything follows. The leaf value $-G/(H+\lambda)$ is what you get by minimizing a quadratic. The split gain is the difference of structure scores. $\gamma$ prunes because a leaf that can’t pay $\gamma$ shouldn’t exist. The Hessian weights each point by how much the loss curves there, so confident points step aside for uncertain ones. The quantile sketch is weighted by $h_i$ because completing the square says the problem is weighted least squares. And the loss function disappears after the second line, which is why one implementation serves regression, classification, and ranking without the tree-builder knowing which it’s doing.

The decision-tree article made the point that the right model is the one whose assumptions match your data. XGBoost’s assumptions are unusually explicit: your features mean something individually, the interactions between them are sparse, and you’d like the penalty for complexity written into the objective rather than bolted on afterward. When that describes your data, thirty minutes of tuning is hard to beat. When it doesn’t — when the features are pixels, or tokens, or a graph — no amount of second-order optimization will invent a representation that isn’t there. The ten production systems above are mostly the first case. Etsy is the second.