What a weight matrix actually does to a vector space

In our earlier article on how neural networks learn, we focused on how training adjusts weights and biases: backpropagation computes gradients, and gradient descent uses them to update the parameters. Here, we will explore the forward pass: how a network uses its current parameters to transform inputs into outputs. This computation takes place during both training and inference.

Vectors hold the representations, and weight matrices transform them. We will follow these operations through a classifier small enough to calculate by hand, starting with dots a straight line can separate and then turning to four criss-crossed XOR points where no line works. Separating the XOR classes with a linear output layer requires a nonlinear transformation of their representation. A hidden layer with hand-chosen weights will let us inspect how that happens.

In larger trained networks, knowing how to calculate every output does not automatically reveal what the internal representations encode or how the network uses them. Vectors provide a general way to encode objects, their properties, and relationships between them. The same mathematical tools can represent apple measurements for ripeness classification, image pixels for object recognition, or concepts and relationships for language understanding and generation.

Representational analysis studies what information internal vectors encode and how it is organized. Mechanistic interpretability investigates how a network’s components use that information to produce its behavior. The approaches overlap, but finding information in a representation does not by itself show that the network uses it.

This article assumes some familiarity with the following linear algebra concepts. The focus is on showing how a neural network uses them, rather than teaching them from scratch. Our dot-classification example will make their application concrete:

• Vectors and matrices: how a vector represents one point, and how a matrix holds a collection of points.
• Dot products: how each neuron combines its inputs with its weights to produce a scalar weighted sum, before adding the bias and applying any activation.
• Linear transformations: how a weight matrix produces new coordinates, and what its rows and columns tell us about the transformation.
• Matrix multiplication and composition: how successive linear transformations combine, and why a nonlinear activation prevents us from generally collapsing the whole network into one matrix.

A one-neuron classifier

Classical machine learning offers a variety of classification methods, including logistic regression, decision trees, and support vector machines. Here, we will implement a classifier using a neural network. Neural networks underpin deep learning; our small example will demonstrate the same building blocks while keeping every calculation manageable.

Let’s start with a dataset a single line can separate. Imagine four apples of the same variety, named A, B, C, and D. Each point in the dataset below represents one apple, described by two continuous measurements: a color score ($x_1$, from greener to redder) and a softness score ($x_2$, from firmer to softer). The labels tell us whether each apple is unripe (0) or ripe (1).

The values we use for color and softness are illustrative and have been centered and scaled, so zero represents a reference level for each feature: negative values mean greener or firmer, and positive values mean redder or softer, relative to those reference levels.

Our first network has two inputs and one output neuron, with no hidden layer.

The neuron computes a score for each apple, then applies a threshold to predict its class. We can organize the neuron’s two weights into a one-row matrix $W$, and the apple’s two inputs into a column vector $\mathbf{x}$:

To compute the score for one apple, we multiply the weight matrix $W$ by its input vector $\mathbf{x}$ and add the bias $b$. We write the result as $s(\mathbf{x})$, where $s$ stands for score and $\mathbf{x}$ contains the apple’s color and softness measurements:

Here, $W\mathbf{x}$ computes one dot product between two vectors: the neuron’s weight vector $(w_1,w_2)$, stored as the row of $W$, and the apple’s input vector $(x_1,x_2)$. Multiply each weight by its corresponding input, then add the two products. Adding the bias gives the output score.

The multiplication $W\mathbf{x}$ is already a linear transformation from $\mathbb{R}^2$ to $\mathbb{R}$: from a plane to a line. A transformation can turn a vector into a scalar; it need not keep the same number of coordinates. We will return to the geometric interpretation through basis vectors when we build the hidden layer.

This model is a perceptron, a single-neuron linear classifier. Its step activation turns the score into a predicted class label, $\hat y$:

We chose 1 to mean ripe and 0 to mean unripe when labeling the dataset. One output neuron is enough: a positive score produces label 1, while a zero or negative score produces label 0.

Let’s now visualize how our neuron classifies the four apples. In the default 2D view, each apple’s color and softness determine its position in the plane. The shaded background shows the predicted class for every possible input pair, with a straight line separating the unripe and ripe regions. Each marker’s color shows the apple’s known label; a pink ring marks a wrong prediction.

Adjusting the weights and bias changes the decision regions while the apples stay fixed. We chose their measurements to make the classes linearly separable, so suitable parameter values place all four apples in their correct regions.

Switch to Angled view (3D) to see the scores behind those predictions. The input plane holds the apples’ color and softness coordinates, while the blue plane shows the calculated score as a height above or below it. Where the two planes intersect, the score is zero: this is the same boundary line shown in 2D. For a closer look at how these views are constructed, see how to read the neural network graphs.

The formula above the plot is the neuron’s score calculation with the current parameters substituted. The sliders control the weights $w_1$ and $w_2$ and the bias $b$ shown in the network diagram. These are the parameters training would learn.

In our earlier example of fitting a line, one neuron learned one weight and a bias. Here there are two input measurements, so there are two weights and a bias. The same gradient-descent idea applies: a differentiable loss on the scores and known labels supplies gradients for updating the parameters; the hard threshold reports class predictions.

Let’s now trace the calculation behind the line shown in the widget. At the widget’s initial settings, $w_1=1$, $w_2=1$, and $b=-0.5$, so the score is $s(\mathbf{x})=x_1+x_2-0.5$. Its decision boundary contains the points whose score is zero:

That is a straight line. For example, for point C at $(0.5,0.5)$, we substitute $x_1=0.5$ and $x_2=0.5$ into the score formula: $s(\mathbf{x}_C)=0.5+0.5-0.5=0.5$. The score is positive, so the prediction is ripe (1). For point B at $(-0.5,0.5)$, we substitute $x_1=-0.5$ and $x_2=0.5$: $s(\mathbf{x}_B)=-0.5+0.5-0.5=-0.5$. The score is negative, so the prediction is unripe (0). The line separates the whole plane into those two kinds of answers.

We can also compute the scores for all four apples in one matrix multiplication. This is how we process a batch: collect the input vectors into a matrix $X$, with one apple per row and color and softness in the two columns:

The parameters are still the same two weights and one bias. If we start with $w_1=1$, $w_2=1$, and $b=-0.5$, as in the widget, the weight matrix and its transpose are:

The bias $b=-0.5$ is separate from $W$ and is added to each apple’s weighted sum.

For one apple written as a column vector, we used $W\mathbf{x}$. With apples stored as rows, we use $XW^\top$: the transpose turns the neuron’s weight row into a column, so each apple’s row takes a dot product with the same weights. Adding the same bias to every score gives:

Here, $\mathbf{1}$ is a column of four ones, so $b\mathbf{1}$ repeats the bias for all four apples. The rows of $S$ contain their scores in A–D order. For apple A, the first row computes $1(-1)+1(-0.5)-0.5=-2$, exactly as when we process it alone. Applying the step function to each score gives predictions $(0,0,1,1)$, all correct. Batching does not change how the score is calculated for an individual apple. Our batch contains the entire four-example dataset; larger datasets are often split into smaller batches, as discussed in how batch size affects training.

Four apples one neuron can’t separate

Let’s continue with our four apples, still described by color ($x_1$) and softness ($x_2$). We change their input values $(x_1,x_2)$ by hand while keeping their labels: A and B remain unripe (0), and C and D remain ripe (1). The new coordinates put each class on a diagonal of a square:

This is the familiar XOR pattern. We have shifted the usual binary coordinates, 0 and 1, to $-0.5$ and $+0.5$, placing them on opposite sides of zero on each axis. The four apple points form a square centered at the origin: apples with opposite signs for their two input values are labeled ripe (1), while those with matching signs are labeled unripe (0).

Each class occupies a different diagonal of that square. The diagonals cross, so no straight line can place all unripe apples on one side and all ripe apples on the other. This is a classic illustration of the perceptron limitations analyzed by Marvin Minsky and Seymour Papert in their 1969 book Perceptrons.

When we put the apples from the updated dataset through our original classifier, the same parameters $w_1=1$, $w_2=1$, and $b=-0.5$ now classify only one of four correctly. Open the widget’s parameter controls and try changing $w_2$ to $-1$: this reaches three of four, with C still misclassified.

To solve XOR, we will keep the two inputs and one output neuron, and insert a hidden layer with two neurons that use the nonlinear ReLU activation. This layer transforms each apple’s measurements into new coordinates $(h_1,h_2)$, where the output neuron can separate the classes with a straight line:

The illustration shows a multilayer perceptron (MLP): a network of fully connected layers, where each neuron receives all outputs from the previous layer. It is also a feed-forward network (FFN), meaning information flows from input to output without looping back. Other architectures use different kinds of layers. Convolutional neural networks (CNNs) use convolutional layers that apply shared filters to local regions, and can also include fully connected layers, for example to perform the final classification. Recurrent neural networks (RNNs) use recurrent layers that carry state between sequence steps.

We will now explore the linear algebra behind this network and build its forward pass in NumPy as we go. We will use individual apples to work through the calculations, then apply each operation to all four as a batch. The complete NumPy example brings those operations together.

Representing our apples as vectors and matrices

Each apple is represented as a vector of two numbers: its color score and its softness score. We use lowercase $\mathbf{x}$ to denote one apple’s input vector, containing its color score $x_1$ and softness score $x_2$. For apple C, we write this vector as a column:

$\mathbb{R}^2$ is the space of all pairs of real numbers. It is two-dimensional because two independent coordinate directions suffice to locate any point: move along the first axis, then along the second.

We can draw the vector as an arrow from the origin or mark its endpoint as a dot. Both pictures encode the same coordinates. Here the dot is useful because we are classifying examples.

Start the NumPy example with this representation of apple C:

import numpy as np

# One apple's representation: color and softness, in that order.
x = np.array([-0.5, 0.5])        # Shape (2,)

We write $\mathbf{x}_C$ as a column in the mathematics. In NumPy, this one-dimensional array supplies the input vector when we multiply it by $W$.

The representation space is the whole plane. Our dataset contains just four points in that plane. Having four examples does not make the space four-dimensional; each example still has two coordinates.

We can collect the four vectors into a representation matrix, with one example per row. Uppercase $X$ denotes the matrix holding all four apples, with one input vector per row.

The rows count examples (our four apples); the columns count features (color and softness). The ripeness labels are stored separately.

These are the same four input positions shown in the XOR widget. The arrows make explicit that every point represents a vector from the origin.

A representation matrix can hold vectors from the original input data, as $X$ does here, or vectors produced by a previous layer. In both cases, each row represents one example; what changes between layers is the information encoded in its coordinates, and possibly the number of coordinates. We distinguish this role from that of a transformation matrix: $X$ holds the representations, while the weight matrix $W$ defines the linear transformation applied to them.

Our apples already come with two numerical measurements, so forming their input vectors is straightforward. For other kinds of data, we need to decide how to represent each example as a vector. A $28 \times 28$ grayscale image can become a vector of 784 pixel values in a fixed order, while a token can be represented by an embedding vector. In each case, the number of coordinates in the vector determines the number of inputs a dense layer accepts.

Our layer needs two inputs, even if we collect a million more examples.

How basis vectors define a linear transformation

Before we get to the transformations performed by a layer, let’s see how a matrix describes a linear transformation through the destinations of the input basis vectors. Once we know where those vectors land, linearity tells us where every other input vector lands.

In two-dimensional space, we need two linearly independent vectors to form a basis. We use the usual unit vectors, one along each coordinate axis:

They form the usual basis of the input plane. Putting these two vectors into columns gives the identity matrix:

The basis matrix makes it possible for us to reconstruct any vector from its coordinates in that basis. Suppose we define a vector as

Its first coordinate, $x_1$, scales the first column, $\mathbf{e}_1$, and its second coordinate, $x_2$, scales the second column, $\mathbf{e}_2$. We now use a linear combination of the basis vectors, with $x_1$ and $x_2$ as coefficients, to reconstruct the original vector:

This means that to define a linear transformation, we can choose new coordinates for where the basis vectors $\mathbf{e}_1$ and $\mathbf{e}_2$ land and put those destinations into the matrix’s columns. We then use the same linear-combination logic, keeping the input vector’s original coordinates as coefficients, to construct the transformed vector and see where it lands in the output space.

Suppose a linear transformation sends $\mathbf{e}_1$ to $[a,b]^\top$ and $\mathbf{e}_2$ to $[c,d]^\top$. The transformation matrix records these new destinations as its columns:

Just as we saw above, we can reconstruct the transformed input vector by taking a linear combination of these transformed directions, using its original coordinates as coefficients. We scale the first column of $M$ by $x_1$ and the second by $x_2$, then add the results:

The input coordinates $(x_1,x_2)$ have become the output coordinates $(ax_1+cx_2,\;bx_1+dx_2)$. Some linear transformations collapse the plane onto a line or a point. They are still valid, and neural networks can use such transformations to discard information while preserving distinctions useful for the task.

The same rule carries every point of a grid, including points outside our dataset. Next, we will apply it to the matrix of our hidden layer.

A layer’s matrix defines a transformation

Let’s first look at how the hidden layer produces a new representation for any input vector. Then we will choose concrete weights and follow apple A through the matrix multiplication, calculating where its vector lands.

The hidden layer’s weight matrix $W$ is our transformation matrix: it defines how vectors from the input data or a previous layer are linearly transformed. Its shape depends on both the number of inputs and the number of neurons: with $d$ inputs and $m$ neurons, $W$ has $m$ rows and $d$ columns.

Each neuron defines one row of weights, which calculates one output coordinate. As we saw in the basis-vector explanation, each column shows where one input basis vector lands across all those output coordinates. Adding a neuron therefore adds an output coordinate, not another input basis vector. We can associate that coordinate with an axis in the output space, but the neuron’s weight vector is not itself that output basis vector.

Our layer has two neurons and two inputs, so $W$ has two rows and two columns. Each row holds one neuron’s weight for color ($x_1$) and its weight for softness ($x_2$).

These weights are parameters the network would learn during training. Here, we have chosen them by hand so every step is exact and easy to inspect.

To apply this transformation, we multiply the incoming data vector $\mathbf{x}$ by $W$, as $W\mathbf{x}$. This computes one weighted sum per neuron.

Alongside its weight matrix, our hidden layer also has a bias vector $\mathbf{b}$, with one bias for each of its two neurons. Adding $\mathbf{b}$ gives $\mathbf{z}=W\mathbf{x}+\mathbf{b}$, the pre-activation vector. Together, the multiplication and bias addition form an affine transformation: a linear transformation followed by a shift. The same bias vector shifts every result. A linear transformation always maps the origin to the origin; adding the bias moves it to $\mathbf{b}$. In our example below, $\mathbf{b}=\mathbf{0}$, so there is no shift and the map is also linear.

ReLU then replaces each negative coordinate of $\mathbf{z}$ with zero, producing $\mathbf{h}=(h_1,h_2)$, the hidden layer’s output. This is the apple’s new representation, which the output neuron will use to calculate its score. The complete sequence is:

Define the same transformation matrix and bias in NumPy:

# Two input coordinates -> two pre-activation coordinates.
# Rows hold neuron weights; columns record input basis-vector images.
W = np.array([[1.0, -0.8],
              [-0.8, 1.0]])     # Shape (2, 2)
b = np.array([0.0, 0.0])        # Bias vector; no shift here

We will use apple A as an example, but the widget below lets you follow any of the four apples through the same matrix $W$. A’s original vector is

Using the column-combination rule, we can calculate $W\mathbf{x}_A$ step by step from its original coordinates and the columns of $W$.

1. Read the columns of $W$ as destinations of the basis vectors $\mathbf{e}_1$ and $\mathbf{e}_2$. The first column tells us where $\mathbf{e}_1=(1,0)$ lands; the second tells us where $\mathbf{e}_2=(0,1)$ lands:

   $$W=\begin{bmatrix}1&-0.8\\-0.8&1\end{bmatrix}, \qquad W\mathbf{e}_1=\begin{bmatrix}1\\-0.8\end{bmatrix}, \qquad W\mathbf{e}_2=\begin{bmatrix}-0.8\\1\end{bmatrix}.$$

   These are the original basis vectors after applying $W$.

2. Use A’s original coordinates as coefficients. As we showed above, we can use a linear combination of the transformed basis vectors to construct A’s vector in the transformed space:

   $$W\mathbf{x}_A=-0.5(W\mathbf{e}_1)+(-0.5)(W\mathbf{e}_2)$$

   The coefficients stay the same; the vectors they multiply are now the transformed directions. The first coordinate, $-0.5$, scales the first column of $W$, which is the transformed first basis vector, $W\mathbf{e}_1$. The second coordinate, also $-0.5$, scales the second column, the transformed second basis vector, $W\mathbf{e}_2$.

   So the full calculation looks like this:

   $$\begin{aligned} W\mathbf{x}_A &=-0.5\begin{bmatrix}1\\-0.8\end{bmatrix} +(-0.5)\begin{bmatrix}-0.8\\1\end{bmatrix}\\[6pt] &=\begin{bmatrix}(-0.5)\times 1\\(-0.5)\times(-0.8)\end{bmatrix} +\begin{bmatrix}(-0.5)\times(-0.8)\\(-0.5)\times 1\end{bmatrix}\\[6pt] &=\begin{bmatrix}-0.5\\0.4\end{bmatrix} +\begin{bmatrix}0.4\\-0.5\end{bmatrix} =\begin{bmatrix}-0.5+0.4\\0.4+(-0.5)\end{bmatrix} =\begin{bmatrix}-0.1\\-0.1\end{bmatrix} \end{aligned}$$

The two contributions partially offset each other, leaving $(-0.1,-0.1)$. We used A’s original coordinates to combine the transformed basis directions. The same rule applies to B, whose coefficients are both $+0.5$:

A and B move closer to the origin but remain different points. The matrix changes their representations without merging them.

The animation below shows both steps together: the basis arrows move to the destinations recorded in $W$, and the selected apple follows their linear combination using its original coordinates as coefficients. The blue grid follows the same transformation, while the original grid stays fixed for reference.

Select A, B, C, or D, then choose Transform. With A selected initially, its vector moves from $(-0.5,-0.5)$ to $(-0.1,-0.1)$.

We can see what the transformation does by following the two diagonals:

The A–B direction shrinks to one fifth of its length, while the C–D direction stretches by a factor of $1.8$. Neither direction disappears. This changes the shape of the four-point square but preserves the crossing of the class diagonals. The linear transformation alone has not made XOR separable. For that, we’ll need to add nonlinearity through the ReLU activation, which we will examine below.

Here’s how to implement this transformation in NumPy, using the matrix W defined above. The @ operator performs matrix multiplication:

x_A = np.array([-0.5, -0.5])    # A's original coordinates

# Apply W to the whole vector.
transformed_A = W @ x_A
print(transformed_A)             # [-0.1 -0.1]

For matrix–vector multiplication, W @ x_A is equivalent to scaling each column of W by the corresponding coordinate of x_A and adding the resulting vectors. We can write that linear combination explicitly:

# W[:, 0] is the first column; W[:, 1] is the second.
transformed_A = x_A[0] * W[:, 0] + x_A[1] * W[:, 1]
print(transformed_A)             # [-0.1 -0.1]

How neurons work together to transform a vector

Above, we used the layer’s transformation matrix to transform apple A’s input vector:

Now let’s look at how this matrix is constructed from the neurons in the layer and the dimensionality of each input vector, and where each neuron’s work appears in that calculation.

In our fully connected layer, every neuron receives the same complete input vector. Each apple has two input coordinates, so each neuron needs two weights: one for $x_1$ and one for $x_2$. We have two neurons, each supplying one row of weights:

Each row holds one neuron’s weights, while each column records where one input basis vector lands. For example, the first column contains both neurons’ weights for $x_1$. For input $\mathbf{e}_1=(1,0)$, those weights give the two output coordinates, $W\mathbf{e}_1=(1,-0.8)$. So a column combines contributions from both neurons.

Stacking these rows gives our $2\times2$ matrix $W$. More generally, a dense layer with $d$ input coordinates and $m$ neurons has

The input dimension determines the number of columns; the number of neurons determines the number of rows and output coordinates. These counts determine the matrix’s shape. The weight values determine which linear transformation it performs. For example, two neurons receiving three-dimensional inputs would give a $2\times3$ matrix, mapping three input coordinates to two output coordinates.

Interestingly, taking a linear combination of the matrix columns, using the input coordinates as coefficients, and taking a dot product of each matrix row with the input vector give exactly the same transformed vector.

In the previous section, we obtained $W\mathbf{x}_A=(-0.1,-0.1)$ by scaling the columns by A’s original coordinates and adding them. Now we will obtain the same result through the neurons’ calculations: each neuron takes a dot product of its row of weights with the complete input vector, producing one output coordinate.

For fixed weights, that dot product defines a function that takes an input vector and returns one scalar number: $f_i(\mathbf{x})=\mathbf{w}_i\cdot\mathbf{x}$. For our first neuron, this scalar-valued linear function is $f_1(\mathbf{x})=x_1-0.8x_2$, mapping $\mathbb{R}^2$ to $\mathbb{R}$. The transformation matrix, and the linear part of the layer it describes, can therefore be viewed as a collection of these functions, one per neuron. Together, their results form $T(\mathbf{x})=(f_1(\mathbf{x}),f_2(\mathbf{x}))^\top=W\mathbf{x}$, mapping $\mathbb{R}^2$ to $\mathbb{R}^2$.

With our zero biases, the two functions’ results are also the pre-activation coordinates. Writing out their dot products gives

For the same apple A, both neurons receive $(-0.5,-0.5)$:

Collecting these two numbers gives $\mathbf{z}_A=(-0.1,-0.1)$, exactly the vector we constructed by combining columns. The neurons’ weighted sums together perform the layer’s matrix multiplication. Each neuron contributes one coordinate of the result; it does not apply the whole matrix independently. The same weights are used for every apple.

The two views group the same arithmetic differently. A column collects one input coordinate’s contributions to all neurons. A row collects all the weights used by one neuron to calculate its output coordinate. Neither column belongs to a single neuron. Our chosen $W$ is symmetric, so its rows and columns happen to contain the same numbers, but their roles are different.

We can make the individual neurons’ calculations explicit in NumPy:

# Each row is one neuron's weight vector.
z1_A = np.dot(W[0], x_A)
z2_A = np.dot(W[1], x_A)
z_A = np.array([z1_A, z2_A])
print(z_A)                      # [-0.1 -0.1], the same as W @ x_A

Adding each neuron’s bias makes its function affine; applying ReLU then makes it nonlinear. Together, these outputs form the hidden representation $\mathbf{h}$. We will examine the effect of ReLU below. To continue the forward-pass code for apple C, using its earlier input vector x, we compute both neurons’ weighted sums and add their biases in one operation:

z = W @ x + b
print(z)                        # [-0.9  0.9] for apple C

Now when we combine all input vectors into a representation matrix, we can apply the layer’s transformation to all four apples in one matrix multiplication. We put A, B, C, and D in the rows of $X$, with their two input coordinates in the columns.

For one column vector, we used $W\mathbf{x}$. Because our batch stores vectors as rows, we use $XW^\top$: transposing $W$ puts each neuron’s weights into a column, so each apple’s row takes a dot product with each neuron’s weights.

The transpose follows our storage convention: examples are rows of $X$, and neurons’ weights are rows of $W$. If the weights were stored with one neuron per column instead, we would use $XW$ directly. Transposing aligns the dimensions; it does not change the intended transformation.

With our zero biases, the calculation is

Each row of $Z$ holds one apple’s transformed vector; each column holds one neuron’s weighted sums across the batch. The first row is still $(-0.1,-0.1)$ for A, exactly as when we processed it individually. Our chosen $W$ is symmetric, so $W^\top$ happens to have the same entries as $W$, but the transpose expresses the correct operation for examples stored as rows.

In NumPy, using the same W and b defined above:

# One input vector per row, in A, B, C, D order.
X = np.array([
    [-0.5, -0.5],
    [ 0.5,  0.5],
    [-0.5,  0.5],
    [ 0.5, -0.5],
])

# (4 apples, 2 inputs) @ (2 inputs, 2 neurons) -> (4, 2)
# Each apple's row takes a dot product with each neuron's weights.
Z = X @ W.T + b                  # Add the same bias vector to every row
print(Z)
# [[-0.1 -0.1]   A
#  [ 0.1  0.1]   B
#  [-0.9  0.9]   C
#  [ 0.9 -0.9]]  D

Batching groups the calculations without changing the transformation of any individual vector. Every apple goes through both neurons using the same weights and biases; its measurements are never combined with another apple’s.

The activation makes the new representation separable

So far, we have calculated the linear part of the hidden layer: each neuron produces one coordinate of $\mathbf{z}=W\mathbf{x}$, with no shift because our biases are zero. These are the coordinates before activation:

The A–B segment is shorter and the C–D segment is longer, but they still cross. A straight line cannot separate the two classes.

The hidden layer has one more operation to perform. Each neuron applies the nonlinear ReLU activation to its own weighted sum. Together, these results form the hidden layer’s output vector, $\mathbf{h}$:

It keeps positive values and replaces negative values with zero. Our two hidden coordinates become

$h_1$ measures the positive part of $x_1-0.8x_2$, while $h_2$ measures the positive part of $x_2-0.8x_1$. These are the two features the hidden layer computes.

To see the effect of ReLU on each apple’s representation, select A, B, C, or D, then choose Apply ReLU. The widget starts with the coordinates produced by $W$. With A selected initially, its vector changes from $(-0.1,-0.1)$ to $(0,0)$. Selecting another apple resets the animation to that apple’s coordinates before activation.

Here are the coordinates before and after ReLU for all four apples, matching the results shown in the widget:

A moves to the origin because both its coordinates are negative. B stays at $(0.1,0.1)$ because both are positive. C and D each lose their negative coordinate and land on different positive axes. The class segments no longer cross.

In NumPy, apply ReLU to the batch matrix Z calculated above. The maximum operation compares each coordinate with zero and keeps the larger value. Each row of H holds one apple’s hidden representation:

# Keep positive coordinates; replace negative ones with zero.
H = np.maximum(0, Z)
print(H)
# [[0.  0. ]   A
#  [0.1 0.1]   B
#  [0.  0.9]   C
#  [0.9 0. ]]  D

ReLU does not restrict its output to 0 or 1. Here it retains values such as $0.1$ and $0.9$. The final classification threshold is the operation that returns only 0 or 1.

The output layer classifies the hidden representations

The hidden layer has produced a new vector for each apple. The output neuron receives its two coordinates, $h_1$ and $h_2$.

This output neuron has its own weights, stored in $V$, and bias, $c$. Training would learn these parameters alongside the hidden layer’s weights and biases. For our demonstration, we choose them by hand:

Its score is another dot product, followed by the bias:

As before, a positive score predicts ripe (1); a zero or negative score predicts unripe (0). The decision boundary contains the representations whose score is zero:

This threshold falls between the sums for the two classes:

A and B have sums below $0.5$, while C and D have sums above it. That is why the line $h_1+h_2=0.5$ separates the classes. Our hand-chosen threshold is not unique: any value strictly between $0.2$ and $0.9$ would also separate these four representations.

Let’s look at how the complete network classifies our apples in the original input plane. The widget opens in Top view (2D), showing the decision regions with all four apples classified correctly at our hand-chosen parameter values. Switch to Angled view (3D) to see $(x_1,x_2,s(\mathbf{x}))$: each apple’s original coordinates together with its final score as height. This third axis visualizes the scalar output; the hidden representation still has two coordinates. The blue score surface has creases where the hidden neurons switch between zero and positive outputs, and its intersection with the input plane forms the decision boundary. Adjust the weight and bias sliders to explore how the surface and predictions change, or select Reset to restore our parameter values.

The boundary is straight in hidden space, but bends in the original input space. A and B lie in the unripe region; C and D lie in the ripe region on opposite sides of it. With these weights, the two ripe sides connect farther along the positive diagonal, beyond the four example points. Correctly classifying four examples does not specify what the network should predict everywhere else.

To implement the output layer in NumPy, we apply its weights and bias to all four hidden representations. With one apple per row of $H$, we compute $S=HV^\top+c$, adding the same bias to every apple’s score:

# Output transformation: two hidden coordinates -> one score.
V = np.array([[1.0, 1.0]])       # Shape (1, 2)
c = -0.5                        # Output bias

# (4 apples, 2 hidden coordinates) @ (2, 1) -> (4, 1)
# Each row takes a dot product with the same output weights.
S = H @ V.T + c
predictions = (S > 0).astype(int)  # 1 = ripe, 0 = unripe

for apple, score, prediction in zip("ABCD", S[:, 0], predictions[:, 0]):
    print(f"{apple}: score={score:.1f}, prediction={prediction}")
# A: score=-0.5, prediction=0
# B: score=-0.3, prediction=0
# C: score=0.4, prediction=1
# D: score=0.4, prediction=1

Why ReLU alone does not solve this dataset

What if the hidden layer used the identity matrix $I$ for its weights, with zero biases? It would compute $\mathbf{h}=\operatorname{ReLU}(I\mathbf{x})=\operatorname{ReLU}(\mathbf{x})$, exactly the same as applying ReLU directly to the original inputs. The result would still be an XOR arrangement: negative input coordinates become zero, leaving A at $(0,0)$, B at $(0.5,0.5)$, C at $(0,0.5)$, and D at $(0.5,0)$. These are the four corners of a smaller square, with each class still occupying a different diagonal. No straight line separates them.

Switch between Original, ReLU(x), and ReLU(Wx) to compare the three arrangements. The second view uses identity weights; the third uses our chosen $W$. Both hidden-layer alternatives start from the original input vectors and use zero biases.

The matrix $W$ first changes the coordinates so that ReLU acts on a more useful arrangement. It places B at $(0.1,0.1)$ and C and D at $(-0.9,0.9)$ and $(0.9,-0.9)$. ReLU then keeps B near the origin and places C and D farther along the positive axes, while A lands at the origin. The resulting representations can be separated by a straight line.

Neither $W$ alone nor ReLU applied directly to these inputs solves this example; their combination makes the classes linearly separable. The output layer can separate the resulting representations, as we saw above.

Layers compose functions

Some books describe neural networks as a composition of functions that map vectors from one vector space to another. Our hidden layer maps $\mathbb{R}^2$ to $\mathbb{R}^2$, producing a hidden representation, and our output layer maps $\mathbb{R}^2$ to $\mathbb{R}$, producing one scalar score. Each function receives the result of the previous one. These functions can be linear or nonlinear, as we saw with the weight matrix and ReLU.

For our network, call the hidden-layer function $F$ and the output-layer function $G$:

The forward pass we have followed through the widgets is therefore

Substituting the hidden layer’s output into the output-layer function gives

This is what function composition means here. The common notation for this composition is $G\circ F$, meaning apply $F$ first, then $G$. A deeper network follows the same pattern, $F_L\circ\cdots\circ F_2\circ F_1$: each layer transforms the representation it receives and passes its result to the next.

Without nonlinear activations, the layers combine into one affine map

You may also have read that without nonlinear activations, all the layers collapse into one. This is easy to see with our two weight matrices. Temporarily remove ReLU. Because our hidden biases are zero, the output neuron now receives $W\mathbf{x}$ directly:

We can multiply $V$ and $W$ first and use their product as a single weight matrix:

The resulting network computes

We are back to the same kind of score as our original single neuron: two weighted inputs plus a bias, with a straight-line decision boundary. These particular weights give scores $-0.7$, $-0.3$, $-0.5$, and $-0.5$ for A, B, C, and D, predicting unripe for all four. Changing the weights could improve those predictions, but no single affine score can classify this XOR arrangement correctly.

The word “collapse” refers to the function the network computes: a single affine layer can produce exactly the same outputs as the whole stack. Nonzero biases do not change this conclusion. They combine into one bias:

Repeating this calculation combines any stack of affine layers into one affine map. Applying our final step threshold still produces a straight-line boundary; the threshold alone cannot give us the effect of a nonlinear hidden layer.

Matrix multiplication records the composed transformation

This connects directly to our earlier view of columns as destinations of the input basis vectors. The columns of $W$ tell us where $\mathbf{e}_1$ and $\mathbf{e}_2$ land after the first transformation. Applying $V$ to those destinations tells us where they land after both:

Each input basis vector ends up at $0.2$ on the output number line. For any input, the same column-combination rule therefore gives $(VW)\mathbf{x}=0.2x_1+0.2x_2$.

More generally, multiplying two transformation matrices applies the left matrix to every column of the right matrix. If we apply $M_1$ and then $M_2$, their product records the final destinations of the input basis vectors:

The rightmost matrix acts first. The product represents both linear transformations as one.

ReLU changes what the next layer receives

Now put ReLU back between $W$ and $V$. The output neuron receives $\operatorname{ReLU}(W\mathbf{x})$, so it no longer simply adds the two coordinates produced by $W$. It adds their nonnegative parts:

For example, C’s coordinates after $W$ are $(-0.9,0.9)$. Without ReLU, they cancel and give a score of $-0.5$. With ReLU, the output neuron receives $(0,0.9)$ and gives a score of $0.4$. The same output weights now produce the correct prediction because they receive a different representation.

The layers still compose as functions, but we cannot replace this network with one affine layer. ReLU keeps or removes each coordinate according to its sign, making the complete score a nonlinear function of the original input.

This also connects the two views in our classifier widget. In hidden space, the output boundary is the straight line $h_1+h_2=0.5$. Substituting the hidden-layer functions gives the same boundary in the original input coordinates:

Where only the first neuron is active, this reduces to $x_1-0.8x_2=0.5$. Where only the second is active, it becomes $x_2-0.8x_1=0.5$. Where both are active, it becomes $0.2(x_1+x_2)=0.5$. These straight pieces join at corners, producing the boundary we see in the original input plane.

Every new apple goes through this same composition: its measurements become a hidden representation, which becomes a score and then a predicted label. A straight boundary in hidden space can therefore correspond to a nonlinear boundary in the original space.

Run the model on all four apples with NumPy

Let’s now bring the operations together in one runnable NumPy example. Each row remains one apple throughout the forward pass:

We use the same row-per-example convention as above: $Z=XW^\top+\mathbf{b}$, $H=\operatorname{ReLU}(Z)$, and $S=HV^\top+c$, with each bias added to every row. Batch size and representation size are separate dimensions: the four examples stay the same, while each apple’s two hidden coordinates become one output score.

import numpy as np

# Representation matrix X, shape (4, 2):
# rows = apples A, B, C, D; columns = color and softness.
X = np.array([
    [-0.5, -0.5],
    [ 0.5,  0.5],
    [-0.5,  0.5],
    [ 0.5, -0.5],
])
labels = np.array([0, 0, 1, 1])   # Targets, kept separate from X

# Transformation matrix W, shape (2, 2), shared by all apples.
# Rows hold neuron weights; columns are transformed basis vectors.
W = np.array([
    [ 1.0, -0.8],
    [-0.8,  1.0],
])
b = np.array([0.0, 0.0])

# Output transformation V, shape (1, 2): two coordinates -> one score.
V = np.array([[1.0, 1.0]])
c = -0.5

# The examples are rows, so transpose W to compute each
# apple's dot product with each neuron's weights: (4, 2) @ (2, 2).
# NumPy adds the same bias vector to every row.
Z = X @ W.T + b                  # Shape (4, 2)

# ReLU replaces negative coordinates with zero.
H = np.maximum(0, Z)             # Shape (4, 2): new representations

# Compose the output map with the hidden map, row by row:
# (4, 2) @ (2, 1) -> (4, 1), then add the same scalar bias.
scores = H @ V.T + c             # One score per apple
predictions = (scores[:, 0] > 0).astype(int)

print(H)
# [[0.  0. ]   A
#  [0.1 0.1]   B
#  [0.  0.9]   C
#  [0.9 0. ]]  D

print(scores[:, 0])             # [-0.5 -0.3  0.4  0.4]
print(predictions)              # [0 0 1 1]

$X$ and $H$ describe the same examples in different spaces. $W$ and $V$ specify transformations between those spaces. The activation between them is what lets the complete network express a nonlinear rule.

Why changing the representation helps

We followed four apples through a network, but the purpose was never simply to move points around. It was to construct coordinates in which the distinction we wanted to make became easier to express.

XOR makes this concrete. In the original color–softness plane, no straight line separates the classes. The weight matrix reshapes that arrangement, and ReLU changes it nonlinearly. In the resulting representation, a weighted sum and a threshold are enough. The classification problem has stayed the same; the representation used to solve it has changed.

This gives us a small example of how a network can express a rule through geometry and computation. The hidden vectors encode derived properties of the inputs, and the output neuron combines those properties to make a decision. The rule emerges from the complete sequence of operations.

Larger networks can learn representations with many more coordinates, encoding properties and relationships useful for their tasks. Their architecture determines the available dimensions; training adjusts how inputs are represented within them. Concepts need not correspond to individual coordinates: information can be distributed across many coordinates and used by later layers. Networks can also represent more features than they have dimensions by encoding them in overlapping directions, a phenomenon called superposition.

Our hand-chosen network lets us inspect how useful features are computed and used. Linear algebra supplies the transformations, nonlinear activations expand what they can accomplish, and training searches the space of possible weights and biases for a combination that minimizes the loss. Each combination defines a candidate input-to-output function.