How neurons became linear algebra

In our article on weight matrices, we moved between two views of a neural network: a layer of neurons connected to the previous layer, and a matrix sitting between the layers, transforming the vector. The connection between these views is far from obvious: how did a network of interacting units come to be understood as transformations of vectors? Historically, this transition required a substantial change in what researchers thought they were studying.

We can see how far that change had gone by opening Frank Rosenblatt’s Principles of Neurodynamics, published in 1962. He describes connections between units with an interaction matrix and treats the network’s adjustable parameters as coordinates in a Euclidean space. Learning can be described as movement through that space.

Here, each complete set of weight values is a point in the space of possible network weights. When learning changes the weights, that point moves to new coordinates, so successive updates trace a path through this parameter space. This movement describes how the network’s weights change during training; input activity passing through the network is a separate process.

DEFINITION 16: The interaction matrix for a network of S, A, and R units is the matrix of coupling coefficients, $v_{ij}$, for all pairs of units, $u_i$ and $u_j$. If there is no connection from $u_i$ to $u_j$, $v_{ij}$ is defined as zero. Specifying an interaction matrix is equivalent to specifying a point in the phase space.

Here, the matrix records the strengths of connections between units. This gives us a link between the two views: connections in a network become coefficients in a matrix, which we can use to calculate how activity passes between units.

However, Rosenblatt’s matrix marks only one stage in a longer story. To understand how neural networks came to be described through linear algebra, we need to look both at the ideas that preceded it and at the developments that followed. Researchers were tackling computation, recognition, and memory, and each problem gave them different reasons to use the mathematics.

The trouble with a perfectly specified circuit

The starting point was a simplified picture of a biological neuron: it either fires or it doesn’t. Warren McCulloch and Walter Pitts built on this idea in their 1943 paper. If neural activity could be idealized as an all-or-none event, perhaps networks of neurons could be studied through logic.

Their simplified units had thresholds and delays. Some inputs excited a unit; inhibitory inputs could prevent it from firing. With the network’s structure fixed, they could investigate how its activity expressed logical relationships, including relationships between events at different times.

The attraction of this approach is easy to see. Instead of confronting all the biological detail at once, you could specify a circuit and reason about what it computed. A drawing of connected cells became a mathematical object.

But specifying the circuit leaves out much of what makes learning interesting. An animal acquires behavior through experience. Something about the system must be able to change.

Donald Hebb’s The Organization of Behavior, published in 1949, placed that change in the effectiveness of connections. When one cell repeatedly helped cause another to fire, he proposed, a lasting change could make that influence stronger. This was a biological hypothesis, expressed in words. It gave later mathematical models a concrete place to put the effects of experience.

Rosenblatt wanted a quantitative account of how a system with partly random wiring could learn. In his 1958 perceptron paper, he pointed to a difficulty with the logical approach: the broad organization of a nervous system might be known while its precise connections remained unknown. He therefore chose probability theory as the basis for his model.

That choice changes the question. Given a precisely specified circuit, we can ask what it computes. Given a statistically described population of units, we can ask under what conditions it will acquire useful behavior.

Rosenblatt’s perceptron contained sensory, association, and response units, with randomness in parts of their connectivity. Learning altered how association activity influenced responses. It was a much richer proposal than the single threshold neuron that usually introduces perceptrons today.

Now there were patterns of activity to distinguish, connections to adjust, and learning behavior to predict. Logic had made neural computation tractable. Probability allowed Rosenblatt to investigate it without knowing every wire.

When a connection became a coordinate

Once connection strengths are adjustable numbers, we can describe learning geometrically. For a neuron with two weights and a bias, each setting is a point $(w_1,w_2,b)$ in a three-dimensional space. Each axis represents one adjustable parameter. Changing the parameters during learning moves that point, so a sequence of updates traces a path through what we call parameter space.

Bernard Widrow and Marcian Hoff made this view concrete in their 1960 work on ADALINE. In their analysis, inputs were vectors and mean squared error formed a surface over the device’s adjustable coefficient settings. They measured that error before the final threshold converted the signal into a discrete decision, allowing adaptation to follow the surface toward a minimum. The same coefficient was a setting in a device and a coordinate in an optimization problem.

This brings us back to Rosenblatt’s book. His interaction matrix collected the connection weights between neurons, using zero where no connection existed. These adjustable weights retained the effects of learning—the network’s “memory.” Different weight settings became points in parameter space. He went on to use rank and singularity to analyze which classifications perceptrons could realize.

The following simplified diagram shows only the neurons’ connections and weights; their activation functions are omitted. It is an illustrative example, not a reproduction of Rosenblatt’s architecture. Each arrow is labeled with its weight:

We can record the same connections in a table, with rows for receiving units and columns for sending units:

Each cell gives the weight from the column’s unit to the row’s unit; 0 means there is no connection in that direction in this example. Keeping the same row and column order, we get the matrix $W$:

The entry in row $B$, column $A$ is $2$, matching the arrow $A\to B$. Row $C$ contains the two incoming weights: $-1$ from $A$ and $3$ from $B$. Row $A$ is all zeros because no arrows enter $A$, and the diagonal is zero because no unit connects to itself. Changing one connection strength changes one matrix entry.

The matrix was doing real explanatory work. A network could be examined as a whole, and properties of that whole could tell you something that was difficult to see by inspecting its connections individually.

The same mathematical tools could also reveal a network’s limitations. In their 1969 book Perceptrons, Marvin Minsky and Seymour Papert examined models that combined the outputs of feature detectors through a single weighted sum and threshold. They showed that, for some tasks, success depended on what those detectors could represent: adjusting the final weights alone was not enough.

Why counting is not enough

One of the problems Minsky and Papert studied was easy to state: given an image in which each pixel is either on or off, decide whether the number of active pixels is odd or even. This property is called parity. The following diagram shows all four possible images with two pixels, $x_1$ and $x_2$, and their desired outputs:

Suppose we feed these two pixel values directly into a single threshold neuron. For each input pair, it produces one number, called its activation. It calculates this using two stored weights, one per input, and a bias. The neuron adds the weighted inputs and bias, then applies its activation function: a binary threshold that outputs 1 if the result is greater than zero and 0 otherwise. Other activation functions can produce values such as 0.3 or 2.7.

To solve the parity task by counting, we need to add up the active pixels and check whether the total is odd. Setting both weights to 1 handles the counting: the weighted sum becomes $x_1+x_2$. But the neuron’s threshold can only check whether this count exceeds a cutoff set by the bias. For example, a bias of $-0.5$ gives the comparison $x_1+x_2-0.5>0$, or equivalently $x_1+x_2>0.5$. This checks whether at least one pixel is active, which includes the case where both are active and the count is even.

Written as steps:

sum = x₁ + x₂
if sum > 0.5:
    output 1
else:
    output 0

Compare its outputs with the desired parity labels:

The last row exposes the limitation: accepting a count of 1 also accepts 2. Moving the cutoff cannot select only the middle count. This two-pixel parity task is called XOR.

A single neuron with a linear weighted sum and a single-threshold activation cannot solve XOR. Changing its weights or bias still leaves a straight-line decision boundary, $w_1x_1+w_2x_2+b=0$. The four input pairs form a square, with the odd and even classes occupying its two diagonals. No straight line separates them.

A different activation could solve it: output 1 only when the sum lies between 0.5 and 1.5, and 0 otherwise. With both weights set to 1 and zero bias, one neuron would succeed. But this custom activation already encodes the rule we want to implement.

Modern networks usually combine many neurons with simple, general-purpose activation functions and learn the weights that make them work together. ReLU, for example, outputs zero for negative values and preserves positive values. A hidden layer using this operation can learn to transform the inputs into features that the output neuron can separate. Our XOR example shows how.

This example isolates a single threshold neuron receiving raw pixels. Rosenblatt’s 1958 model already included sensory, association, and response units, so earlier processing stages existed at the time. A response neuron could receive features produced by those stages. What mattered was whether those features supported the required decision.

Minsky and Papert’s parity result addressed that broader situation. They proved that a perceptron combining feature detectors through a single weighted threshold could not solve parity for every input unless at least one detector depended on the entire image.

The failure could therefore come from the architecture’s inability to supply the necessary features. In that case, adjusting the final connection strengths could not solve the problem.

A matrix that could remember

Recognition was only one reason to care about changing connections. Memory posed another problem: how could an experience leave a trace that a later cue could recover?

Karl Steinbuch had approached this problem in Die Lernmatrix, published in 1961. The name means learning matrix. His proposed circuit had a learning phase, in which signals representing properties appeared alongside their associated meaning, and a recall phase, in which presenting one could recover the other. He proposed implementations using magnetic storage elements or electrochemical processes, and envisaged applications such as character recognition and information retrieval.

This made memory an engineering problem: how to change an array during learning so that signals passing through it later recovered an association.

Teuvo Kohonen took up that problem in Correlation Matrix Memories, published in 1972. He explicitly replaced Steinbuch’s switching matrix with a correlation matrix, studying the mathematics of associative recall while leaving its possible biological role outside the paper.

In the version that stored every pairwise product, each association paired a key vector with a data vector. Multiplying every data component by every key component formed an outer product: a matrix of pairwise products. The memory stored a scaled sum of these matrices. Applying it to a key then produced a recalled pattern.

The same mathematics explained why recall could go wrong. Similar keys could bring back unwanted contributions from other associations. Memories stored together could interfere with one another.

James Anderson’s work published that year approached the same product-based construction through local synaptic changes. A connection changed in proportion to the product of the activity in the sending cell and the activity in the receiving cell. With connections between every pair of cells in the two groups, these changes formed the same kind of outer product. Here a numerical Hebbian rule became a matrix update: local changes accumulated into a store from which an associated pattern could later be retrieved.

John Hopfield gave the memory problem a particularly recognizable form in his 1982 paper: imagine recovering a complete bibliographic reference from a fragment, perhaps even a fragment containing a misspelling.

His network fed activity back through recurrent connections. An incomplete pattern could evolve toward a stored one. The coupling matrix described how the units influenced one another, while a separate vector described their current states.

With symmetric connections, no self-connections, and asynchronous updates, Hopfield could show that an energy function would not increase as the states changed. That connected neural recall to the behavior of interacting physical systems. Stored patterns could act as attractors: states toward which nearby activity moved.

During recall, the connections stayed fixed. The activity settled. The coupling matrix and the nonlinear updates together explained how a partial cue could lead to a more complete pattern.

We have come a long way from collecting connection strengths in a table. Matrices could help explain how experience was stored, why memories interfered, and how a network recovered from an imperfect cue.

Learning what the next layer should see

The recognition problem still contained the difficulty exposed by perceptron analysis: useful decisions depended on useful features. Could the network acquire those features through experience as well?

One revealing result came from Erkki Oja in 1982. He studied a linear neuron with a Hebbian learning rule modified to control the growth of its weights. Under the conditions of his analysis, the weights approached a dominant eigenvector of the input’s second-moment matrix. For zero-mean inputs, that direction is the first principal component.

A rule expressed as local connection changes could therefore extract a known statistical structure. Linear algebra helped identify what the neuron was learning.

For a network with several stages, there was an additional challenge. Training examples could specify the answer, but they generally did not specify what each intermediate unit ought to detect. The desired behavior at the output had to guide changes deeper inside the network.

David Rumelhart, Geoffrey Hinton, and Ronald Williams made that possibility vivid in Learning Representations by Back-Propagating Errors, published in 1986. Their examples adjusted hidden connections as well as output connections. In a network trained to recognize mirror symmetry, they examined the learned weights to understand how the intermediate units contributed to the solution.

The derivative machinery had earlier roots in Seppo Linnainmaa’s work on reverse accumulation, originating in his 1970 thesis, and in Paul Werbos’s 1974 thesis, described in his later account. The 1986 demonstration made clear why this machinery mattered for neural learning: earlier stages could learn to supply the features that later stages needed.

Our article on backpropagation explains the calculation. For this story, the important change is what became adjustable. Training could shape the transformations that produced the internal representation, as well as the final decision made from it.

The picture of connected neurons still described the network. The picture of successive learned transformations described what the network was accomplishing.

How the diagram became a program

By this point, vectors and matrices could describe connections, memories, statistical features, and transformations between layers. Someone entering the subject needed a way to understand those descriptions together.

Michael I. Jordan supplied one in the first volume of Parallel Distributed Processing, published in 1986. His contribution was chapter 9, a tutorial on the linear algebra used to analyze the volume’s models.

He taught vector spaces, inner products, and linearity, showing how simple connectionist models corresponded to operations on vectors. The reader learned a mathematical language that could be carried from one model to another.

This gives us a concrete point at which to see the connection becoming part of the curriculum. Researchers had been using matrices for decades. Now a major presentation of connectionist models included an introduction designed to teach readers how to think in those terms.

That educational shift is visible again in Christopher Bishop’s Neural Networks for Pattern Recognition, published in 1995. In the foreword, Hinton presented Bishop’s book as filling a need for a clear account built on linear algebra, calculus, and probability.

Bishop built from simpler statistical models toward multilayer networks. A newcomer could learn neural networks as part of a mathematical treatment of estimation, optimization, and generalization. The journey from a biological analogy to numerical models was increasingly built into the way the subject was taught.

Software made that language practical. Ian Nabney’s NETLAB toolbox put the matrix description directly into code. Its forward-pass routine takes a matrix of input examples. A matrix multiplication, bias addition, and nonlinear activation compute the hidden layer. Another matrix multiplication takes its activity to the output stage.

Each row holds an example, so the same operations process a whole batch through both layers. The connections in the diagram have become the coefficients in the program.

For a student learning from these books and tools, the two descriptions arrive together. There is no separate moment of discovering that a layer can be a matrix: it is already how the layer is explained and implemented.

GPUs subsequently made the organization of those operations even more consequential. In their 2009 work on large-scale unsupervised learning, Rajat Raina, Anand Madhavan, and Andrew Ng expressed computations for deep belief networks through matrix operations that parallel numerical routines could execute efficiently. Batching work and limiting transfers between CPU and GPU memory were essential to obtaining the benefit. Our CPU and GPU comparison explores that practical issue.

The algebra also offered a way to discuss architectural choices. In the 2017 Transformer paper, the authors grouped queries, keys, and values into matrices and pointed to optimized matrix multiplication as a speed and memory advantage of dot-product attention. Some of the matrices now described interactions computed from the current input, extending the vocabulary beyond fixed connection strengths.

The two pictures we started with became familiar together because researchers found so much they could do by moving between them. Treating neural activity as numbers and connections as coefficients gave weighted summation an exact algebraic description, alongside the network’s nonlinear operations. That description helped explain learning and memory, then became the language of textbooks and software. When we study the subspaces of a weight matrix today, we inherit that history: a way of seeing a network that became natural through use.