One matrix multiplication, two views

A matrix has three rows and four columns. In the row view, each row takes a dot product with the input and produces one output coordinate. In the column view, each of the four input coordinates scales a column, and we add those contributions.

Why do both procedures produce an output in R3\mathbb R^3? What happens to the four contributions in the column view?

You will need vector addition, multiplication by a scalar, and the dot product. The matrix-equations lesson introduces both views while solving systems; here we will follow one multiplication closely. Throughout, inputs and outputs are column vectors, and Rn\mathbb R^n means vectors with nn real coordinates.

Predict the shape before calculating

Keep this matrix as our reference:

W=[120301212014],h=[abcd].W=\begin{bmatrix} 1&2&0&3\\ 0&1&2&1\\ 2&0&1&4 \end{bmatrix},\qquad h=\begin{bmatrix}a\\b\\c\\d\end{bmatrix}.

Start with h=(1,1,1,1)⊤h=(1,1,1,1)^\top. Before opening the calculation, write your predictions:

  1. How many numbers will the row calculation produce?
  2. How many coordinates does one column have? How many will the sum of four scaled columns have?
  3. What is the output? Calculate it once using rows and once using columns.

Record the three row results and all four column contributions. Compare the outputs and explain why their sizes agree before reading on.

Compare the two calculations

The row view produces three scalars, which we stack into one vector:

Wh=[1+2+0+30+1+2+12+0+1+4]=[647].Wh=\begin{bmatrix} 1+2+0+3\\0+1+2+1\\2+0+1+4 \end{bmatrix} =\begin{bmatrix}6\\4\\7\end{bmatrix}.

The column view produces four vectors, which we add coordinate by coordinate:

Wh=[102]+[210]+[021]+[314]=[647].Wh= \begin{bmatrix}1\\0\\2\end{bmatrix} +\begin{bmatrix}2\\1\\0\end{bmatrix} +\begin{bmatrix}0\\2\\1\end{bmatrix} +\begin{bmatrix}3\\1\\4\end{bmatrix} =\begin{bmatrix}6\\4\\7\end{bmatrix}.

Each column already has three coordinates. Adding vectors adds their matching coordinates; it does not place their entries end to end. Four contributions therefore still give one three-coordinate output.

Change one input coordinate

Keep WW fixed. From the reference h=(1,1,1,1)⊤h=(1,1,1,1)^\top, change only bb from 1 to 2. Before changing the control, predict which output coordinates will change and by how much. Which column can explain the entire difference?

In the explorer, record the reference, then change Input b to 2. Record the new row results, the four column contributions, and the output difference.

Change an input coordinate

W stays fixed. Each control changes one coordinate of h.

1
1
1
1
Current W: three rows, four columns
OutputFrom aFrom bFrom cFrom d
y11203
y20121
y32014

Row view: three scalar results

y1 = 1 × 1 + 2 × 1 + 0 × 1 + 3 × 1 = 6

y2 = 0 × 1 + 1 × 1 + 2 × 1 + 1 × 1 = 4

y3 = 2 × 1 + 0 × 1 + 1 × 1 + 4 × 1 = 7

Column view: four vector contributions

Column 1 × a

1 × (1, 0, 2)

= (1, 0, 2)

Three output coordinates
Column 2 × b

1 × (2, 1, 0)

= (2, 1, 0)

Three output coordinates
Column 3 × c

1 × (0, 2, 1)

= (0, 2, 1)

Three output coordinates
Column 4 × d

1 × (3, 1, 4)

= (3, 1, 4)

Three output coordinates

Add matching coordinates of the four contributions:

y1 = 1 + 2 + 0 + 3 = 6

y2 = 0 + 1 + 2 + 1 = 4

y3 = 2 + 0 + 1 + 4 = 7

Current output: (6, 4, 7)

Reference output: (6, 4, 7)

Change from reference: (0, 0, 0)

Tuples list vector coordinates in order. Reset restores the original W and h = (1, 1, 1, 1). Record your prediction before changing a control, then record both calculations.

Use this record in a notebook or text file:

CaseThree row resultsFour column contributionsSum of contributionsChange from reference
h=(1,1,1,1)⊤h=(1,1,1,1)^\top…………
Only bb becomes 2…………

Compare your prediction with both calculations. Why can changing one input affect several outputs? Why does the third output stay fixed in this comparison? Write your explanation before revealing ours.

Follow the extra copy of column 2

The output changes from (6,4,7)⊤(6,4,7)^\top to (8,5,7)⊤(8,5,7)^\top. The difference is

Δy=[210].\Delta y=\begin{bmatrix}2\\1\\0\end{bmatrix}.

In the column view, only the second contribution changes: 1(2,1,0)⊤1(2,1,0)^\top becomes 2(2,1,0)⊤2(2,1,0)^\top. The other three contributions stay fixed.

In the row view, the first row receives an extra 22, the second an extra 11, and the third an extra 00. Those are exactly the entries of column 2.

More generally, increasing only input coordinate hjh_j by δ\delta changes the output by δwj\delta w_j, where wjw_j is column jj. The change may affect several output coordinates, but a zero in that column leaves the corresponding output unchanged.

See the same arithmetic in both views

For arbitrary input coordinates, the row view gives

Wh=[a+2b+3db+2c+d2a+c+4d].Wh=\begin{bmatrix} a+2b+3d\\ b+2c+d\\ 2a+c+4d \end{bmatrix}.

The column view gives

Wh=a[102]+b[210]+c[021]+d[314].Wh= a\begin{bmatrix}1\\0\\2\end{bmatrix} +b\begin{bmatrix}2\\1\\0\end{bmatrix} +c\begin{bmatrix}0\\2\\1\end{bmatrix} +d\begin{bmatrix}3\\1\\4\end{bmatrix}.

Look at just the second coordinate of this sum. The columns contribute 0a0a, bb, 2c2c, and dd. Adding them produces b+2c+db+2c+d, exactly the second row’s dot product. The same reasoning applies to every output coordinate.

This is why the agreement holds for every input, including negative and zero coordinates. The experiments illustrated it; matching the terms establishes it generally. Switching views does not transpose WW or change the transformation.

View of an m×nm\times n matrixWhat you calculateHow you form the output
Rowsmm dot products, each using all nn inputsStack the mm scalar results
Columnsnn scaled vectors, each with mm coordinatesAdd matching coordinates to get one vector in Rm\mathbb R^m

For our 3×43\times4 matrix, each row has four entries so it can pair with the input. Each column has three entries so it can contribute to the output. Four input coefficients control four contributions in the same three-dimensional space.

Write the equivalence for any matrix

Let WijW_{ij} mean the entry in row ii, column jj. The row view computes

yi=∑j=1nWijhj.y_i=\sum_{j=1}^{n}W_{ij}h_j.

The summation symbol means to add one term for each input coordinate. In the column view, y=∑j=1nhjwjy=\sum_{j=1}^{n}h_jw_j. Coordinate ii of column wjw_j is WijW_{ij}, so coordinate ii of that vector sum is the same sum of numbers.

There are mm choices of output coordinate ii, regardless of how many contributions nn we add.

Change one row instead

Return to the reference h=(1,1,1,1)⊤h=(1,1,1,1)^\top and the original WW. This is a separate comparison: keep the input and rows 1 and 3 fixed, and double row 2 from (0,1,2,1)(0,1,2,1) to (0,2,4,2)(0,2,4,2).

Predict the new output before changing Second row scale from 1 to 2 in the explorer below. Which output coordinates can change? In the column view, which entries of the contributions change?

Scale the second row

h = (1, 1, 1, 1) stays fixed. Only the second row of W changes.

1
Current W: three rows, four columns
OutputFrom aFrom bFrom cFrom d
y11203
y20121
y32014

Row view: three scalar results

y1 = 1 × 1 + 2 × 1 + 0 × 1 + 3 × 1 = 6

y2 = 0 × 1 + 1 × 1 + 2 × 1 + 1 × 1 = 4

y3 = 2 × 1 + 0 × 1 + 1 × 1 + 4 × 1 = 7

Column view: four vector contributions

Column 1 × a

1 × (1, 0, 2)

= (1, 0, 2)

Three output coordinates
Column 2 × b

1 × (2, 1, 0)

= (2, 1, 0)

Three output coordinates
Column 3 × c

1 × (0, 2, 1)

= (0, 2, 1)

Three output coordinates
Column 4 × d

1 × (3, 1, 4)

= (3, 1, 4)

Three output coordinates

Add matching coordinates of the four contributions:

y1 = 1 + 2 + 0 + 3 = 6

y2 = 0 + 1 + 2 + 1 = 4

y3 = 2 + 0 + 1 + 4 = 7

Current output: (6, 4, 7)

Reference output: (6, 4, 7)

Change from reference: (0, 0, 0)

Tuples list vector coordinates in order. Reset restores the original W and h = (1, 1, 1, 1). Record your prediction before changing a control, then record both calculations.

Record the three row results, all four column contributions, and the difference from the reference. Explain how several changed column contributions can affect just one output coordinate before opening the explanation.

Follow the second coordinate through every column

The output becomes (6,8,7)⊤(6,8,7)^\top: only the second row’s dot product doubles. In the column view, the new contributions are

[102],[220],[041],[324].\begin{bmatrix}1\\0\\2\end{bmatrix},\quad \begin{bmatrix}2\\2\\0\end{bmatrix},\quad \begin{bmatrix}0\\4\\1\end{bmatrix},\quad \begin{bmatrix}3\\2\\4\end{bmatrix}.

Only second coordinates can change. Their sum becomes 0+2+4+2=80+2+4+2=8; the first and third coordinate sums remain 6 and 7. The zero in column 1 stays zero even when doubled.

For any fixed input, changing only row ii can change only output coordinate ii. It does not guarantee a change: the new row might have the same dot product with that particular input.

Design your own comparison

Choose an input coordinate other than bb and your own nonzero increase or decrease. In the first explorer, use Reset and keep WW and the other inputs fixed. You can use paper for values outside the controls’ range.

Write the reference, your chosen change, and a prediction of the entire output difference. State what observation would contradict your prediction. Then calculate both views, record the difference, and explain the result using one column. Include a negative or zero input in your chosen case.

For a second challenge on paper, choose an input and a genuine change to one row that leaves the output unchanged. Predict why the change will be invisible, then calculate the before and after outputs. Explain what your example says about the claim “changing a row always changes its output.”

Hints and checks for your investigation

For the input experiment, multiply the selected column by the change in the input, not its new value. For example, changing cc from 1 to −1-1 gives δ=−2\delta=-2, so the output difference is −2(0,2,1)⊤=(0,−4,−2)⊤-2(0,2,1)^\top=(0,-4,-2)^\top.

For the row experiment, choose a row difference whose dot product with your input is zero. With h=(1,1,1,1)⊤h=(1,1,1,1)^\top, adding (1,−1,0,0)(1,-1,0,0) to one row changes its weights but leaves its dot product unchanged. Alternatively, changing a weight attached to a zero input makes that change invisible for this input.

Your record should show equal row and column results in both states. The row experiment proves that a changed matrix can agree with the original on a particular input; it does not mean the two matrices agree on every input.

What a column means in a neural network

Suppose hh contains four hidden activations and WhWh supplies three outputs. Row ii collects the weights used to compute receiving neuron ii‘s weighted sum. Column jj collects the outgoing weights from source activation hjh_j to all three receiving neurons.

For example, when h2=5h_2=5, its contribution is

5[210]=[1050].5\begin{bmatrix}2\\1\\0\end{bmatrix} =\begin{bmatrix}10\\5\\0\end{bmatrix}.

That is one source activation’s contribution to the three outputs. The full output also includes the contributions from the other source activations. Both views apply to this same layer.

Setting just hj=1h_j=1 and all other inputs to zero isolates column jj. This input is the standard unit vector eje_j, so Wej=wjWe_j=w_j: each column is where one input basis vector lands. The input basis vector has four coordinates; its image has three.

If a layer adds bias, add that vector after the column sum. If it applies a nonlinear activation, apply it to the resulting weighted sums; in general, applying it separately to each column contribution and adding gives a different result.

Transfer to a taller matrix

Try a new shape without the explorer:

B=[120−130−21],x=[2−1].B=\begin{bmatrix}1&2\\0&-1\\3&0\\-2&1\end{bmatrix},\qquad x=\begin{bmatrix}2\\-1\end{bmatrix}.

Before calculating, predict the input and output dimensions, the number of row dot products, and the number of column contributions. Compute BxBx both ways. Then predict and check the output difference if only the second input changes from −1-1 to 0.

Check the new shape and the change

The map takes R2\mathbb R^2 to R4\mathbb R^4. Four row dot products give (0,1,6,−5)⊤(0,1,6,-5)^\top. The two column contributions are

2[103−2]−[2−101]=[016−5].2\begin{bmatrix}1\\0\\3\\-2\end{bmatrix} -\begin{bmatrix}2\\-1\\0\\1\end{bmatrix} =\begin{bmatrix}0\\1\\6\\-5\end{bmatrix}.

Increasing the second input by 1 adds the second column, (2,−1,0,1)⊤(2,-1,0,1)^\top, giving (2,0,6,−4)⊤(2,0,6,-4)^\top. The third output stays fixed because the third entry of that column is zero.

Finally, explain in your own words: Why do four columns in R3\mathbb R^3 produce a three-coordinate output, while two columns in R4\mathbb R^4 produce a four-coordinate output? Use the difference between counting contributions and counting coordinates.

Continue with what makes a function linear to investigate the rules behind these combinations, or what a weight matrix actually does to follow them through a neural network.