in the last piece i did autograd, and the whole point of it was that you call loss.backward() once and pytorch fills in every gradient for you, and a couple of people basically said the same thing back to me, ok that is nice but on a real network with more than one layer what is it actually doing in there. so this time i want to do the thing you will genuinely never do inside a training loop, i want to take a two-layer network and compute every gradient by hand, one node at a time, and then type the same thing into pytorch and watch autograd land on the exact same numbers. the one-neuron version was the warm-up. this is the one where the chain rule has to travel back through a hidden layer, and where the interesting failures live.
The network, small enough to hold in your head
Two layers. one input, a hidden neuron with a ReLU, then an output neuron, then a squared-error loss. that is it. i am going to write it as plain scalars first because the whole trick is visible that way and nothing is hiding inside a matrix.
x ──▶ h1 = w1·x + b1 ──▶ a1 = ReLU(h1) ──▶ h2 = w2·a1 + b2 ──▶ L = (h2 − y)²
four parameters to find gradients for: w1, b1, w2, b2. and i am picking numbers so there is nothing to hide behind:
x = 1.0
w1 = 2.0 b1 = 0.0
w2 = 3.0 b2 = 1.0
y = 2.0 (the target)
Forward pass, left to right, writing everything down
you compute the answer first, and you keep every intermediate value, because the backward pass needs them. this is not optional bookkeeping, the local derivatives literally reuse these numbers.
h1 = w1·x + b1 = 2·1 + 0 = 2.0
a1 = ReLU(h1) = max(0, 2) = 2.0
h2 = w2·a1 + b2 = 3·2 + 1 = 7.0
L = (h2 − y)² = (7 − 2)² = 25.0
loss is 25. now the only question that matters is how each of the four parameters should move to make that 25 smaller, and that is exactly what a gradient is, ∂L/∂w1 and so on, the slope of the
loss with respect to that one number while everything else is held still.
Backward pass, one node at a time, right to left
the entire method is this, and it does not get more complicated no matter how deep the net is: at each node you take the gradient arriving from the right, and you multiply it by that node's own
local derivative, and you pass the result further left. that is the chain rule and there is nothing else to it.
loss → h2. L = (h2 − y)², so the derivative of the loss with respect to h2 is 2(h2 − y).
∂L/∂h2 = 2·(7 − 2) = 10.0
h2 → the output parameters. h2 = w2·a1 + b2. the derivative of h2 with respect to w2 is just a1, and with respect to b2 it is 1, so multiply each by the 10 coming in:
∂L/∂w2 = 10 · a1 = 10 · 2 = 20.0
∂L/∂b2 = 10 · 1 = 10.0
so far identical in spirit to the one-neuron case. here is the part that only shows up once you have a hidden layer.
h2 → a1, i.e. keep going left into the first layer. h2 = w2·a1 + b2, so the derivative of h2 with respect to a1 is w2. the gradient does not stop at the output layer, it flows backthrough w2 into the hidden activation:
∂L/∂a1 = 10 · w2 = 10 · 3 = 30.0
a1 → h1, through the ReLU. ReLU passes its input when it was positive and its derivative is 1 there, 0 when the input was negative. our h1 was 2, positive, so the gate is open:
∂a1/∂h1 = 1 (because h1 > 0)
∂L/∂h1 = 30 · 1 = 30.0
h1 → the first-layer parameters. h1 = w1·x + b1, derivative with respect to w1 is x, with respect to b1 is 1:
∂L/∂w1 = 30 · x = 30 · 1 = 30.0
∂L/∂b1 = 30 · 1 = 30.0
done, by hand, four gradients:
∂L/∂w1 = 30 ∂L/∂b1 = 30 ∂L/∂w2 = 20 ∂L/∂b2 = 10
notice the shape of what happened. the 10 computed at the output got carried all the way back to the first layer, multiplied by w2 on the way through, then by the ReLU's 1, then by x. every
gradient in the network is that same number from the loss, multiplied by a chain of local slopes between it and the parameter. deeper nets are just longer chains.
Now ask PyTorch the same question
same numbers, same five-ish lines, mark the four parameters as needing gradients, run forward, call backward once, print:
import torch
x = torch.tensor(1.0)
w1 = torch.tensor(2.0, requires_grad=True)
b1 = torch.tensor(0.0, requires_grad=True)
w2 = torch.tensor(3.0, requires_grad=True)
b2 = torch.tensor(1.0, requires_grad=True)
y = torch.tensor(2.0)
h1 = w1 * x + b1
a1 = torch.relu(h1)
h2 = w2 * a1 + b2
loss = (h2 - y) ** 2
loss.backward()
print(f"loss = {loss.item()}") # 25.0
print(f"dL/dw1 = {w1.grad.item()}") # 30.0
print(f"dL/db1 = {b1.grad.item()}") # 30.0
print(f"dL/dw2 = {w2.grad.item()}") # 20.0
print(f"dL/db2 = {b2.grad.item()}") # 10.0
all four match, and the loss matches. autograd traced the same chain of operations, stored the same intermediate values on the forward pass, and multiplied the same local derivatives on the way back.
the only thing it did that you did not is bookkeeping, it just did it without asking you and it would do it the same way with ten million parameters instead of four.
if you read nothing else in this article, run that block and put it next to the hand calculation above. the whole point is the moment where the pen and the computer agree.
The part that turns this into a debugging skill
keep the network the same but break the hidden neuron on purpose. flip w1 to -2 and leave everything else. now the first layer produces a negative pre-activation:
h1 = w1·x + b1 = -2·1 + 0 = -2.0
a1 = ReLU(-2) = 0.0
h2 = w2·a1 + b2 = 3·0 + 1 = 1.0
L = (1 − 2)² = 1.0
run backward on this one and look at what comes out:
w1 = torch.tensor(-2.0, requires_grad=True) # the only change
b1 = torch.tensor(0.0, requires_grad=True)
w2 = torch.tensor(3.0, requires_grad=True)
b2 = torch.tensor(1.0, requires_grad=True)
x, y = torch.tensor(1.0), torch.tensor(2.0)
h1 = w1 * x + b1
a1 = torch.relu(h1)
h2 = w2 * a1 + b2
loss = (h2 - y) ** 2
loss.backward()
print(w1.grad.item(), b1.grad.item()) # 0.0 0.0
print(w2.grad.item(), b2.grad.item()) # 0.0 -2.0
three of the four gradients are zero, and the loss is not, the network is wrong and mostly getting no signal about it. and because you just did the hand version you can say exactly why each zero
happened instead of guessing.
w1 and b1 are zero because the ReLU's derivative is 0 when its input was negative, and that 0 sits in the middle of the chain, so everything behind it gets multiplied down to nothing. that is
the dead-ReLU problem and this is it in miniature, a neuron whose input is negative gets no gradient and stops learning, quietly, no error printed.
but look at w2, it is also zero, and the ReLU is not the reason. ∂L/∂w2 = ∂L/∂h2 · a1, and a1is 0 here, so the weight that reads from the dead neuron gets a zero gradient too, because you
cannot learn how to weight an input that is always zero. so a single dead neuron in the hidden layer takes out both the parameters feeding it and the weight reading out of it.
b2 is the survivor, ∂L/∂b2 = ∂L/∂h2 = -2, the output bias sits after the dead neuron so the zero never reaches it, and it keeps learning. that is the whole reason this is worth doing on two
layers and not one, on one neuron you just see "gradient is zero", on two you see the zero spread along the chain and stop where the chain does.
this is the payoff of the exercise. you will never backprop by hand in a real loop, autograd owns that. but the day a model just sits there and does not improve and you print a gradient and it is 0,
you will not be staring at it, you will go straight to asking what on the path back is multiplying by zero, a dead ReLU, a detach you forgot about, an activation that saturated, and you will know that
a zero at one node zeros everything behind it but not in front of it, because you have already watched it happen on four parameters you could hold in your head.
Two things that may be confusing at first glance
the gradient does not stop at the layer that produced the loss. the most common beginner mental model is that each layer computes its own gradient locally and independently. it does not. the 10
from the output travelled all the way to w1. every layer's gradient depends on every layer after it, that dependence is the · w2 step, and it is why the order is strictly right to left and why you cannot parallelise the backward pass across depth the way you can the forward one.
gradients accumulate, they do not overwrite. if you run loss.backward() twice without clearing,
the second set of numbers gets added onto the first and you will read 60, 60, 40, 20 on the second call and wonder what happened. that is deliberate, it is what lets you sum gradients over several mini-batches, but it means optimizer.zero_grad() before every backward is load-bearing, not boilerplate. i wrote about why that is in the autograd piece.
Try it before you close the tab
take the healthy network again, x=1, w1=2, b1=0, w2=3, b2=1, y=2, and this time add a second hidden step of your own, say multiply a1 by another weight w3 = 0.5 before the output. write out
the full chain for ∂L/∂w1 on paper, there is just one more factor in it now, then check against autograd. it takes a few minutes and it is worth more than rereading this.
then the mean one, set b1 = -5 so the hidden ReLU dies, predict which of the four gradients go to zero before you run it, and see if you called it right. if you can predict the zeros you have
actually got the chain rule, if you cannot you have more to gain from running it than from any amount of reading.
what is the worst "model just would not learn and the gradient was zero" you have hit? for me the first real one was a dead ReLU exactly like this, a whole layer of them, and it cost me most of a day before i thought to print a gradient. i am curious whether that is the usual first one or whether everyone finds their own way to multiply by zero.
This is one chapter's worth of an idea from my book, PyTorch From Ground Up, which builds everything from tensors upward so nothing stays vague. If it helped:
8 chapters are free, no email required, there's a free one-page tensor cheat-sheet here, every example runs in
the companion notebooks on GitHub, and the full book is on Leanpub or in
paperback and Kindle on Amazon.
More in this series
How Training Actually Works, the part where the training loop stops being magic:
The shape mechanics underneath all of it, worth having solid first:
Coming next in How Training Actually Works:
- What an optimizer actually does with the gradient
