Blame travels backward

When a neural network makes a mistake, training works out how much each dial (each weight) was to blame. That blame starts at the answer and travels backward, layer by layer, so every dial knows which way to turn. This is backpropagation.

On the way back, the blame passes through each layer's bend, the activation function. At every layer it gets multiplied by that bend's slope. This is the chain rule: one slope per layer, all multiplied together. In a deep network, that long chain of multiplications can shrink to almost nothing, or grow out of control.

A quarter at a time

Older networks used an S-shaped bend, the sigmoid: σ(s) = 1 / (1 + e⁻ˢ). Its slope is σ(s)(1 − σ(s)). That slope is steepest in the middle, at s = 0, where it is exactly 0.25. Everywhere else it is smaller, and far from the middle it is almost zero (the lecture calls these neurons saturated: they "kill the gradient flow").

So each sigmoid layer passes back at most a quarter of the blame. Ten layers in a row, even at their steepest, pass back 0.25¹⁰ = 0.00000095 of it. That is less than a millionth.

We measured it in a real network: 10 layers of 256 sigmoid neurons with a good (Xavier) start, 512 random inputs and targets, and a squared-error mistake score (numpy's default_rng(0)). Here is the blame that reached each layer's dials, compared with layer 10:

  • Layer 10: 1
  • Layer 9: 0.19
  • Layer 8: 0.051
  • Layer 7: 0.013
  • Layer 6: 0.003
  • Layer 5: 0.00069
  • Layer 4: 0.00015
  • Layer 3: 0.000035
  • Layer 2: 0.0000079
  • Layer 1: 0.00000031

The blame shrinks plane by plane: layer 1 gets 0.0000003 of what layer 10 gets.

The blame shrinks plane by plane: layer 1 gets 0.0000003 of what layer 10 gets.

The first layer gets about three ten-millionths of the blame, so its dials barely move. This is the vanishing gradient problem. Sepp Hochreiter worked it out in his 1991 diploma thesis at the Technical University of Munich, long before deep learning took off. Bengio, Simard and Frasconi analysed the same problem in 1994.

Sepp Hochreiter's 1991 diploma thesis, "Untersuchungen zu dynamischen neuronalen Netzen" (Studies of dynamic neural networks).

Sepp Hochreiter's 1991 diploma thesis, "Untersuchungen zu dynamischen neuronalen Netzen" (Studies of dynamic neural networks).

Too big: exploding blame

The chain can also go the other way. If the dials start too big, the blame grows at every layer instead of shrinking. We took a healthy ReLU network and made every dial 2× too big. The blame reaching layer 1 was about 500,000× bigger than in the healthy network (512,003× when measured). With dials just 1.5× too big, it was still about 2,000× bigger.

Huge blame means huge steps. The lecture warns that one giant step can throw the dials into a terrible spot with a large mistake, and in the worst case the numbers overflow to Inf or NaN, and training has to restart from an earlier checkpoint. The lecture also points out that this is not only a problem for recurrent networks: it hits all very deep networks.

Gradient clipping: a cap line, and every layer's blame scaled down to fit under it.

Gradient clipping: a cap line, and every layer's blame scaled down to fit under it.

The fixes

  • ReLU. ReLU is max(0, s). When a neuron is on, its slope is exactly 1, so it passes the blame back at full strength. In the same 10-layer network with ReLU and a He start, the blame stayed between 0.4 and 2.1 of layer 10's at every layer (layer 1: 0.42). It helps a lot, but it is not a cure-all: a neuron that is off passes back nothing, and the dials can still start too big.
  • Good starting dials. Starting the dials at the right size keeps the blame level from layer to layer (Deep Learning, Visually #18).
  • Batch normalization. It keeps each layer's values in a healthy range during training (coming in #20).
  • Skip connections. Shortcuts that let the blame jump over layers, used in ResNets (coming in #35).
  • Gradient clipping, for exploding blame. If the total size of the blame is bigger than a limit, scale all of it down to the limit. The direction stays the same; only the step gets smaller.

ReLU: the blame reaches the first layers at full strength.

ReLU: the blame reaches the first layers at full strength.

Try it in PyTorch

Use ReLU layers with a He start, and clip the blame before every step:

import torch import torch.nn as nn layer = nn.Linear(256, 256) nn.init.kaiming_normal_(layer.weight, nonlinearity="relu") model = nn.Sequential(layer, nn.ReLU()) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) optimizer.step()

Or watch the blame vanish yourself. This sends one number through ten bends and asks how much blame comes back:

import torch as t for f in(t.sigmoid,t.relu): x=t.ones(1).requires_grad_() y=x for _ in'x'*10:y=f(y) y.backward();print(x.grad)

It prints tensor([2.7848e-07]) for the sigmoid and tensor([1.]) for ReLU. Change 10 to 30: how small does the sigmoid's blame get now?

References

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.