Explain like I'm five · AI Theory

What is Gradient Descent?

Gradient descent, told as a story about a teacher who measures the slope of the sorter's miss score and turns its dials, little by little, toward fewer misses.

so many dials — which way should I turn to cut the misses?the sorter's dials (parameters)MISS SCOREBOARD38 misses this roundtrying every setting is impossible, so we need the direction that cuts misses

The teacher can't see which way to turn the sorter's dials to cut the misses.Trying every dial setting is impossible, so the teacher needs the direction that cuts misses.

turned it any which wayMISS SCOREBOARD38 misses before52 misses aftermeant to cut the misses, but the count went upwithout knowing the direction, more turning can mean more misses

Turning a dial at random can even raise the number of misses.Without knowing the direction, more turning can mean more misses.

gradient descent = measure the miss score's slope, and turn the dial a little toward fewer missesmiss score (loss)stop when the slope is nearly zerodial position (parameter)turn a little away from the uphill side, and stop when the miss score stops falling

Gradient descent measures the slope of the miss score (loss) and turns the dials (parameters) a little toward a lower score, cutting the model's error bit by bit.Turn a little away from the uphill side, and stop when the miss score stops falling.

Measure the slopecalculates how the score changes as a dial turns
Turn against the slopethe score falls in the opposite direction from the slope
Set the turn size (learning rate)too big overshoots, too small is slow
Stop when it stops fallinga near-zero slope means stop
turning too far at oncekeeps overshooting the low point, so misses barely fallturning a little at a timeslow, but misses keep fallingso set the learning rate (how far each turn goes) neither too big nor too small

If each turn of the dial (the learning rate) is too big, it overshoots the low point; too small, and it takes far too long.So the learning rate is set neither too big nor too small.

this must be the setting with the fewest misses!local minimuma setting with even fewer misses lies elsewhereglobal minimumfollowing only the slope near the current spot can miss a deeper valley elsewhere

A shallow dip (a local minimum) can be mistaken for the setting with the fewest misses.Gradient descent follows only the slope near the current spot, so it can miss a deeper valley elsewhere.

In one breath

Gradient descent = measuring the slope (gradient) of the loss at the current point, then changing the parameters a little in the opposite direction, repeatedly, to reach a point of lower loss.

Gradient descent is an optimization algorithm that computes the gradient of a loss function (how wrong the model currently is), then updates the parameters a little in the opposite direction, by an amount set by the learning rate, repeatedly reducing the loss. Too large a learning rate makes the process oscillate around the minimum or diverge; too small makes convergence painfully slow. When the loss surface isn't convex, the process can settle near a local minimum or a saddle point (where the gradient is zero but it isn't a minimum) and miss the global minimum. In practice, stochastic gradient descent (SGD), which computes the gradient from only part of the data (a mini-batch), and its variants are widely used.

When grown-ups say it

Gradient Descent
Turning the dials a little at a time toward a lower miss score. A widely used way to train neural networks. → The Eye That Scans Through a Small Window
Gradient
Which way, and how fast, the miss score changes as a dial turns. The score falls in the opposite direction from the gradient.
Learning Rate
How far the dial turns in a single step. Too big overshoots; too small takes too long.
Loss Function
The miss score showing how wrong the sorter currently is. Lowering this score is the goal of training; logistic regression can be trained this way too. → Feeding the Scale's Number Through an S-Curve
Local Minimum
A shallow dip: lowest nearby, but maybe not lowest overall. Stop here, and the deeper valley goes unfound.
Global Minimum
The setting with the truly lowest miss score on the whole curve. If the loss surface isn't convex, gradient descent isn't guaranteed to find it.
Stochastic Gradient Descent (SGD)
Measuring the slope from a sample of the produce instead of all of it. Each update is much faster than computing over all the data.
Convergence
The dial's moves shrinking until it settles at one setting. Usually taken as the sign that training can stop.
Next story
A Sorter Trained on Rotten Seed →
Want another term as a picture book? Request →