Following the Gradient Downhill | Visual AI Learning Guide

Watch the loss curve as training proceeds. Each step computes gradients by backpropagation, which tells every parameter how the loss would change if that parameter moved slightly. The optimizer then nudges each parameter in the opposite direction, scaled by the learning rate. Early on the loss drops quickly because the parameters are far from good values; later the curve flattens as the model refines. Notice that each update uses a batch, a small group of examples, so the step is an estimate of the true gradient and the curve is a little noisy. An epoch is one full pass over the data. If the learning rate were much larger, the curve would jump around or even climb; if much smaller, it would crawl. The downward trend across batches and epochs is the model learning to predict the corpus.

0:00 / 0:00

Auto-play

Once the loss is computed, training adjusts the model's parameters to reduce it. Backpropagation applies the chain rule through the network to compute, for every parameter, the partial derivative of the loss with respect to that parameter. This gradient points in the direction of steepest increase of the loss, so moving each parameter a small step opposite the gradient reduces the loss. Gradient descent expresses this as an update: each parameter \(\theta\) becomes \(\theta - \eta \, \nabla_\theta L\), where \(\eta\) is the learning rate and \(\nabla_\theta L\) is the gradient of the loss with respect to that parameter.

The learning rate controls the size of each step. If it is too large, the loss can oscillate or diverge; if it is too small, training takes impractically long. In practice the update is computed from a batch, a small group of examples, rather than the entire dataset, so each step is an estimate of the true gradient. An epoch is one full pass over the training data. Across many batches and epochs the loss trends downward as the parameters settle into values that predict the corpus well.