Backpropagation
Backpropagation is the process a neural network uses to calculate how much each weight contributed to the prediction error. These gradients are then used to update the weights and improve the model.
Backpropagation works backward through a neural network to calculate gradients.
The network first makes a prediction and calculates the loss. Backpropagation then moves backward through the network and calculates how each parameter contributed to that loss.
What Is Backpropagation?
The word "backpropagation" means propagating information backward through the neural network.
The network first moves forward to make a prediction. After calculating the loss, it works backward to calculate gradients.
Forward Pass Input ↓ Neural Network ↓ Prediction ↓ Loss Backward Pass Loss ↓ Gradients ↓ Weights
This forward-and-backward process is repeated during training.
Why Do We Need Backpropagation?
A neural network can contain thousands, millions, or even billions of weights.
When the model makes a wrong prediction, we need to know which weights should change and by how much.
Prediction is wrong
↓
Which weight caused the error?
↓
How much did it contribute?
↓
Calculate gradients
↓
Update weights
Backpropagation provides an efficient way to calculate this information.
Forward Pass Comes First
Before backpropagation can happen, the neural network must make a prediction.
This is called the forward pass.
Input ↓ Layer 1 ↓ Layer 2 ↓ Output ↓ Prediction
Each layer performs mathematical calculations using weights, biases, and activation functions.
Calculate the Loss
After the model makes its prediction, we compare the prediction with the correct answer.
Actual value = 10 Prediction = 8 Error = 2 Loss = 4
The loss tells us how wrong the model was.
But we still don't know how each weight contributed to that loss. That is where backpropagation begins.
Backward Pass
Once the loss has been calculated, the network moves backward through its layers.
Loss ↓ Output Layer ↓ Hidden Layer ↓ Input Layer
At each step, the network calculates gradients that tell us how changes in the parameters affect the loss.
What Is Being Calculated?
The main question is:
"How much does this weight affect the loss?"
Mathematically, this is represented using a partial derivative.
∂Loss ────── ∂w
This means: how much the loss changes when the weight w changes.
A Very Simple Example
Imagine a very small neural network with one input and one weight.
Input = 2 Weight = 3 Prediction = Input × Weight = 2 × 3 = 6
Suppose the correct answer is 10.
Actual = 10 Prediction = 6 Error = 4
The model is making a significant error. Backpropagation calculates how the weight contributed to this error.
Why Does It Work Backward?
The final prediction depends on the output layer. The output depends on the hidden layers. The hidden layers depend on earlier layers.
Input ↓ Weight 1 ↓ Hidden Layer ↓ Weight 2 ↓ Output ↓ Loss
To understand how the loss depends on an earlier weight, we need to trace the chain of dependencies backward.
This is why backpropagation works from the output toward the earlier layers.
The Chain Rule
Backpropagation relies heavily on the mathematical chain rule.
The chain rule allows us to calculate how one value affects another through multiple intermediate steps.
A affects B B affects C C affects Loss Therefore: A affects Loss through the chain A → B → C → Loss
In a neural network, there can be many such connections. The chain rule lets us calculate their combined effect.
Simple Chain Rule Example
Suppose:
y = x² z = y + 1
Here, x affects y, and y affects z.
The chain rule tells us:
dz/dx = dz/dy × dy/dx
Backpropagation uses this same mathematical idea across the layers of a neural network.
Backpropagation With Multiple Weights
Real neural networks have many weights. Backpropagation calculates a gradient for each one.
Weight 1 Gradient = 0.5 Weight 2 Gradient = -0.8 Weight 3 Gradient = 0.2 Weight 4 Gradient = 1.4
Each gradient tells the optimizer how that particular weight should be adjusted.
From Gradient to Weight Update
Backpropagation calculates the gradients. An optimization algorithm such as gradient descent uses those gradients to update the weights.
Backpropagation
↓
Calculate gradients
↓
Gradient Descent
↓
Update weights
These are related concepts, but they are not the same thing.
Simple Weight Update Example
Suppose backpropagation calculates:
Weight = 5 Gradient = 2 Learning rate = 0.1
Gradient descent can update the weight:
New weight = Old weight - Learning rate × Gradient = 5 - (0.1 × 2) = 4.8
Backpropagation calculated the gradient. Gradient descent used that gradient to change the weight.
Complete Neural Network Training Process
Now we can connect all the concepts from this lesson.
Input Data
↓
Forward Pass
↓
Prediction
↓
Loss Function
↓
Calculate Loss
↓
Backpropagation
↓
Calculate Gradients
↓
Gradient Descent
↓
Update Weights
↓
Repeat
This process happens repeatedly during training.
Why Backpropagation Is Important in AI
Modern neural networks can contain an enormous number of parameters.
Manually checking how every parameter affects the loss would be impractical.
Millions of parameters
↓
Backpropagation
↓
Calculate gradients efficiently
↓
Optimizer
↓
Update parameters
This efficient calculation of gradients is one of the key reasons neural networks can be trained at large scale.
A Real-World Example
Imagine an AI model that predicts whether an image contains a cat.
Image
↓
Neural Network
↓
Prediction
"Cat = 0.30"
Actual answer:
"Cat = 1"
↓
Large Loss
↓
Backpropagation
↓
Calculate gradients
↓
Update weights
↓
Next prediction
After many training examples, the model can learn weights that produce better predictions.
Backpropagation in Simple Words
Think of a student solving a problem and getting the answer wrong.
Student gives wrong answer
↓
Find the mistake
↓
Trace where the mistake came from
↓
Understand what should change
↓
Try again
Backpropagation follows a similar idea for a neural network. It traces the prediction error backward and calculates how the model's parameters contributed to that error.
Backpropagation vs Gradient Descent
These two concepts are often confused, so keep them separate.
Backpropagation "What gradients do I have?" Gradient Descent "What should I do with those gradients?"
Backpropagation calculates the gradients. Gradient descent uses those gradients to update the parameters.
Backpropagation vs Loss Function
The loss function and backpropagation also have different jobs.
Loss Function "How wrong is the prediction?" Backpropagation "How did each weight contribute to that loss?"
Together they provide the information needed to improve the model.
The Big Picture
At this point, you can connect the major mathematical concepts behind neural network training.
Activation Functions
↓
Prediction
↓
Loss Function
↓
How wrong?
↓
Backpropagation
↓
Calculate Gradients
↓
Gradient Descent
↓
Update Weights
↓
Better Prediction
This cycle is repeated over and over during training.
Backpropagation calculates how each weight contributes to the loss.
The neural network first makes a prediction during the forward pass. The loss function measures the error. Backpropagation then works backward through the network using the chain rule to calculate gradients. An optimizer such as gradient descent uses those gradients to update the weights.