Calculating Gradients

What Is a Gradient?

A gradient measures how much the loss changes when a parameter, such as a weight, changes.

Change the Weight
       ↓
Observe the Loss
       ↓
How did the Loss change?
       ↓
Gradient

In simple terms, the gradient gives the network useful information about which direction a weight should move to help reduce the loss.

Why Do We Need Gradients?

Suppose a neural network has hundreds or thousands of weights.

Weight 1
Weight 2
Weight 3
Weight 4
Weight 5
...
Weight 1000

The network makes a prediction and gets a large loss. It now needs to know which weights need to change.

Large Loss
    ↓
Which weights caused it?
    ↓
How should they change?
    ↓
How much should they change?

Gradients provide the mathematical information needed to answer these questions.

A Simple Example

Imagine a very small model with one weight.

Input  = 2
Weight = 3

The model calculates:

Prediction = Input × Weight

Prediction = 2 × 3

Prediction = 6

Suppose the correct answer is:

Actual = 10

The prediction is too low.

Prediction = 6
Actual     = 10

We now calculate a simple squared-error loss:

Loss = (Actual - Prediction)²

Loss = (10 - 6)²

Loss = 16

The model needs to change its weight. But how do we know how the weight affects the loss?

That is where the gradient comes in.

Change the Weight and Observe the Loss

Let's increase the weight from 3 to 4.

Input  = 2
Weight = 4

Prediction = 2 × 4

Prediction = 8

Now calculate the loss:

Loss = (10 - 8)²

Loss = 4

Notice what happened:

Weight = 3
Loss   = 16


Weight = 4
Loss   = 4

Increasing the weight caused the loss to decrease significantly.

This tells us that the weight has an important effect on the loss.

Gradient Gives Direction

The gradient tells us the direction in which the loss changes when the weight changes.

Gradient
    ↓
How does Loss respond
to a Weight change?

If increasing a weight makes the loss increase, the gradient points in one direction.

If increasing a weight makes the loss decrease, the gradient points in the opposite direction.

Positive Gradient
      ↓
Increasing the weight
increases the loss


Negative Gradient
      ↓
Increasing the weight
decreases the loss

This direction is extremely important when the optimizer decides how to update the weight.

Gradient Also Gives the Size of the Effect

The gradient is not only about direction. Its magnitude tells us how strongly the loss responds to a change in the weight.

Small Gradient
      ↓
Weight has a smaller effect
on the loss


Large Gradient
      ↓
Weight has a larger effect
on the loss

For example:

Weight A
Gradient = 0.1

Small effect


Weight B
Gradient = 5.0

Much larger effect

The optimizer can use this information when updating the weights.

Gradient and Derivative

For a single weight, the gradient is essentially the derivative of the loss with respect to that weight.

Gradient = ∂Loss / ∂Weight

You can read this as:

"How much does the Loss change
when the Weight changes?"

For a neural network with many weights, we calculate a gradient for each parameter.

Weight 1 → Gradient 1
Weight 2 → Gradient 2
Weight 3 → Gradient 3
Weight 4 → Gradient 4

A Simple Gradient Calculation

Let's use an even simpler mathematical example.

Loss = Weight²

The derivative is:

dLoss / dWeight = 2 × Weight

If:

Weight = 3

then:

Gradient = 2 × 3

Gradient = 6

So the gradient at weight 3 is 6.

This tells us that the loss is increasing as the weight increases at that point.

How Does the Gradient Help Update the Weight?

The optimizer uses the gradient to decide how to change the weight.

A common update rule is gradient descent:

New Weight
=
Old Weight
-
Learning Rate × Gradient

Suppose:

Old Weight   = 3
Gradient     = 6
Learning Rate = 0.1

Then:

New Weight
= 3 - (0.1 × 6)

New Weight
= 3 - 0.6

New Weight
= 2.4

The weight moves in the direction that gradient descent expects will reduce the loss.

Why Do We Subtract the Gradient?

The gradient points in the direction of increasing loss. Gradient descent therefore moves in the opposite direction.

Gradient
   ↓
Direction of increasing Loss

Gradient Descent
   ↓
Move in the opposite direction

   ↓

Lower Loss

This is why the basic gradient-descent equation uses subtraction.

New Weight
=
Old Weight
-
Learning Rate × Gradient

What Happens With Multiple Weights?

A real neural network has many weights, so each weight gets its own gradient.

Weight 1 → Gradient 1
Weight 2 → Gradient 2
Weight 3 → Gradient 3
Weight 4 → Gradient 4
       ...
Weight N → Gradient N

The optimizer uses each gradient to update its corresponding weight.

Weight 1 → Update
Weight 2 → Update
Weight 3 → Update
Weight 4 → Update
       ...
Weight N → Update

How Does Backpropagation Calculate These Gradients?

In a deep neural network, the loss depends on many calculations across many layers.

Input
  ↓
Hidden Layer 1
  ↓
Hidden Layer 2
  ↓
Output
  ↓
Loss

Backpropagation works backward through these calculations and uses the chain rule to determine how each parameter affects the final loss.

Loss
 ↓
Output Layer
 ↓
Hidden Layer 2
 ↓
Hidden Layer 1
 ↓
Gradients

This allows the network to efficiently calculate gradients for many weights.

The Chain Rule in Simple Terms

You do not need to memorize complicated calculus yet. The basic idea is enough for now.

Imagine:

Weight
  ↓
Neuron Output
  ↓
Final Prediction
  ↓
Loss

The weight affects the neuron output, which affects the prediction, which affects the loss.

How Weight affects Neuron
          ×
How Neuron affects Prediction
          ×
How Prediction affects Loss
          ↓
How Weight affects Loss

The chain rule allows these effects to be combined.

Complete Example

Let's connect everything together.

Input = 2
Weight = 3
Target = 10

First, the forward pass:

Prediction
= Input × Weight

= 2 × 3

= 6

Calculate the squared error:

Loss
= (Target - Prediction)²

= (10 - 6)²

= 16

Now we need to know how the loss changes when the weight changes.

For this simple model:

Prediction = Input × Weight

Loss = (Target - Prediction)²

Using the chain rule, the derivative of the loss with respect to the weight is:

dLoss / dWeight
=
2 × (Prediction - Target) × Input

Substitute the values:

Gradient
= 2 × (6 - 10) × 2

= 2 × (-4) × 2

= -16

The gradient is negative. This tells us that increasing the weight would move the model in the direction of reducing the loss for this example.

Using the Gradient to Update the Weight

Suppose the learning rate is 0.1.

Old Weight   = 3
Gradient     = -16
Learning Rate = 0.1

Apply gradient descent:

New Weight
= Old Weight - Learning Rate × Gradient

= 3 - (0.1 × -16)

= 3 + 1.6

= 4.6

The weight increased because the gradient was negative.

The important point is not the particular numbers. The important idea is that the gradient tells the optimizer which direction to move the weight.

Every Weight Gets Its Own Gradient

Consider a small neural network:

Input
  ↓
Weight 1
  ↓
Hidden Neuron
  ↓
Weight 2
  ↓
Output
  ↓
Loss

During backpropagation:

Loss
 ↓
Gradient for Weight 2
 ↓
Gradient for Weight 1

The optimizer can then update both weights independently.

Weight 1 + Gradient 1
        ↓
Updated Weight 1


Weight 2 + Gradient 2
        ↓
Updated Weight 2

Real-Life Example

Imagine you are trying to reach the lowest point of a mountain while standing in fog.

You cannot see the entire mountain, but you can determine which direction the ground slopes.

Current Position
      ↓
Check the slope
      ↓
Which direction goes downward?
      ↓
Take a step
      ↓
Check again
      ↓
Repeat

The gradient is similar to the slope. It tells you the direction in which the loss increases, so gradient descent moves in the opposite direction to reduce the loss.

Complete Learning Flow

Input
  ↓
Forward Pass
  ↓
Prediction
  ↓
Calculate Loss
  ↓
Backpropagation
  ↓
Calculate Gradients
  ↓
Optimizer
  ↓
Update Weights
  ↓
Forward Pass Again
  ↓
Better Prediction

This cycle repeats during training so that the model can gradually adjust its parameters.

Remember This

Loss:
"How wrong is my prediction?"


Gradient:
"How does this weight affect my loss?"


Backpropagation:
"Calculate those gradients."


Optimizer:
"Use those gradients to update
the weights."

Once you understand this distinction, backpropagation becomes much easier to follow.