Gradients

What Is a Gradient?

Before understanding gradients, remember what we learned about derivatives.

A derivative tells us how quickly one value changes when another value changes.

A gradient is the same basic idea extended to situations where there can be multiple values changing at the same time.

Derivative
→ How one value changes

Gradient
→ How multiple values affect the output

This becomes extremely useful in neural networks because a model can contain thousands, millions, or even billions of parameters.

Why Do We Need Gradients?

A neural network makes predictions, calculates a loss, and then needs to know how to improve.

The problem is that a neural network can have many weights. We need a mathematical way to know which weights should change and in which direction.

Prediction
    ↓
Calculate Loss
    ↓
Calculate Gradients
    ↓
Change Weights
    ↓
Better Prediction

Gradients provide the information needed for the weight update.

A Simple Example

Imagine you are standing on a hill and want to reach the lowest point.

        Hill

        /\
       /  \
      /    \
     /      \
    /        \
___/__________\___
        ↓
      Lowest
      Point

You need to know which direction goes downward.

A gradient gives you information about the direction in which the value increases most quickly. To reduce the loss, we move in the opposite direction.

Gradient Direction

The direction of a gradient is extremely important.

Positive Gradient
→ Increasing the value increases the loss

Negative Gradient
→ Increasing the value decreases the loss

Gradient = 0
→ The loss is not changing at that point

When training a neural network, we normally move in the opposite direction of the gradient because our goal is to reduce the loss.

Simple Mathematical Example

Consider this simple function:

Loss = x²

The derivative is:

d(Loss) / dx = 2x

Suppose:

x = 3

Gradient = 2 × 3

Gradient = 6

The gradient is positive, which tells us that increasing x would increase the loss at this point.

Moving in the Opposite Direction

Remember that our goal during training is to reduce the loss.

If the gradient is positive, we generally need to move the parameter in the opposite direction.

x = 3

Gradient = 6

Gradient direction
→ Increase x

Opposite direction
→ Decrease x

This simple idea is the foundation of gradient descent.

Gradient With a Neural Network

A neural network has weights that control how strongly different inputs affect its prediction.

Input
  ↓
Weight 1 ──┐
           │
Weight 2 ──┼──→ Neural Network
           │
Weight 3 ──┘
                 ↓
             Prediction
                 ↓
                Loss

The gradient tells us how the loss changes with respect to each weight.

Gradient for Weight 1
→ How Weight 1 affects the loss

Gradient for Weight 2
→ How Weight 2 affects the loss

Gradient for Weight 3
→ How Weight 3 affects the loss

Gradients for Multiple Weights

Real neural networks have many weights, so we calculate a gradient for each parameter.

Weights:

w1 = 2
w2 = 5
w3 = 1

Gradients:

∂Loss/∂w1 = 0.5
∂Loss/∂w2 = -1.2
∂Loss/∂w3 = 0.1

These values tell us how each weight affects the loss.

The collection of these gradients is called the gradient.

What Does the Gradient Tell Us?

A gradient provides two important pieces of information: direction and sensitivity.

Gradient = 0.1
→ Small effect


Gradient = 2.0
→ Larger effect


Gradient = -1.5
→ Effect is in the opposite direction

A larger magnitude means that a small change in that parameter can have a larger effect on the loss.

Gradient and Learning

The gradient itself does not directly mean "make the model better." It tells us how the loss changes.

The training algorithm uses this information to decide how the weights should be updated.

Loss
 ↓
Gradient
 ↓
Direction of change
 ↓
Update weights
 ↓
New prediction
 ↓
New loss

Gradient Descent

Gradient descent is the optimization method commonly used to update model parameters in machine learning.

It uses the gradient to move the parameters in a direction that reduces the loss.

Current Weight
      ↓
Calculate Gradient
      ↓
Move Opposite to Gradient
      ↓
New Weight
      ↓
Lower Loss

We will study gradient descent in detail in the next lesson.

Learning Rate

The gradient tells us the direction, but we also need to decide how large a step to take.

This is controlled by the learning rate.

Large Learning Rate
→ Large steps

Small Learning Rate
→ Small steps

If the learning rate is too large, the model may jump past a good solution. If it is too small, training can become very slow.

Simple Weight Update

A simplified gradient descent update looks like this:

new_weight =
old_weight - learning_rate × gradient

For example:

old_weight = 5
learning_rate = 0.1
gradient = 2

new_weight
= 5 - (0.1 × 2)

= 5 - 0.2

= 4.8

The weight moved from 5 to 4.8 because the gradient was positive.

Gradient in Simple Words

Think about driving down a mountain.

Mountain
    ↓
Which direction goes uphill?
    ↓
Gradient tells you

To go downhill:
    ↓
Move opposite to the gradient

In AI, the "mountain" represents the loss landscape and the goal is to move toward a region where the loss is smaller.

Gradient in a Real AI Example

Imagine a model predicting house prices.

Input:
House size = 2000 sq ft

Model prediction:
₹40 lakh

Actual price:
₹50 lakh

Large error
     ↓
Large loss
     ↓
Calculate gradients
     ↓
Adjust weights
     ↓
Make a better prediction

The gradients help identify how the model's parameters contributed to the prediction error.

Gradient and Backpropagation

In a neural network, gradients are calculated using a process called backpropagation.

Prediction
    ↓
Loss
    ↓
Backpropagation
    ↓
Gradients
    ↓
Weight Updates
    ↓
Better Model

Backpropagation calculates how the loss changes with respect to the different weights in the network.

We will study backpropagation in more detail after understanding the remaining mathematics behind neural network training.

Why Gradients Matter in AI

Without gradients, a neural network would have a very difficult time knowing how to adjust millions of parameters efficiently.

Millions of weights
        ↓
Calculate how each affects loss
        ↓
Gradients
        ↓
Update weights
        ↓
Improve predictions

This is why gradients are one of the fundamental mathematical ideas behind modern neural networks.

The Complete Learning Loop

Now we can connect the concepts we have learned so far.

Input Data
    ↓
Neural Network
    ↓
Prediction
    ↓
Loss Function
    ↓
Calculate Loss
    ↓
Gradients
    ↓
Gradient Descent
    ↓
Update Weights
    ↓
Repeat

This loop is repeated many times during training until the model learns useful patterns from the data.