Gradient Descent

What Is Gradient Descent?

Gradient Descent is a mathematical method for finding a better solution to a problem.

In machine learning, the problem is usually:

How can we make the model's predictions better?

        ↓

Reduce the model's error

The model starts with some parameter values and repeatedly changes them to reduce the error.

Start
  ↓
Calculate Error
  ↓
Calculate Gradient
  ↓
Update Parameters
  ↓
Calculate Error Again
  ↓
Repeat

Why Do We Need Gradient Descent?

A machine learning model has parameters such as weights and biases. We need to find values for these parameters that make the model's predictions accurate.

The problem is that there can be thousands, millions, or even billions of parameters.

Weight 1 = ?
Weight 2 = ?
Weight 3 = ?
Weight 4 = ?
...
Weight 1,000,000 = ?

Trying every possible value would be extremely inefficient.

Gradient Descent gives us a systematic way to improve these values instead of randomly guessing.

Think About a Mountain

Imagine that you are standing on a mountain and want to reach the lowest point in the valley.

        Mountain
           /\
          /  \
         /    \
        /      \
       /        \
      /          \
     /            \
    ↓
  Lowest Point

You do not need to know the entire mountain. You only need to know which direction goes downhill.

You take a small step downhill, check your position again, and continue.

Current Position
       ↓
Find Downhill Direction
       ↓
Take Small Step
       ↓
Check Position
       ↓
Repeat

This is the basic idea behind Gradient Descent.

What Is the "Gradient"?

The gradient tells us how the error changes when we change a parameter.

For one parameter, we can think about it as a derivative.

Gradient = Direction of increasing error

If the gradient is positive, increasing the parameter makes the error increase.

If the gradient is negative, increasing the parameter makes the error decrease.

Positive Gradient
      ↓
Error increases in this direction
      ↓
Move in the opposite direction

That is why Gradient Descent moves in the opposite direction of the gradient.

The Gradient Descent Formula

The basic update formula is:

new_parameter =
    old_parameter - learning_rate × gradient

In mathematical notation:

θnew = θold - α × ∇J(θ)

You do not need to be scared by the notation. Each part has a simple meaning.

θ       = parameter

α       = learning rate

∇J(θ)   = gradient

-       = move opposite to the gradient

What Is the Learning Rate?

The learning rate controls how large each update should be.

Think about walking down a mountain.

Small learning rate
        ↓
Small steps
        ↓
Slow but controlled

Large learning rate
        ↓
Large steps
        ↓
Can move too far

For example:

Learning Rate = 0.01

Small update

Learning Rate = 1.0

Much larger update

Choosing the learning rate is important because a value that is too large can cause the model to jump around instead of reaching a good solution.

Simple Numerical Example

Let's say our model currently has:

Parameter = 10

Gradient = 4

Learning Rate = 0.1

Apply the Gradient Descent formula:

new_parameter =
    old_parameter - learning_rate × gradient

new_parameter =
    10 - (0.1 × 4)

new_parameter =
    10 - 0.4

new_parameter =
    9.6

The parameter changed from:

10 → 9.6

The model moved in the direction indicated by the gradient.

A Simple Python Example

We can implement the basic update using Python.

parameter = 10
gradient = 4
learning_rate = 0.1

parameter = parameter - learning_rate * gradient

print(parameter)

Output:

9.6

This is the core operation that happens repeatedly during model training, although real machine learning algorithms calculate gradients from data and loss functions.

Repeating the Process

Gradient Descent normally does not update a parameter only once. It repeats the process many times.

Step 1
Parameter = 10

        ↓

Step 2
Parameter = 9.6

        ↓

Step 3
Parameter = 9.28

        ↓

Step 4
Parameter = 9.02

        ↓

...

Better Parameter

The exact values depend on the loss function, gradient, learning rate, and training data.

The important idea is that the model gradually moves toward parameter values that produce lower error.

Loss Function and Gradient Descent

The model needs a way to measure how wrong its prediction is. This is the job of the loss function.

Prediction
    ↓
Compare with Actual Value
    ↓
Loss
    ↓
Calculate Gradient
    ↓
Update Parameters

A high loss means the model is performing poorly. Gradient Descent tries to move the parameters toward a region where the loss is smaller.

Gradient Descent in Machine Learning

Imagine a model predicting house prices.

Input:
House Size = 2000 sq ft

        ↓

Model

        ↓

Predicted Price = ₹80 lakh

        ↓

Actual Price = ₹90 lakh

        ↓

Calculate Loss

        ↓

Calculate Gradient

        ↓

Update Weights

        ↓

Make another prediction

After many updates, the model can learn weights that produce better predictions.

Gradient Descent in Neural Networks

Neural networks contain many weights and biases. During training, these parameters are updated repeatedly.

Input
  ↓
Neural Network
  ↓
Prediction
  ↓
Loss
  ↓
Backpropagation
  ↓
Gradients
  ↓
Gradient Descent
  ↓
Update Weights
  ↓
Train Again

Backpropagation calculates the gradients, while Gradient Descent uses those gradients to update the parameters.

These two concepts are related, but they are not the same thing.

Gradient Descent and the Chain Rule

In the previous lesson, we learned that the Chain Rule helps calculate derivatives through connected functions.

Chain Rule
     ↓
Calculate Gradients
     ↓
Gradient Descent
     ↓
Update Parameters
     ↓
Reduce Loss

So the concepts we have learned are connected.

The Chain Rule helps us find the gradient, and Gradient Descent uses that gradient to improve the model.

What Happens If the Learning Rate Is Too Large?

A very large learning rate can cause the model to take huge steps.

Good Learning Rate

Loss
 ↓
100
 ↓
70
 ↓
45
 ↓
25
 ↓
10
 ↓
5

The loss gradually decreases.

But with a learning rate that is too large:

Loss
 ↓
100
 ↓
20
 ↓
80
 ↓
15
 ↓
100
 ↓
30

The model may jump past good solutions and fail to settle down.

What Happens If the Learning Rate Is Too Small?

A very small learning rate can make training extremely slow.

Large Steps

10 → 8 → 6 → 4 → 2


Small Steps

10 → 9.9 → 9.8 → 9.7 → 9.6 → ...

A smaller learning rate can provide more controlled updates, but training may require many more iterations.

Therefore, the learning rate is an important hyperparameter in machine learning.

The Complete AI Learning Process

Now connect everything we have learned in Calculus.

Input Data
    ↓
Neural Network
    ↓
Prediction
    ↓
Loss Function
    ↓
Loss
    ↓
Backpropagation
    ↓
Chain Rule
    ↓
Gradients
    ↓
Gradient Descent
    ↓
Update Weights
    ↓
Better Prediction
    ↓
Repeat

This process happens repeatedly during training until the model learns useful parameter values.