Gradient Descent

What Is Gradient Descent?

A neural network makes predictions using its weights. If the prediction is wrong, we need to change those weights.

Gradient Descent provides a systematic way to make those changes.

Prediction
     ↓
Calculate Loss
     ↓
Calculate Gradient
     ↓
Gradient Descent
     ↓
Update Weight
     ↓
Lower Loss
     ↓
Repeat

The goal is simple: find weight values that produce a smaller loss.

Why Do We Need Gradient Descent?

Suppose a neural network starts with a random weight:

weight = 0.50

The model makes a prediction and calculates a loss. Backpropagation tells us the gradient.

gradient = 0.20

We now need to use that gradient to change the weight. Gradient Descent gives us the rule for doing that.

The Gradient Descent Formula

The basic update formula is:

new_weight =
    old_weight - learning_rate * gradient

There are three important parts:

old_weight
→ The current weight


learning_rate
→ Controls how large the step is


gradient
→ Tells us how the loss changes
  when the weight changes

The minus sign is important because we want to move in the direction that decreases the loss.

Simple Example

Suppose we have:

Old Weight    = 0.50
Gradient      = 0.20
Learning Rate = 0.10

Apply the formula:

new_weight =
    old_weight - learning_rate * gradient


new_weight =
    0.50 - (0.10 × 0.20)


new_weight =
    0.50 - 0.02


new_weight =
    0.48

So the weight changes:

0.50
 ↓
0.48

That is one Gradient Descent update.

What Does the Gradient Tell Us?

The gradient tells us how the loss changes when a parameter changes.

For a simple one-dimensional example, imagine the loss behaves like this:

Loss
 ↑
 |       ●
 |      / \
 |     /   \
 |    /     \
 |   ●       ●
 |  /         \
 | ●           ●
 +----------------→ Weight

The lowest point represents a smaller loss.

Gradient Descent tries to move toward that lower-loss region.

Easy Mountain Analogy

Imagine standing somewhere on a mountain. Your goal is to reach the lowest point.

                 ●
                You
               /  \
              /    \
             /      \
            /        \
           /          \
          /            \
         /______________\
              Lowest
               Point

You look at the slope and take a step downhill.

Then you check the slope again and take another step.

Step 1
  ↓
Check slope
  ↓
Step 2
  ↓
Check slope
  ↓
Step 3
  ↓
...
  ↓
Lowest region

In machine learning:

Mountain height
     ↓
Loss

Slope
     ↓
Gradient

Step size
     ↓
Learning Rate

Moving downhill
     ↓
Gradient Descent

This is only an analogy, but it captures the basic idea very well.

What Happens With a Positive Gradient?

Suppose:

gradient = +0.20

The update becomes:

new_weight =
    old_weight - learning_rate * gradient

Using:

old_weight = 0.50
learning_rate = 0.10
gradient = +0.20

new_weight =
0.50 - (0.10 × 0.20)

= 0.48

The weight decreases.

What Happens With a Negative Gradient?

Now suppose:

gradient = -0.20

The update becomes:

new_weight =
0.50 - (0.10 × -0.20)

= 0.50 + 0.02

= 0.52

The weight increases.

This is why you should not memorize that Gradient Descent always makes weights smaller. That is wrong.

Positive Gradient
→ Weight decreases


Negative Gradient
→ Weight increases

The optimizer is following the gradient information to move toward lower loss.

Learning Rate

The learning rate controls the size of each Gradient Descent step.

new_weight =
    old_weight - learning_rate * gradient

Consider:

old_weight = 0.50
gradient = 0.20

With a small learning rate:

learning_rate = 0.01

new_weight =
0.50 - (0.01 × 0.20)

= 0.498

With a larger learning rate:

learning_rate = 0.50

new_weight =
0.50 - (0.50 × 0.20)

= 0.40

Same gradient, but a very different update size.

Learning Rate Too Small

If the learning rate is extremely small, the model may make very tiny updates.

Loss
 ↓
Tiny Update
 ↓
Tiny Update
 ↓
Tiny Update
 ↓
Very Slow Training

Example:

learning_rate = 0.000001

The model may require a huge number of updates to make meaningful progress.

Learning Rate Too Large

If the learning rate is too large, the model can jump past a good solution.

Too Large Step
      ↓
Overshoot
      ↓
Another Large Step
      ↓
Overshoot Again
      ↓
Unstable Training

So choosing a learning rate is important.

Gradient Descent Happens Repeatedly

One update is not enough for a real neural network. Gradient Descent repeatedly updates the parameters.

Initial Weight
     ↓
0.80

Gradient
     ↓
0.30

Update
     ↓
0.77

Gradient
     ↓
0.20

Update
     ↓
0.75

Gradient
     ↓
0.10

Update
     ↓
0.74

The actual values in a real neural network depend on the model, data, loss function, and optimizer settings. This example is only showing the mechanism.

Gradient Descent With Python

We can implement the basic update rule directly in Python.

weight = 0.50
gradient = 0.20
learning_rate = 0.10

weight = weight - learning_rate * gradient

print(weight)

Output:

0.48

The important line is:

weight = weight - learning_rate * gradient

This is the basic Gradient Descent update.

Gradient Descent as a Function

We can make the update reusable:

def gradient_descent(
    weight,
    gradient,
    learning_rate
):
    return weight - learning_rate * gradient


weight = 0.50
gradient = 0.20
learning_rate = 0.10

weight = gradient_descent(
    weight,
    gradient,
    learning_rate
)

print(weight)

Output:

0.48

Gradient Descent in a Training Loop

In neural network training, the process is repeated.

weight = 0.50
learning_rate = 0.10

gradients = [0.20, 0.10, 0.05, 0.02]

for gradient in gradients:

    weight = (
        weight
        - learning_rate * gradient
    )

    print(weight)

The idea is:

Gradient 1
    ↓
Update Weight
    ↓
Gradient 2
    ↓
Update Weight
    ↓
Gradient 3
    ↓
Update Weight
    ↓
...

In a real neural network, these gradients are calculated from the current loss through backpropagation.

Gradient Descent in Neural Network Training

Training Data
      ↓
Forward Propagation
      ↓
Prediction
      ↓
Loss Function
      ↓
Loss
      ↓
Backpropagation
      ↓
Gradient
      ↓
Gradient Descent
      ↓
Updated Weights
      ↓
Forward Propagation Again
      ↓
New Prediction
      ↓
New Loss
      ↓
Repeat

Notice the important distinction: backpropagation calculates the gradient, while Gradient Descent uses that gradient to update the weights.

One Complete Example

Let's put everything together with one simple weight.

weight = 0.80
learning_rate = 0.10

Suppose backpropagation gives us:

gradient = 0.30

Gradient Descent calculates:

new_weight =
0.80 - (0.10 × 0.30)

= 0.80 - 0.03

= 0.77

The weight is now:

0.77

The model performs another forward pass. Suppose the new gradient is:

gradient = 0.15

Update again:

new_weight =
0.77 - (0.10 × 0.15)

= 0.755

The process continues.

0.80
 ↓
0.77
 ↓
0.755
 ↓
...
 ↓
Better parameter values

Do Not Confuse These Concepts

Loss Function
→ Measures prediction error.


Backpropagation
→ Calculates gradients.


Gradient
→ Describes how loss changes
  with respect to a parameter.


Gradient Descent
→ Uses gradients to update parameters.


Learning Rate
→ Controls the size of the update.

These are different pieces of the same training process.

Remember This

Gradient Descent

1. Start with weights
2. Make a prediction
3. Calculate loss
4. Calculate gradients
5. Update weights
6. Repeat

Basic formula:

new_weight =
    old_weight
    - learning_rate × gradient

The most important idea is: Gradient Descent repeatedly moves the model's parameters in a direction intended to reduce the loss.