Momentum

What Is Momentum?

Normal Gradient Descent calculates the current gradient and immediately uses it to update the weights.

Momentum adds a small amount of memory. It remembers the direction and size of previous updates and uses that information for the next update.

Gradient Descent

Current Gradient
      ↓
Update Weight


Momentum

Current Gradient
      +
Previous Update
      ↓
Calculate New Update
      ↓
Update Weight

So the simple idea is: Momentum helps the optimizer keep moving in a useful direction instead of reacting completely to every individual gradient.

Why Do We Need Momentum?

SGD and mini-batch Gradient Descent can produce noisy gradients because each update is calculated from only a portion of the training data.

Imagine the gradients keep changing slightly:

Gradient 1 → 0.40
Gradient 2 → 0.35
Gradient 3 → 0.45
Gradient 4 → 0.38
Gradient 5 → 0.42

Without momentum, the optimizer reacts directly to each gradient.

Momentum smooths these updates by carrying some information from previous steps.

Previous Update
       +
Current Gradient
       ↓
New Update

Easy Real-World Example

Imagine pushing a heavy ball down a hill.

        Ball
          ●
         / \
        /   \
       /     \
      /       \
     /_________\

       Downhill
       direction

If you push the ball repeatedly in the same direction, the ball gains momentum and keeps moving.

In optimization:

Ball movement
     ↓
Momentum

Hill
     ↓
Loss landscape

Downhill direction
     ↓
Direction that reduces loss

This is an analogy, not the mathematical definition. The important concept is that previous updates influence future updates.

How Momentum Works

Momentum keeps a value commonly called velocity.

This velocity represents the accumulated direction of previous updates.

Current Gradient
      ↓
Combine with Previous Velocity
      ↓
New Velocity
      ↓
Update Weight

A simplified formula is:

velocity =
    momentum × previous_velocity
    + gradient

Then the weight is updated using the velocity:

weight =
    weight - learning_rate × velocity

The exact implementation can vary between optimizers and libraries, but this captures the basic idea.

What Is the Momentum Value?

The momentum value controls how much of the previous velocity is carried forward.

A common value is something like:

momentum = 0.9

This means a large portion of the previous velocity is retained.

Previous Velocity
       ↓
      × 0.9
       ↓
Keep most of it

The current gradient is then added to that accumulated information.

Simple Numeric Example

Suppose we start with:

weight = 0.50
velocity = 0
learning_rate = 0.10
momentum = 0.90

The first gradient is:

gradient = 0.20

Calculate the new velocity:

velocity =
    (0.90 × 0) + 0.20

velocity =
    0.20

Now update the weight:

weight =
    0.50 - (0.10 × 0.20)

weight =
    0.48

So after the first update:

Weight    = 0.48
Velocity  = 0.20

Second Update

Now suppose the next gradient is:

gradient = 0.10

The previous velocity was:

previous_velocity = 0.20

Calculate the new velocity:

velocity =
    (0.90 × 0.20) + 0.10

velocity =
    0.18 + 0.10

velocity =
    0.28

Notice what happened.

The new velocity is 0.28, not simply 0.10.

Why?

Previous movement
      +
Current gradient
      ↓
New movement

This is the core idea behind Momentum.

Updating the Weight Again

Using the new velocity:

velocity = 0.28
learning_rate = 0.10

Update the weight:

weight =
    0.48 - (0.10 × 0.28)

weight =
    0.48 - 0.028

weight =
    0.452

The weight is now:

0.50
 ↓
0.48
 ↓
0.452

The second update was influenced by the first update.

What If the Gradient Changes Direction?

Suppose the next gradient becomes negative:

gradient = -0.05

The previous velocity is:

previous_velocity = 0.28

Calculate:

velocity =
    (0.90 × 0.28) + (-0.05)

velocity =
    0.252 - 0.05

velocity =
    0.202

Notice that the velocity is still positive.

The previous movement is still influencing the current movement.

Previous direction
        ↓
        ++++++++++
                   \
                    \
Current gradient     ←
        ↓
Combined direction
        ↓
Still moving forward

This is one reason Momentum can help prevent the optimizer from changing direction too aggressively because of small gradient fluctuations.

Momentum Can Reduce Oscillation

Imagine the optimizer is moving through a narrow valley.

        Loss
         ↑
         | \      /
         |  \    /
         |   \  /
         |    \/
         |    /\
         |   /  \
         +----------------→ Weight

Without momentum, updates can bounce from one side to the other.

Left
 ↓
Right
 ↓
Left
 ↓
Right
 ↓
Left

Momentum can smooth this behavior by accumulating movement in the useful direction.

Gradient Descent vs Momentum

Gradient Descent

Current Gradient
      ↓
Update Weight


Momentum

Current Gradient
      +
Previous Velocity
      ↓
New Velocity
      ↓
Update Weight

The key difference is that standard Gradient Descent does not explicitly keep this velocity term, while Momentum does.

Momentum With Python

We can implement a simple version ourselves.

weight = 0.50
velocity = 0.0

learning_rate = 0.10
momentum = 0.90

gradients = [0.20, 0.10, -0.05, 0.08]

for gradient in gradients:

    velocity = (
        momentum * velocity
        + gradient
    )

    weight = (
        weight
        - learning_rate * velocity
    )

    print(weight)

This code demonstrates the main idea:

gradient
    +
previous velocity
    ↓
new velocity
    ↓
weight update

Understand the Python Code

weight = 0.50

This is the initial weight.

velocity = 0.0

We start with no previous movement.

learning_rate = 0.10

This controls the size of the weight update.

momentum = 0.90

This controls how much previous velocity is carried forward.

gradients = [0.20, 0.10, -0.05, 0.08]

These represent gradients from successive training updates.

velocity = (
    momentum * velocity
    + gradient
)

This is the important Momentum calculation.

It combines the previous velocity with the current gradient.

weight = (
    weight
    - learning_rate * velocity
)

The weight is then updated using the new velocity.

Momentum During Neural Network Training

Training Data
      ↓
Forward Propagation
      ↓
Prediction
      ↓
Loss
      ↓
Backpropagation
      ↓
Gradient
      ↓
Momentum
      ↓
Velocity
      ↓
Update Weights
      ↓
Next Batch
      ↓
Repeat

Momentum is therefore part of the optimizer step. It does not replace forward propagation, the loss function, or backpropagation.

Why Use Momentum?

Momentum can provide several useful effects during training.

1. Smoother Updates

Previous movement
       +
Current gradient
       ↓
Less sudden movement


2. Faster Movement in Consistent Directions

Same direction
      ↓
Velocity builds
      ↓
Larger effective movement

In practice, Momentum is useful when the gradient direction contains noise or when optimization has a tendency to oscillate.

Momentum Does Not Guarantee Better Training

Momentum is useful, but it is not magic.

If the learning rate or momentum value is poorly chosen, training can still become unstable or converge poorly.

Bad Learning Rate
       +
Bad Momentum Setting
       ↓
Poor Training

The optimizer is only one part of the training system.

Understanding the Momentum Value

Consider:

momentum = 0.0

There is no contribution from previous velocity in the simplified formula.

velocity =
    (0.0 × previous_velocity)
    + gradient

= gradient

Now consider:

momentum = 0.9

Most of the previous velocity is retained:

velocity =
    (0.9 × previous_velocity)
    + gradient

So increasing momentum generally gives the previous direction more influence.

Complete Example

Start with:

weight = 0.80
velocity = 0

learning_rate = 0.10
momentum = 0.90

First gradient:

gradient = 0.30

velocity =
    (0.90 × 0) + 0.30

= 0.30

weight =
    0.80 - (0.10 × 0.30)

= 0.77

Second gradient:

gradient = 0.20

velocity =
    (0.90 × 0.30) + 0.20

= 0.27 + 0.20

= 0.47

weight =
    0.77 - (0.10 × 0.47)

= 0.723

Notice that the second update uses information from the first update.

First velocity  = 0.30
Second velocity = 0.47

That accumulated movement is what gives Momentum its name.

Remember This

Momentum

1. Calculate the current gradient
2. Remember previous velocity
3. Combine previous velocity
   with current gradient
4. Calculate new velocity
5. Update the weight
6. Repeat

The simplified formulas are:

velocity =
    momentum × previous_velocity
    + gradient


weight =
    weight
    - learning_rate × velocity

The most important sentence is: Momentum gives Gradient Descent memory by allowing previous updates to influence the current update.