Adam Optimizer

What Is Adam Optimizer?

Adam stands for Adaptive Moment Estimation.

Adam is an optimizer that uses the gradients calculated during backpropagation to update the neural network's weights.

Forward Propagation
        ↓
Calculate Loss
        ↓
Backpropagation
        ↓
Calculate Gradients
        ↓
Adam Optimizer
        ↓
Update Weights

Unlike basic Gradient Descent, Adam does not simply use the same learning rate in the same way for every parameter. It keeps information about previous gradients and uses the size of recent gradients to adjust the updates.

Simple Idea Behind Adam

Adam combines two important ideas.

Adam
 │
 ├── First Moment
 │      ↓
 │   Tracks average gradient
 │   Similar to Momentum
 │
 └── Second Moment
        ↓
     Tracks average squared gradient
     Helps adjust the step size

So Adam remembers both:

1. Which direction have gradients
   generally been moving?

2. How large have the gradients
   generally been?

It then uses this information to decide how much each weight should change.

Why Do We Need Adam?

Basic Gradient Descent uses a learning rate to control the size of the update.

weight =
    weight - learning_rate × gradient

The problem is that neural networks can have thousands, millions, or even billions of parameters. Different parameters can have very different gradient behavior.

Adam tries to make the updates more adaptive.

Parameter 1
→ One update size

Parameter 2
→ Different effective update size

Parameter 3
→ Different effective update size

This is one reason Adam is widely used for training neural networks.

Adam and Momentum

You already learned that Momentum remembers previous gradients.

Momentum

Current Gradient
       +
Previous Gradient Information
       ↓
Update Direction

Adam also maintains a moving average of gradients.

Current Gradient
       ↓
Moving Average
       ↓
First Moment
       ↓
Update Direction

This first moment is commonly represented by:

m

You can think of m as Adam's memory of the recent gradient direction.

What Is the Second Moment?

Adam also keeps track of the squared gradients.

Gradient
    ↓
Square the Gradient
    ↓
Moving Average
    ↓
Second Moment

This value is commonly represented by:

v

The second moment gives Adam information about the typical magnitude of recent gradients.

m → direction information

v → gradient magnitude information

This is the key difference between basic Momentum and Adam.

Easy Real-World Example

Imagine you are walking down a mountain and trying to reach the lowest point.

You need to know two things:

1. Which direction should I walk?

2. How large should my next step be?

Adam tries to answer both questions.

Gradient history
      ↓
Which direction?
      ↓
First Moment


Gradient size history
      ↓
How large should the step be?
      ↓
Second Moment

So a simple way to remember Adam is: direction + adaptive step size.

Adam Formulas

Adam uses two moving averages.

First moment:

m = β₁ × m + (1 - β₁) × gradient

Second moment:

v = β₂ × v + (1 - β₂) × gradient²

Then Adam uses these values to calculate the weight update.

A simplified version looks like:

weight =
    weight
    - learning_rate × m / (√v + ε)

The actual Adam algorithm also uses bias correction, which we will include in the complete example below.

Common Adam Parameters

learning_rate = 0.001
β₁ = 0.9
β₂ = 0.999
ε = 1e-8

These are commonly used default values.

They are not universal laws. Different problems can require different settings.

β₁
→ Controls the first moment


β₂
→ Controls the second moment


ε
→ Prevents division by zero

Simple Numeric Example

Let's make the calculation easier by using:

weight = 0.50
gradient = 0.20

learning_rate = 0.10
β₁ = 0.9
β₂ = 0.999

m = 0
v = 0

First calculate the first moment:

m =
    0.9 × 0 + (1 - 0.9) × 0.20

m =
    0 + 0.02

m =
    0.02

Now calculate the second moment:

v =
    0.999 × 0
    + (1 - 0.999) × (0.20²)

v =
    0.001 × 0.04

v =
    0.00004

Adam now has:

m = 0.02
v = 0.00004

Why Does Adam Need Bias Correction?

At the beginning of training:

m = 0
v = 0

Because both moving averages start at zero, they can be biased toward zero during the first few updates.

Adam corrects this using:

m_hat =
    m / (1 - β₁ᵗ)

v_hat =
    v / (1 - β₂ᵗ)

Here t represents the current optimization step.

For the first step:

t = 1

Therefore:

m_hat =
    0.02 / (1 - 0.9¹)

= 0.02 / 0.1

= 0.20

And:

v_hat =
    0.00004 / (1 - 0.999¹)

= 0.00004 / 0.001

= 0.04

Now Adam has corrected values:

m_hat = 0.20
v_hat = 0.04

Updating the Weight

Adam uses:

new_weight =
    weight
    - learning_rate ×
      m_hat / (√v_hat + ε)

Using our values:

weight = 0.50
learning_rate = 0.10
m_hat = 0.20
v_hat = 0.04
ε = 0.00000001

First:

√v_hat = √0.04

√v_hat = 0.20

Then:

update =
    0.10 × 0.20 / (0.20 + ε)

≈ 0.10

Therefore:

new_weight =
    0.50 - 0.10

= 0.40

This is a simplified numerical demonstration. Real neural networks calculate these values independently for many parameters.

Another Example: Two Parameters

Imagine a neural network has two weights:

Weight 1 = 0.50
Weight 2 = 0.50

Suppose their gradients are very different:

Gradient 1 = 0.10
Gradient 2 = 2.00

A basic optimizer may apply the same global learning-rate setting to both.

Adam keeps separate first- and second-moment information for each parameter.

Weight 1
    ↓
m₁, v₁
    ↓
Its own adaptive update


Weight 2
    ↓
m₂, v₂
    ↓
Its own adaptive update

This is what people mean when they say Adam uses an adaptive learning rate.

Adam With Python

Here is a simplified implementation of Adam for a single weight.

import math

weight = 0.50

learning_rate = 0.10
beta1 = 0.9
beta2 = 0.999
epsilon = 1e-8

m = 0.0
v = 0.0

gradients = [0.20, 0.10, -0.05, 0.08]

for t, gradient in enumerate(gradients, start=1):

    # First moment
    m = beta1 * m + (1 - beta1) * gradient

    # Second moment
    v = beta2 * v + (1 - beta2) * (gradient ** 2)

    # Bias correction
    m_hat = m / (1 - beta1 ** t)
    v_hat = v / (1 - beta2 ** t)

    # Weight update
    weight -= (
        learning_rate
        * m_hat
        / (math.sqrt(v_hat) + epsilon)
    )

    print(
        f"Step {t}: weight = {weight:.4f}"
    )

Understand the Python Code

weight = 0.50

This is the initial model parameter.

learning_rate = 0.10

This controls the overall size of the Adam update.

beta1 = 0.9

This controls the first moment, which tracks recent gradient direction.

beta2 = 0.999

This controls the second moment, which tracks recent squared-gradient magnitude.

m = 0.0
v = 0.0

These store Adam's memory.

for t, gradient in enumerate(gradients, start=1):

We process one gradient at a time. The variable t represents the current optimization step.

m = beta1 * m + (1 - beta1) * gradient

This updates the first moment.

v = beta2 * v + (1 - beta2) * (gradient ** 2)

This updates the second moment using the squared gradient.

m_hat = m / (1 - beta1 ** t)
v_hat = v / (1 - beta2 ** t)

These are the bias-corrected first and second moments.

weight -= (
    learning_rate
    * m_hat
    / (math.sqrt(v_hat) + epsilon)
)

Finally, Adam uses the corrected moments to update the weight.

Adam During Neural Network Training

Training Data
      ↓
Forward Propagation
      ↓
Prediction
      ↓
Loss
      ↓
Backpropagation
      ↓
Gradients
      ↓
Adam
 ┌────┴─────┐
 ↓          ↓
First      Second
Moment     Moment
 ↓          ↓
 └────┬─────┘
      ↓
Bias Correction
      ↓
Calculate Update
      ↓
Update Weights
      ↓
Next Batch
      ↓
Repeat

Notice that Adam comes after backpropagation. Adam does not calculate the gradients. Backpropagation calculates the gradients, and Adam decides how to use them to update the parameters.

Momentum vs Adam

Momentum

Gradient
   ↓
Previous Direction
   ↓
Velocity
   ↓
Update Weight


Adam

Gradient
   ↓
 ┌───────────────┐
 ↓               ↓
First Moment     Second Moment
 ↓               ↓
Direction        Magnitude
 └───────┬───────┘
         ↓
Adaptive Update
         ↓
Update Weight

The important distinction is that Momentum mainly adds accumulated direction information, while Adam also uses squared-gradient information to adapt the update size.

SGD vs Adam

SGD

gradient
   ↓
learning rate
   ↓
update


Adam

gradient
   ↓
first moment
   ↓
second moment
   ↓
bias correction
   ↓
adaptive update

Adam has more internal state than basic SGD, but that extra information can make optimization easier for many neural network problems.

When Is Adam Useful?

Adam is often a strong default choice when you are starting to train a neural network and do not yet have a reason to choose another optimizer.

It is particularly convenient because it automatically adapts updates based on gradient history.

Starting a new neural network
          ↓
Try Adam
          ↓
Monitor training
          ↓
Tune learning rate
          ↓
Compare with other optimizers
if necessary

But do not make the mistake of thinking "Adam is always the best optimizer." Optimizer performance depends on the model, dataset, learning rate, regularization, and training objective.

Important Adam Parameters

learning_rate = 0.001
beta1 = 0.9
beta2 = 0.999
epsilon = 1e-8

The parameter you will most commonly tune is the learning rate.

For example:

learning_rate = 0.001

# Smaller
learning_rate = 0.0001

# Larger
learning_rate = 0.01

A learning rate that is too large can make training unstable. A learning rate that is too small can make training unnecessarily slow.

One Important Thing to Understand

Adam does not magically make a bad neural network good.

Bad Data
   +
Bad Architecture
   +
Bad Learning Rate
   +
Adam
   ↓
Still can produce bad results

The optimizer controls how parameters are updated. It does not replace good data, architecture, preprocessing, loss functions, or proper evaluation.

Easy Way to Remember Adam

Adam remembers:

"Which direction have I been moving?"
             ↓
         First Moment

"How large have my gradients been?"
             ↓
         Second Moment

"How should I adjust this weight?"
             ↓
        Adaptive Update

If you understand those three questions, you understand the core idea of Adam.

Remember This

Adam

1. Get gradient from backpropagation
2. Update first moment
3. Update second moment
4. Correct the bias
5. Calculate adaptive update
6. Update weights
7. Repeat

The two most important values are:

m
→ Average gradient
→ Direction information


v
→ Average squared gradient
→ Magnitude information

The most important sentence is: Adam combines Momentum-like gradient memory with adaptive step sizes based on the magnitude of recent gradients.