Learning Rate

What Is Learning Rate?

During training, a neural network makes predictions, calculates the loss, and uses backpropagation to determine how its weights should change.

But the model needs to know: How much should I change the weights?

The learning rate controls the size of that change.

Calculate Loss
      ↓
Calculate Gradients
      ↓
Learning Rate
      ↓
Update Weights

So, learning rate does not decide which direction the weights should move. The gradient provides the direction. The learning rate controls how big the step is.

A Simple Example

Imagine you are trying to walk toward a target.

Current Position
       ●

       ↓

     Target
       ★

You need to move toward the target. But how far should you move each time?

That is similar to the learning rate.

Small Learning Rate
→ Small steps


Large Learning Rate
→ Large steps

A very small learning rate can make training slow. A very large learning rate can make the model jump around and fail to find a good solution.

Learning Rate and Weight Updates

A simplified weight-update formula is:

new weight
=
old weight
-
learning rate × gradient

For example, suppose:

Old Weight = 0.80
Gradient = 0.40
Learning Rate = 0.10

The update is:

New Weight
=
0.80 - (0.10 × 0.40)

=
0.80 - 0.04

=
0.76

The learning rate determines how much of the gradient is used to change the weight.

Small Learning Rate

Suppose the learning rate is:

Learning Rate = 0.01

Using the same weight and gradient:

Old Weight = 0.80
Gradient = 0.40

New Weight
=
0.80 - (0.01 × 0.40)

=
0.80 - 0.004

=
0.796

The weight changed only a little.

0.800
  ↓
0.796

Small change

Small learning rates can make training more stable, but training may take many iterations to reach a good solution.

Large Learning Rate

Now suppose:

Learning Rate = 0.50

Using the same values:

Old Weight = 0.80
Gradient = 0.40

New Weight
=
0.80 - (0.50 × 0.40)

=
0.80 - 0.20

=
0.60

The weight changes much more:

0.800
  ↓
0.600

Large change

If the learning rate is too large, the model can make updates that are too aggressive.

What Happens If the Learning Rate Is Too Small?

A very small learning rate means the model takes tiny steps during training.

Target
  ★

        ●
        ↑
     tiny step

        ●
        ↑
     tiny step

        ●
        ↑
     tiny step

The model may eventually reach a good solution, but it can require a very large number of iterations.

So the main problem is: training can become unnecessarily slow.

What Happens If the Learning Rate Is Too Large?

A very large learning rate makes the model take huge steps.

Target
  ★

● ─────────────→

                  ●
                  │
                  ↓

        ● ←────────────

                  ↓

              Target

Instead of gradually moving toward a good solution, the model can jump over it repeatedly.

This can make the loss fluctuate or even become unstable.

What Is a Good Learning Rate?

A good learning rate allows the model to make meaningful progress without making excessively large jumps.

Start
  ●
   \
    ●
     \
      ●
       \
        ●
         ★
       Target

During successful training, the model should generally move toward a region where the loss becomes smaller.

There is no single learning rate that works perfectly for every neural network. It depends on the model, dataset, optimizer, and other training settings.

Common Learning Rate Values

Learning rates are often small decimal values.

0.1
0.01
0.001
0.0001

For example, if we use:

learning_rate = 0.001

the optimizer uses that value when calculating weight updates.

Do not assume that 0.001 is always the best value. It is simply a commonly encountered starting point in many neural-network examples.

Learning Rate During Training

Suppose the model starts with:

Learning Rate = 0.01

During every weight update, the learning rate controls the size of the update.

Iteration 1
    ↓
Calculate Gradient
    ↓
Update Weight

Iteration 2
    ↓
Calculate Gradient
    ↓
Update Weight

Iteration 3
    ↓
Calculate Gradient
    ↓
Update Weight

...

Iteration N
    ↓
Calculate Gradient
    ↓
Update Weight

The gradient can change from one iteration to another, while the learning rate determines how strongly that gradient affects the weight update.

Learning Rate vs Gradient

These two concepts are easy to confuse.

Gradient
→ Tells the model which direction the weight should move.


Learning Rate
→ Tells the model how large the movement should be.

Think of it like driving:

Gradient
→ Direction

Learning Rate
→ How far you move in that direction

Both are required to update the weights effectively.

Learning Rate and Loss

The learning rate can strongly affect how the loss changes during training.

With a suitable learning rate, the loss may gradually decrease:

Loss

1.00
 ↓
0.70
 ↓
0.50
 ↓
0.32
 ↓
0.20

With a learning rate that is too large, the loss may jump around instead:

Loss

0.50
 ↓
1.20
 ↓
0.40
 ↓
1.00
 ↓
0.90

This does not automatically prove that the learning rate is the problem, but an inappropriate learning rate is one possible cause of unstable training.

Learning Rate in Python

A simple weight update can be written in Python like this:

weight = 0.80
gradient = 0.40
learning_rate = 0.10

weight = weight - learning_rate * gradient

print(weight)

Output:

0.76

The important line is:

weight = weight - learning_rate * gradient

Here, the learning rate controls the size of the weight adjustment.

Learning Rate in a Training Loop

learning_rate = 0.01

for iteration in range(5):

    prediction = model(input_data)

    loss = calculate_loss(
        prediction,
        target
    )

    gradient = calculate_gradient(loss)

    weight = weight - learning_rate * gradient

    print("Iteration:", iteration + 1)
    print("Weight:", weight)

In this example, every iteration calculates a gradient and uses the learning rate to control the weight update.

Iteration
    ↓
Gradient
    ↓
Learning Rate
    ↓
Weight Update
    ↓
Next Iteration

Compare Learning Rates

Suppose the old weight is 0.80 and the gradient is 0.40.

Learning Rate = 0.001

New Weight
= 0.80 - (0.001 × 0.40)
= 0.7996


Learning Rate = 0.01

New Weight
= 0.80 - (0.01 × 0.40)
= 0.796


Learning Rate = 0.1

New Weight
= 0.80 - (0.1 × 0.40)
= 0.76

Notice what changed: the gradient stayed the same, but the size of the weight update became larger as the learning rate increased.

Easy Way to Remember

Imagine you are walking down a mountain and trying to reach the lowest point.

Gradient
→ Tells you which direction goes downhill.


Learning Rate
→ Controls how large your next step is.

Very Small Step
→ Slow progress


Good Step
→ Controlled progress


Very Large Step
→ You may jump past the lowest point

This is the basic idea behind learning rate in neural network training.

Remember This

Learning Rate
=
Controls the size of weight updates.

Small Learning Rate
→ Smaller updates
→ Usually slower learning

Large Learning Rate
→ Larger updates
→ Can be faster
→ Can also become unstable

Weight Update:

new weight
=
old weight
-
learning rate × gradient

The simplest definition to remember is: learning rate controls how big a step the model takes when updating its weights.