Updating Weights

Why Do We Update Weights?

A neural network starts with weights that usually do not produce good predictions.

Input
  ↓
Weights
  ↓
Prediction
  ↓
Loss

High Loss
  ↓
Need to improve the weights

During training, the network changes its weights so that future predictions can produce a smaller loss.

Old Weights
     ↓
Make Prediction
     ↓
Calculate Loss
     ↓
Calculate Gradients
     ↓
Update Weights
     ↓
New Weights
     ↓
Better Prediction

The Weight Update Formula

A basic gradient descent update uses this formula:

New Weight
=
Old Weight
-
Learning Rate × Gradient

There are three important parts:

Old Weight
    ↓
The current value of the weight


Gradient
    ↓
Direction and strength of the loss change


Learning Rate
    ↓
How large the update should be

A Simple Example

Suppose the neural network currently has:

Weight = 5
Gradient = 2
Learning Rate = 0.1

Apply the formula:

New Weight
= 5 - (0.1 × 2)

= 5 - 0.2

= 4.8

The old weight was 5. After the update, it becomes 4.8.

Before:
Weight = 5


After:
Weight = 4.8

What If the Gradient Is Negative?

A gradient can also be negative.

Suppose:

Weight = 5
Gradient = -2
Learning Rate = 0.1

Apply the same formula:

New Weight
= 5 - (0.1 × -2)

= 5 - (-0.2)

= 5 + 0.2

= 5.2

Notice that the weight increased.

Positive Gradient
→ Weight moves downward


Negative Gradient
→ Weight moves upward

This is why the sign of the gradient matters.

What Is the Learning Rate?

The learning rate controls how much the weights change during each update.

Learning Rate
      ↓
Controls the size
of each weight update

For example, suppose:

Weight = 5
Gradient = 2

With a learning rate of 0.1:

Update = 0.1 × 2
       = 0.2

With a learning rate of 0.01:

Update = 0.01 × 2
       = 0.02

So the second update is much smaller.

What If the Learning Rate Is Too Large?

A learning rate that is too large can cause the model to make huge changes to its weights.

Loss
 ↑
 |       ●
 |     ↗   ↘
 |   ●       ●
 | ↗
 |●
 +----------------→ Weight

Instead of gradually moving toward a lower-loss region, the updates can jump over it.

Too Large Learning Rate
          ↓
Large Updates
          ↓
Jump Around
          ↓
Training Can Become Unstable

What If the Learning Rate Is Too Small?

A very small learning rate makes the updates tiny.

Too Small Learning Rate
          ↓
Very Small Updates
          ↓
Very Slow Learning
          ↓
May Take Many Iterations

So the learning rate needs to be chosen carefully.

The Weight Update Loop

Updating weights is not something the model does only once. It happens repeatedly during training.

1. Make Prediction
        ↓
2. Calculate Loss
        ↓
3. Calculate Gradients
        ↓
4. Update Weights
        ↓
5. Make Prediction Again
        ↓
6. Calculate Loss Again
        ↓
7. Repeat

Over many iterations, the model attempts to find weights that produce a lower loss.

Complete Example

Let's connect the previous lessons together using one simple model.

Input = 2
Weight = 3
Target = 10

First, the model makes a prediction:

Prediction
= Input × Weight

= 2 × 3

= 6

The target is 10, so the prediction is too low.

Target     = 10
Prediction = 6

Using squared error:

Loss
= (10 - 6)²

= 16

From backpropagation, suppose the gradient for the weight is:

Gradient = -16

Use a learning rate of 0.1:

Learning Rate = 0.1

Now update the weight:

New Weight
= Old Weight - Learning Rate × Gradient

= 3 - (0.1 × -16)

= 3 + 1.6

= 4.6

The weight changed from 3 to 4.6.

Old Weight = 3
New Weight = 4.6

Make the Prediction Again

Now use the updated weight.

Input = 2
New Weight = 4.6

Calculate the new prediction:

Prediction
= 2 × 4.6

= 9.2

The new prediction is much closer to the target of 10.

Before Update

Prediction = 6
Target     = 10


After Update

Prediction = 9.2
Target     = 10

The weight update moved the prediction closer to the correct answer.

Did the Loss Improve?

Before updating the weight:

Prediction = 6
Target = 10

Loss = (10 - 6)²
Loss = 16

After updating the weight:

Prediction = 9.2
Target = 10

Loss = (10 - 9.2)²
Loss = 0.64

The loss dropped from 16 to 0.64.

This is the basic idea of learning: use the gradient to change the weights in a direction that can reduce the loss.

What Happens With Many Weights?

A real neural network has many weights. Each weight gets its own gradient.

Weight 1 = 0.5
Gradient 1 = 0.2

Weight 2 = 0.8
Gradient 2 = -0.4

Weight 3 = 0.3
Gradient 3 = 0.1

Suppose the learning rate is 0.1.

Weight 1:
0.5 - (0.1 × 0.2)
= 0.48


Weight 2:
0.8 - (0.1 × -0.4)
= 0.84


Weight 3:
0.3 - (0.1 × 0.1)
= 0.29

Every weight can therefore receive a different update.

Where Does the Optimizer Come In?

In the simple gradient descent formula, we manually use the gradient and learning rate to calculate the update.

Gradient
    ↓
Learning Rate
    ↓
Calculate Update
    ↓
New Weight

In real deep learning frameworks, an optimizer such as SGD or Adam handles the parameter updates.

The important thing to understand now is:

Backpropagation
→ Calculates Gradients


Optimizer
→ Uses Gradients to Update Weights

Real-Life Example

Imagine you are walking down a mountain and want to reach the lowest point.

Current Position
      ↓
Look at the slope
      ↓
Gradient tells the direction
      ↓
Take a step
      ↓
Learning rate controls
the step size
      ↓
Repeat

A very large step can make you overshoot the best path. A very small step may take too long.

That is similar to choosing a learning rate in neural network training.

Complete Backpropagation Flow

Input
  ↓
Forward Propagation
  ↓
Prediction
  ↓
Calculate Loss
  ↓
Backpropagation
  ↓
Calculate Gradients
  ↓
Optimizer
  ↓
Update Weights
  ↓
New Prediction
  ↓
Lower Loss
  ↓
Repeat

This cycle is repeated many times during training. The network gradually adjusts its weights based on the gradients.

Remember This

Gradient
    ↓
Tells the direction of change


Learning Rate
    ↓
Controls the size of change


Weight Update
    ↓
New Weight = Old Weight
              - Learning Rate × Gradient


Repeat
    ↓
The model can gradually reduce
the loss