Training Loop

What Is a Training Loop?

A neural network does not learn everything from the dataset in one step.

Instead, it repeatedly performs the same sequence of operations.

Training Data
      ↓
Make Prediction
      ↓
Calculate Loss
      ↓
Calculate Gradients
      ↓
Update Weights
      ↓
Repeat
      ↑
      └───────────────

This repeated process is called the training loop.

The goal is to gradually adjust the model's weights so that its predictions become better and its loss becomes smaller.

A Simple Example

Imagine a model that predicts whether a student will pass based on the number of hours they study.

Input:
Study Hours = 5

Actual Answer:
Pass = 1

Model Prediction:
Pass = 0.60

The prediction is not perfect. The model calculates how wrong the prediction is.

Prediction
    ↓
0.60

Actual
    ↓
1.00

    ↓

Calculate Loss

Backpropagation then calculates how the weights contributed to the error.

Loss
  ↓
Gradients
  ↓
Update Weights
  ↓
Better Prediction
  ↓
Repeat

The Four Main Steps

A basic training loop contains four important operations.

1. Forward Pass
   → Make a prediction

2. Calculate Loss
   → Measure how wrong the prediction is

3. Backward Pass
   → Calculate gradients

4. Update Weights
   → Improve the model

After the weights are updated, the process starts again with another batch.

Step 1 — Forward Pass

First, the training data is given to the neural network.

Input Data
    ↓
Neural Network
    ↓
Prediction

For example:

Study Hours = 5

Model Prediction = 0.60

The model is saying there is approximately a 60% predicted probability for the positive class in this simplified example.

Step 2 — Calculate Loss

The model's prediction is compared with the correct answer.

Prediction = 0.60
Actual     = 1.00

        ↓

Calculate Loss

The loss tells us how far the prediction is from the target.

A lower loss generally means the model's predictions are closer to the targets for that objective.

Step 3 — Calculate Gradients

The model now needs to determine how its weights should change to reduce the loss.

Loss
 ↓
Backpropagation
 ↓
Gradients

A gradient tells the optimizer how changing a parameter affects the loss.

The gradients are then used to determine the direction and size of the weight updates.

Step 4 — Update Weights

The optimizer uses the gradients and learning rate to update the model's weights.

Old Weight
    ↓
Gradient + Learning Rate
    ↓
New Weight

A simplified update looks like this:

new_weight =
    old_weight - learning_rate × gradient

The updated weights are used during the next iteration.

The Complete Training Loop

                 ┌─────────────────────┐
                 │     Training Data   │
                 └──────────┬──────────┘
                            ↓
                 ┌─────────────────────┐
                 │    Forward Pass     │
                 └──────────┬──────────┘
                            ↓
                 ┌─────────────────────┐
                 │   Calculate Loss    │
                 └──────────┬──────────┘
                            ↓
                 ┌─────────────────────┐
                 │   Backpropagation   │
                 └──────────┬──────────┘
                            ↓
                 ┌─────────────────────┐
                 │   Update Weights    │
                 └──────────┬──────────┘
                            ↓
                         Repeat
                            │
                            └──────────────→

This loop continues for many iterations and usually for multiple epochs.

Training Loop With Batches

In practice, the entire dataset is often divided into smaller batches.

Suppose we have 1,000 training examples and a batch size of 100.

1,000 examples
       ↓
10 batches
       ↓

Batch 1 → Training Step
Batch 2 → Training Step
Batch 3 → Training Step
...
Batch 10 → Training Step

       ↓

1 Epoch Complete

Each batch produces one iteration of the training loop.

Training Loop With Epochs

The training loop normally runs for multiple epochs.

Epoch 1
 ├── Batch 1 → Forward → Loss → Backward → Update
 ├── Batch 2 → Forward → Loss → Backward → Update
 ├── Batch 3 → Forward → Loss → Backward → Update
 └── ...

Epoch 2
 ├── Batch 1 → Forward → Loss → Backward → Update
 ├── Batch 2 → Forward → Loss → Backward → Update
 ├── Batch 3 → Forward → Loss → Backward → Update
 └── ...

Epoch 3
 ├── Batch 1 → Forward → Loss → Backward → Update
 ├── Batch 2 → Forward → Loss → Backward → Update
 ├── Batch 3 → Forward → Loss → Backward → Update
 └── ...

Every batch creates a training iteration, while processing all batches once completes one epoch.

Simple Training Loop With Python

Here is a simplified version of what a training loop looks like:

epochs = 3

for epoch in range(epochs):

    prediction = model(input_data)

    loss = calculate_loss(
        prediction,
        target
    )

    gradient = calculate_gradient(loss)

    weight = weight - learning_rate * gradient

    print("Epoch:", epoch + 1)
    print("Loss:", loss)

The important part is the repeated sequence:

Prediction
    ↓
Loss
    ↓
Gradient
    ↓
Weight Update
    ↓
Repeat

Training Loop With Batches in Python

A more realistic simplified structure looks like this:

epochs = 3

for epoch in range(epochs):

    for batch in batches:

        # 1. Forward pass
        prediction = model(batch)

        # 2. Calculate loss
        loss = calculate_loss(
            prediction,
            batch.target
        )

        # 3. Calculate gradients
        gradients = calculate_gradients(loss)

        # 4. Update weights
        update_weights(gradients)

    print("Epoch:", epoch + 1)

The outer loop handles epochs. The inner loop handles the batches inside each epoch.

for epoch
    ↓
    for batch
        ↓
        Forward Pass
        ↓
        Loss
        ↓
        Gradients
        ↓
        Weight Update

Why Does the Loop Repeat?

One update is usually not enough for the model to learn a useful representation.

After every update, the weights are slightly different. The model then makes another prediction using those updated weights.

Before Training

Weights
  ↓
Poor Prediction
  ↓
Large Loss


After Update

New Weights
  ↓
Better Prediction
  ↓
Smaller Loss


Repeat...


After Many Updates

Improved Weights
  ↓
Better Predictions
  ↓
Lower Loss

This repeated improvement is the basic mechanism of gradient-based neural network training.

Training Loop Example

Suppose the model starts with a weight of 0.50.

Initial Weight = 0.50
Learning Rate = 0.10

During the first iteration, suppose the gradient is 0.40:

New Weight
=
0.50 - (0.10 × 0.40)

=
0.46

On the next iteration, suppose the gradient is 0.20:

New Weight
=
0.46 - (0.10 × 0.20)

=
0.44

The model repeatedly adjusts the weight based on the gradients calculated from the current training step.

Iteration 1 → Weight = 0.50 → 0.46
Iteration 2 → Weight = 0.46 → 0.44
Iteration 3 → Weight = ...
Iteration 4 → Weight = ...
...

Training Loop vs Iteration

Do not confuse these two terms.

Iteration
→ One execution of the training step.


Training Loop
→ The repeated process that performs many iterations.

For example:

Training Loop
│
├── Iteration 1
├── Iteration 2
├── Iteration 3
├── Iteration 4
├── Iteration 5
└── ...

Easy Way to Remember

Imagine learning to shoot a basketball.

Take a Shot
    ↓
See Where the Ball Went
    ↓
Find the Mistake
    ↓
Adjust Your Technique
    ↓
Take Another Shot
    ↓
Repeat

This is similar to a neural network:

Prediction
    ↓
Loss
    ↓
Gradients
    ↓
Update Weights
    ↓
Prediction Again
    ↓
Repeat

The model does not magically know the correct weights. It improves them through repeated updates.

Remember This

Training Loop

1. Take a batch of training data
2. Make a prediction
3. Calculate the loss
4. Calculate gradients
5. Update the weights
6. Move to the next batch
7. Repeat for more epochs

The most important idea is: the training loop repeatedly uses predictions, loss, gradients, and weight updates to improve the neural network.