Iterations

What Is an Iteration?

During training, the neural network does not usually process the entire dataset at once.

Instead, the dataset is divided into smaller groups called batches.

The model processes one batch and then updates its weights. That complete training step is called an iteration.

One Batch
    ↓
Make Prediction
    ↓
Calculate Loss
    ↓
Calculate Gradients
    ↓
Update Weights
    ↓
One Iteration Complete

A Simple Example

Suppose we have 10 training examples and choose a batch size of 2.

Total Training Examples = 10

Batch Size = 2

The dataset is divided into five batches:

Batch 1 → Examples 1, 2
Batch 2 → Examples 3, 4
Batch 3 → Examples 5, 6
Batch 4 → Examples 7, 8
Batch 5 → Examples 9, 10

Each batch produces one training iteration.

Batch 1 → Iteration 1
Batch 2 → Iteration 2
Batch 3 → Iteration 3
Batch 4 → Iteration 4
Batch 5 → Iteration 5

Therefore:

10 examples
÷
Batch size of 2
=
5 iterations per epoch

What Happens During One Iteration?

Let's look at exactly what happens during one iteration.

Batch
  ↓
Input Data
  ↓
Forward Propagation
  ↓
Prediction
  ↓
Calculate Loss
  ↓
Backpropagation
  ↓
Calculate Gradients
  ↓
Update Weights
  ↓
Iteration Complete

After this, the model takes the next batch and performs another iteration.

One Iteration Example

Imagine that our batch contains two students:

Student A → 2 study hours → Fail
Student B → 5 study hours → Pass

The neural network processes both examples together.

Batch
  ↓
[Student A, Student B]
  ↓
Make Predictions
  ↓
Calculate Loss
  ↓
Calculate Gradients
  ↓
Update Weights
  ↓
Iteration 1 Complete

The model then moves to the next batch.

Iteration vs Batch

These two terms are closely related, but they are not the same thing.

Batch
→ A group of training examples.


Iteration
→ One training step performed using that batch.

For example:

Batch 1
[Examples 1 - 10]
      ↓
Iteration 1

Batch 2
[Examples 11 - 20]
      ↓
Iteration 2

Batch 3
[Examples 21 - 30]
      ↓
Iteration 3

So the batch is the data group, while the iteration is the training step performed on that group.

Iteration vs Epoch

This distinction is extremely important.

Iteration
→ One training step using one batch.


Epoch
→ One complete pass through the entire dataset.

Suppose we have 100 training examples and a batch size of 10.

Dataset = 100 examples
Batch Size = 10

Batch 1 → Iteration 1
Batch 2 → Iteration 2
Batch 3 → Iteration 3
Batch 4 → Iteration 4
Batch 5 → Iteration 5
Batch 6 → Iteration 6
Batch 7 → Iteration 7
Batch 8 → Iteration 8
Batch 9 → Iteration 9
Batch 10 → Iteration 10

        ↓

All 100 examples processed

        ↓

1 Epoch Complete

Therefore:

10 iterations = 1 epoch

How to Calculate Iterations

When the dataset divides evenly into batches, we can use:

Iterations per Epoch
=
Number of Training Examples
÷
Batch Size

Example:

Training Examples = 1,000
Batch Size = 100

1,000 ÷ 100 = 10

Iterations per Epoch = 10

What If the Numbers Do Not Divide Evenly?

Suppose we have 105 training examples and a batch size of 10.

105 ÷ 10 = 10.5

We cannot have half an iteration. The final 5 examples still need to be processed.

Therefore, if the final smaller batch is kept:

Batch 1  → 10 examples
Batch 2  → 10 examples
Batch 3  → 10 examples
...
Batch 10 → 10 examples
Batch 11 → 5 examples

Total = 11 iterations

In practice, the exact behavior for the final incomplete batch depends on the training framework and its settings.

Iterations Across Multiple Epochs

Suppose we have:

Training Examples = 100
Batch Size = 10
Epochs = 3

One epoch contains:

100 ÷ 10 = 10 iterations

Because we train for 3 epochs:

Epoch 1 → 10 iterations
Epoch 2 → 10 iterations
Epoch 3 → 10 iterations

Total = 30 iterations

So the model performs 30 training steps in total.

Iterations and Weight Updates

In the standard mini-batch training process, each iteration results in one weight update.

Iteration 1
    ↓
Weight Update 1

Iteration 2
    ↓
Weight Update 2

Iteration 3
    ↓
Weight Update 3

Iteration 4
    ↓
Weight Update 4

This is why iterations are useful for understanding how many times the model's weights have been adjusted during training.

Iterations in Python

A simplified training loop might look like this:

epochs = 3
batch_size = 10

for epoch in range(epochs):

    for batch in batches:

        prediction = model(batch)

        loss = calculate_loss(
            prediction,
            target
        )

        gradients = calculate_gradients(loss)

        update_weights(gradients)

        print("Iteration complete")

The outer loop controls the epochs. The inner loop processes each batch.

Epoch 1
 ├── Iteration 1
 ├── Iteration 2
 ├── Iteration 3
 └── ...

Epoch 2
 ├── Iteration 1
 ├── Iteration 2
 ├── Iteration 3
 └── ...

Epoch 3
 ├── Iteration 1
 ├── Iteration 2
 ├── Iteration 3
 └── ...

Easy Way to Remember

Imagine a teacher grading 100 exam papers.

100 papers
    ↓
Divide into groups of 10
    ↓
10 groups

Group 1
    ↓
Review and improve teaching
    ↓
Iteration 1

Group 2
    ↓
Review and improve teaching
    ↓
Iteration 2

...

Group 10
    ↓
Review and improve teaching
    ↓
Iteration 10

    ↓

All papers processed
    ↓

1 Epoch

The analogy is simple: each group is a batch, processing one group is an iteration, and processing all groups once is an epoch.

Batch, Iteration, and Epoch Together

Training Dataset
        ↓
     Split into
      Batches
        ↓
┌───────────────────────┐
│ Batch 1               │
│   ↓                   │
│ Iteration 1           │
│   ↓                   │
│ Weight Update         │
└───────────────────────┘
        ↓
┌───────────────────────┐
│ Batch 2               │
│   ↓                   │
│ Iteration 2           │
│   ↓                   │
│ Weight Update         │
└───────────────────────┘
        ↓
        ...
        ↓
All Batches Processed
        ↓
     1 Epoch

This is the relationship you should remember:

Batch
= Group of training examples

Iteration
= One training step using one batch

Epoch
= One complete pass through all batches

Remember This

Example:

Dataset = 1,000 examples
Batch Size = 100
Epochs = 5

1 Epoch
= 1,000 ÷ 100
= 10 Iterations

5 Epochs
= 10 × 5
= 50 Iterations

Therefore:

1 iteration
= 1 batch processed
= 1 training step
= usually 1 weight update

The simplest definition to remember is: one iteration is one training step performed using one batch of data.