Batch Size

What Is Batch Size?

A batch is a small group of training examples taken from the complete training dataset.

Batch size tells us how many examples are inside that group.

For example, if we have 100 training examples and choose a batch size of 10:

Total Training Data = 100 examples

Batch Size = 10

Batch 1 → Examples 1 - 10
Batch 2 → Examples 11 - 20
Batch 3 → Examples 21 - 30
...
Batch 10 → Examples 91 - 100

The model processes 10 examples, calculates the error, and then updates its weights.

A Simple Example

Imagine we have 10 students in our training dataset. We want a neural network to predict whether each student will pass an exam based on study hours.

Student    Study Hours    Result

A          1              Fail
B          2              Fail
C          3              Fail
D          4              Pass
E          5              Pass
F          6              Pass
G          7              Pass
H          8              Pass
I          9              Pass
J          10             Pass

Suppose we choose:

Batch Size = 2

The 10 students are divided into 5 batches.

Batch 1 → A, B
Batch 2 → C, D
Batch 3 → E, F
Batch 4 → G, H
Batch 5 → I, J

What Happens Inside a Batch?

The neural network processes the examples in a batch, calculates their loss, calculates gradients, and then updates the weights.

Batch
  ↓
Make Predictions
  ↓
Calculate Loss
  ↓
Calculate Gradients
  ↓
Update Weights
  ↓
Next Batch

So if the batch size is 2, the model uses 2 training examples before making one weight update.

Complete Example

Suppose we have 10 training examples and a batch size of 2.

Total Examples = 10
Batch Size = 2

Batch 1
Examples 1, 2
     ↓
Prediction
     ↓
Loss
     ↓
Weight Update

Batch 2
Examples 3, 4
     ↓
Prediction
     ↓
Loss
     ↓
Weight Update

Batch 3
Examples 5, 6
     ↓
Prediction
     ↓
Loss
     ↓
Weight Update

Batch 4
Examples 7, 8
     ↓
Prediction
     ↓
Loss
     ↓
Weight Update

Batch 5
Examples 9, 10
     ↓
Prediction
     ↓
Loss
     ↓
Weight Update

After all 5 batches have been processed, the entire training dataset has been processed once.

5 Batches
    ↓
1 Complete Dataset
    ↓
1 Epoch

Batch Size and Epoch

Batch size and epoch are related, but they mean different things.

Batch Size
→ How many examples are processed before one weight update.


Epoch
→ One complete pass through the entire training dataset.

For example:

Training Examples = 100
Batch Size = 10

100 ÷ 10 = 10 batches

Therefore:

1 Epoch = 10 batches

When Are Weights Updated?

This is one of the most important things to understand.

With a batch size of 10, the model does not normally update its weights after every individual example.

Instead, it processes the batch and then performs a weight update.

10 Examples
    ↓
Calculate Predictions
    ↓
Calculate Loss
    ↓
Calculate Gradients
    ↓
Update Weights Once

Then the next batch is processed.

Batch Size = 1

If the batch size is 1, the model processes one training example before each weight update.

Example 1
   ↓
Weight Update

Example 2
   ↓
Weight Update

Example 3
   ↓
Weight Update

Example 4
   ↓
Weight Update

This is called stochastic or online training.

The updates can be noisy because each update is based on only one example.

Small Batch Size

Suppose we choose:

Batch Size = 4

The model processes four examples before updating its weights.

Examples 1 - 4
      ↓
Weight Update

Examples 5 - 8
      ↓
Weight Update

Examples 9 - 12
      ↓
Weight Update

Small batches use less memory, but they can produce more frequent and sometimes noisier updates.

Large Batch Size

Suppose we have 1,000 training examples and choose:

Batch Size = 500

The model processes 500 examples before updating its weights.

Batch 1
Examples 1 - 500
      ↓
Weight Update

Batch 2
Examples 501 - 1000
      ↓
Weight Update

Larger batches can make better use of hardware such as GPUs, but they require more memory.

Compare Different Batch Sizes

Dataset = 100 examples


Batch Size = 1

100 batches
100 weight updates per epoch


Batch Size = 10

10 batches
10 weight updates per epoch


Batch Size = 25

4 batches
4 weight updates per epoch


Batch Size = 100

1 batch
1 weight update per epoch

Notice the important relationship:

More examples per batch
        ↓
Fewer batches
        ↓
Fewer weight updates per epoch

Batch Size and Iterations

An iteration is one training step using one batch.

Therefore, the number of iterations in one epoch depends on the dataset size and batch size.

Iterations per Epoch
=
Number of Training Examples
÷
Batch Size

For example:

Training Examples = 1,000
Batch Size = 100

1,000 ÷ 100 = 10

Therefore:

10 iterations = 1 epoch

Batch Size in Python

A simplified example can look like this:

training_data = [
    1, 2, 3, 4,
    5, 6, 7, 8,
    9, 10
]

batch_size = 2

for i in range(0, len(training_data), batch_size):

    batch = training_data[i:i + batch_size]

    print("Batch:", batch)

The output would be:

Batch: [1, 2]
Batch: [3, 4]
Batch: [5, 6]
Batch: [7, 8]
Batch: [9, 10]

Each group contains two examples because the batch size is 2.

Batch Size Inside a Training Loop

A simplified neural-network training loop looks like this:

for epoch in range(epochs):

    for batch in batches:

        prediction = model(batch)

        loss = calculate_loss(
            prediction,
            target
        )

        gradients = calculate_gradients(loss)

        update_weights(gradients)

The outer loop controls the number of epochs. The inner loop processes the batches.

Epoch
 ├── Batch 1 → Weight Update
 ├── Batch 2 → Weight Update
 ├── Batch 3 → Weight Update
 └── Batch 4 → Weight Update

        ↓

Epoch Complete

Why Does Batch Size Matter?

Batch size affects memory usage, training speed, and how frequently the model updates its weights.

Small Batch
    ↓
Less memory
    ↓
More frequent updates
    ↓
More noisy updates


Large Batch
    ↓
More memory
    ↓
Fewer updates
    ↓
More stable updates

There is no single batch size that is always best. The right value depends on the dataset, model, hardware, and training problem.

Easy Way to Remember

Imagine a teacher checking 100 exam papers.

Batch Size = 1

Teacher checks 1 paper
        ↓
Reviews the result
        ↓
Adjusts teaching approach


Batch Size = 10

Teacher checks 10 papers
        ↓
Reviews the results
        ↓
Adjusts teaching approach


Batch Size = 100

Teacher checks all 100 papers
        ↓
Reviews the results
        ↓
Adjusts teaching approach

The idea is similar in neural-network training: the batch determines how many examples are considered before a weight update.

Remember This

Batch Size
=
Number of training examples processed
before one weight update.


Example:

Dataset = 100 examples
Batch Size = 10

100 ÷ 10 = 10 batches

Therefore:

1 Epoch
= 10 batches
= 10 iterations
= 10 weight updates

The simplest definition to remember is: batch size tells the model how many training examples to process before updating its weights.