Understanding Loss Values

What Is a Loss Value?

A loss function takes the model's prediction and the actual target and produces a number.

Actual Answer
      +
Model Prediction
      ↓
Loss Function
      ↓
Loss Value

For example:

Loss = 0.25

This number represents the error according to the particular loss function being used.

Does Lower Loss Mean Better?

When comparing values produced by the same loss function on the same or comparable data, a lower loss generally indicates that the predictions are closer to the desired targets according to that loss.

Model A
Loss = 0.20


Model B
Loss = 0.80

If both models are being evaluated with the same loss function under comparable conditions, Model A has the lower loss.

Lower Loss
    ↓
Less Error
    ↓
Generally Better
according to that loss

Loss During Training

A neural network usually starts with weights that are not yet well adjusted. Its initial predictions may therefore have significant errors.

As training updates the weights, the model may produce better predictions.

Epoch 1 → Loss = 1.80
Epoch 2 → Loss = 1.20
Epoch 3 → Loss = 0.82
Epoch 4 → Loss = 0.51
Epoch 5 → Loss = 0.31

The loss is decreasing.

1.80
 ↓
1.20
 ↓
0.82
 ↓
0.51
 ↓
0.31

This is generally a good sign: the model is reducing the error measured by the selected loss function on that training data.

Simple Example

Suppose we are training a model to predict house prices.

Actual Price = $300,000

Model A:
Prediction = $250,000

Model B:
Prediction = $295,000

Model B is closer to the actual price, so a suitable regression loss would generally give Model B a smaller loss.

Model A
Large Prediction Error
        ↓
Larger Loss


Model B
Small Prediction Error
        ↓
Smaller Loss

What Does a Loss of 0 Mean?

For many common loss functions, a loss of zero means the predictions perfectly match the targets in the way that particular loss function measures error.

For example, with MSE:

Actual     = [10, 20, 30]
Prediction = [10, 20, 30]

Errors = [0, 0, 0]

MSE = 0

There is no squared prediction error, so the MSE is zero.

Not All Loss Values Have the Same Scale

This is one of the most important things to understand.

Different loss functions calculate error differently. Therefore, their numerical values cannot automatically be compared.

Model A

MSE = 0.50


Model B

Binary Cross-Entropy = 0.30

You cannot conclude that Model B is better simply because 0.30 is smaller than 0.50.

They are different loss functions measuring error in different ways.

Compare:

MSE ↔ MSE
BCE ↔ BCE
Categorical CE ↔ Categorical CE

Do not directly compare:

MSE ↔ BCE

Example With MSE

Suppose two regression models are evaluated using MSE on the same dataset.

Model A
MSE = 120


Model B
MSE = 75

Because the same loss function is being used, we can compare the values.

75 < 120

Model B
    ↓
Lower MSE
    ↓
Less squared error
    ↓
Better according to MSE

Example With Binary Cross-Entropy

Now suppose two binary-classification models are evaluated using BCE.

Model A
BCE = 0.42


Model B
BCE = 0.18

Since both use BCE:

0.18 < 0.42

Model B
    ↓
Lower BCE
    ↓
Better according to BCE

Loss Across Epochs

An epoch means the model has gone through the training dataset once.

Suppose the training loss looks like this:

Epoch 1  → 1.50
Epoch 2  → 1.10
Epoch 3  → 0.80
Epoch 4  → 0.55
Epoch 5  → 0.40
Epoch 6  → 0.32

The general trend is downward.

1.50
 ↓
1.10
 ↓
0.80
 ↓
0.55
 ↓
0.40
 ↓
0.32

This suggests that the model is learning patterns that reduce the measured training error.

Loss Does Not Have to Decrease Every Step

Do not expect the loss to decrease perfectly after every update.

For example:

Epoch 1 → 1.50
Epoch 2 → 1.10
Epoch 3 → 1.18
Epoch 4 → 0.91
Epoch 5 → 0.72
Epoch 6 → 0.75
Epoch 7 → 0.54

There are small increases at some points, but the overall trend is downward.

Training can fluctuate because of factors such as mini-batches, optimization settings, and the structure of the data.

Training Loss vs Validation Loss

Looking only at training loss is not enough.

We also want to know whether the model works well on data that was not used to update its weights.

Training Data
      ↓
Training Loss


Validation Data
      ↓
Validation Loss

Training loss tells us how well the model fits the training data.

Validation loss helps us evaluate how well the model generalizes to unseen validation data.

Example of Healthy Training

Epoch    Training Loss    Validation Loss

1        1.20              1.30
2        0.90              1.00
3        0.70              0.82
4        0.55              0.68
5        0.44              0.57
6        0.36              0.49

Both losses are decreasing.

Training Loss
1.20 → 0.36


Validation Loss
1.30 → 0.49

This is generally a healthy pattern because performance is improving on both the training and validation data.

What If Training Loss Keeps Falling?

A decreasing training loss does not automatically mean the model is improving on unseen data.

Consider:

Epoch    Training Loss    Validation Loss

1        1.20              1.30
2        0.80              0.90
3        0.55              0.70
4        0.38              0.65
5        0.25              0.72
6        0.16              0.88
7        0.10              1.05

Training loss keeps decreasing, but validation loss starts increasing.

Training Loss
        ↓
Keeps decreasing


Validation Loss
        ↓
Starts increasing

This can be a sign of overfitting: the model is fitting the training data increasingly well but generalizing worse to validation data.

Finding the Best Point

In the previous example, validation loss reached its lowest value around Epoch 4 before starting to increase.

Epoch 1 → Validation Loss = 1.30
Epoch 2 → Validation Loss = 0.90
Epoch 3 → Validation Loss = 0.70
Epoch 4 → Validation Loss = 0.65  ← Lowest
Epoch 5 → Validation Loss = 0.72
Epoch 6 → Validation Loss = 0.88
Epoch 7 → Validation Loss = 1.05

This is why validation loss is useful when monitoring training.

Loss vs Accuracy

Loss and accuracy are related, but they are not the same measurement.

Loss
 ↓
Measures prediction error
using a mathematical function


Accuracy
 ↓
Measures how many predictions
are classified correctly

For example, two models could both classify 90 out of 100 samples correctly but have different loss values because their predicted probabilities are different.

Model A
Accuracy = 90%
BCE = 0.30


Model B
Accuracy = 90%
BCE = 0.55

The accuracy is identical, but the BCE values differ. This is because the loss also considers the model's probability predictions.

Why Confidence Affects Loss

Consider a binary classification problem where the actual answer is 1.

Model A
Prediction = 0.90


Model B
Prediction = 0.55


Model C
Prediction = 0.01

With BCE:

0.90
 ↓
Correct and confident
 ↓
Small loss


0.55
 ↓
Correct but uncertain
 ↓
Larger loss


0.01
 ↓
Confidently wrong
 ↓
Very large loss

This shows why looking only at whether the predicted class is correct can hide important information.

Think About Loss as a Curve

During training, we can record the loss after each epoch.

Loss

1.5 | *
    |  \
1.0 |   *
    |     \
0.5 |       *---*
    |             \
0.0 +----------------
       Training →

The important thing is the overall behavior of the loss rather than one isolated value.

Three Questions to Ask When Looking at Loss

Question 1
↓
Is the loss decreasing?


Question 2
↓
Is validation loss also improving?


Question 3
↓
Am I comparing the same loss function
under comparable conditions?

These questions are much more useful than simply asking, "Is this loss number small?"

Complete Example

Suppose we train a binary classification model for ten epochs.

Epoch    Training Loss    Validation Loss

1        0.90              0.95
2        0.72              0.78
3        0.58              0.64
4        0.47              0.53
5        0.39              0.45
6        0.32              0.40
7        0.27              0.38
8        0.23              0.41
9        0.19              0.47
10       0.16              0.55

What do we see?

Training Loss
0.90 → 0.16
        ↓
Keeps decreasing


Validation Loss
0.95 → 0.38
        ↓
Then increases
        ↓
0.41 → 0.47 → 0.55

The model continues fitting the training data, but its validation performance starts getting worse.

This is a classic pattern to investigate for overfitting.

How to Read Loss During Training

Training
    ↓
Calculate Loss
    ↓
Record Loss
    ↓
Compare With Previous Epoch
    ↓
Check Validation Loss
    ↓
Look at the Overall Trend
    ↓
Decide Whether Training Is Improving