Loss Functions
A loss function tells a neural network how wrong its prediction is. It converts the difference between the predicted value and the actual value into a number that the model can use to improve.
A loss function measures how wrong a model's prediction is.
The model makes a prediction, compares it with the correct answer, and calculates a loss. During training, the model tries to reduce this loss.
What Is a Prediction?
A neural network receives input data and produces an output. That output is called a prediction.
For example, imagine a model predicting the price of a house.
Actual house price = ₹50 lakh Model prediction = ₹45 lakh
The model made a prediction, but the prediction is not exactly correct.
We now need a way to measure how wrong the prediction is.
What Is Loss?
Loss is a numerical value that represents the error of a model's prediction.
In simple terms:
Small loss → Prediction is close to the correct answer Large loss → Prediction is far from the correct answer
During training, the neural network tries to find weights that produce smaller and smaller loss.
A Very Simple Example
Suppose a model predicts the temperature tomorrow.
Actual temperature = 30°C Predicted temperature = 27°C
The prediction is 3 degrees away from the actual value.
Error = Actual - Prediction = 30 - 27 = 3
A loss function takes this difference and converts it into a numerical loss that the model can use during training.
Why Can't We Just Use the Error?
Sometimes the error can be positive and sometimes it can be negative.
Actual = 30
Prediction = 27
Error = 30 - 27
= 3
Actual = 30
Prediction = 33
Error = 30 - 33
= -3
If we simply added these errors together, positive and negative values could cancel each other out.
Loss functions solve this problem by transforming the errors into useful numerical values.
Mean Squared Error
Mean Squared Error, commonly called MSE, is a popular loss function for regression problems.
It works by taking the difference between the actual and predicted values, squaring the difference, and then calculating the average.
MSE = average of (actual - prediction)²
Let's use a very simple example with one prediction.
Actual = 10
Prediction = 8
Error = 10 - 8
= 2
Squared Error
= 2²
= 4
Loss = 4
The square also makes sure that the loss is positive.
Why Does MSE Square the Error?
Squaring the error has an important effect. Larger mistakes receive much larger penalties.
Error = 2 2² = 4 Error = 5 5² = 25 Error = 10 10² = 100
Notice how quickly the loss increases as the error becomes larger.
This makes MSE useful when large prediction errors should be penalized strongly.
MSE With Multiple Predictions
Real models usually make many predictions. MSE calculates the average squared error across those predictions.
Actual: 10 20 30 Predicted: 8 18 33 Errors: 2 2 -3 Squared: 4 4 9 Average: (4 + 4 + 9) / 3 = 17 / 3 ≈ 5.67
So the MSE for these predictions is approximately 5.67.
Binary Cross Entropy
MSE is commonly used for regression problems. For binary classification, another important loss function is Binary Cross Entropy.
Binary classification means there are two possible classes.
Spam or Not Spam Fraud or Not Fraud Cat or Not Cat Yes or No
A model might produce a probability between 0 and 1.
0.95 → Very confident 0.10 → Not very confident
Simple Cross Entropy Example
Imagine a model is predicting whether an email is spam.
Actual answer: Spam Correct label: 1 Model prediction: 0.90
The model is quite confident that the email is spam. Therefore, the loss should be relatively small.
Now imagine the model produces:
Actual answer: Spam Correct label: 1 Model prediction: 0.05
This prediction is very wrong and should receive a much larger loss.
This is the basic idea behind binary cross entropy: confident correct predictions receive low loss, while confident incorrect predictions receive high loss.
Why Loss Functions Are Important
A neural network needs a way to know whether it is getting better or worse.
Model makes prediction
↓
Calculate Loss
↓
How wrong is the model?
↓
Calculate Gradients
↓
Update Weights
↓
Make another prediction
Without a loss function, the training process would not have a clear numerical objective to minimize.
Lower Loss Is Better
During training, the goal is generally to reduce the loss.
Training 1 Loss = 10 Training 2 Loss = 7 Training 3 Loss = 4 Training 4 Loss = 2
The decreasing loss suggests that the model is becoming better at matching its predictions to the training targets.
However, a low training loss alone does not guarantee that the model will perform well on new, unseen data.
Loss Function vs Error
These two ideas are related, but they are not exactly the same.
Error Difference between: Actual value and Predicted value Loss A mathematical measure that represents how bad the prediction is
For example, MSE takes prediction errors, squares them, and averages them to produce a loss value.
Choosing a Loss Function
Different AI problems can require different loss functions.
Regression → Predict a number → MSE is commonly used Binary Classification → Two possible classes → Binary Cross Entropy Multi-Class Classification → Multiple possible classes → Cross Entropy
The loss function should match the type of prediction the model is trying to make.
Simple Python Example
We can calculate a simple squared error using Python.
actual = 10 predicted = 8 error = actual - predicted loss = error ** 2 print(loss)
Output:
4
This is the basic idea behind the squared-error part of MSE.
Loss in a Neural Network
Now let's connect loss with the neural network concepts we have already learned.
Input ↓ Neural Network ↓ Activation Functions ↓ Prediction ↓ Loss Function ↓ Loss ↓ Gradients ↓ Weight Updates
The loss function is therefore a bridge between the model's prediction and the process used to improve the model.
What Happens During Training?
A neural network does not usually become accurate in one step. It repeats the learning process many times.
Step 1 Make prediction Step 2 Calculate loss Step 3 Calculate gradients Step 4 Update weights Step 5 Make prediction again Step 6 Calculate loss again Repeat...
Ideally, the model gradually finds weights that produce better predictions and lower loss.
The Connection to Gradient Descent
In the previous Calculus lesson, we learned about gradients and gradient descent.
Loss functions give gradient descent something to minimize.
Loss Function
↓
Calculate Loss
↓
Gradient
↓
Gradient Descent
↓
Change Weights
↓
Lower Loss
This is why loss functions are a central part of machine learning. The model is essentially trying to find parameters that minimize the chosen loss.
A Real-World Example
Imagine an AI model predicting house prices.
Actual price = ₹60 lakh Prediction 1 = ₹45 lakh Large error → Large loss After training: Actual price = ₹60 lakh Prediction 2 = ₹58 lakh Smaller error → Smaller loss
The goal of training is not simply to make one prediction better. The model learns parameters that allow it to make better predictions across many examples.
Loss Function in Simple Words
Think of a teacher checking an exam.
Student gives an answer
↓
Teacher checks the answer
↓
Wrong by a little
→ Small penalty
Wrong by a lot
→ Large penalty
A loss function plays a similar role for an AI model. It gives the model a numerical measurement of how bad its predictions are.
Loss functions tell a neural network how wrong its predictions are.
MSE is commonly used for regression and measures squared prediction errors. Binary Cross Entropy is commonly used for binary classification and evaluates predicted probabilities. During training, the model uses the loss together with gradients to update its weights and improve its predictions.