Training vs Validation vs Test
A machine learning dataset can be divided into three important parts: training data, validation data, and test data. Each part has a different job.
Train → Decide → Test
Training data teaches the model. Validation data helps us make decisions while developing the model. Test data gives us the final evaluation after development is finished.
The Three Parts
Think of the dataset as one large collection of examples that we divide according to purpose.
The model learns patterns from these examples.
We use these examples to compare choices and improve the model during development.
We use these examples for the final evaluation of the finished model.
Training Data — "Learn From This"
Training data is the data the model learns from.
The model studies these examples and learns relationships or patterns that can help it make predictions.
For example, if we are predicting house prices, the training data may contain house size, number of bedrooms, location, and actual prices.
By studying many such examples, the model learns useful relationships between the input information and the price.
Validation Data — "Help Me Decide"
Validation data is used while we are developing the model.
Suppose we have two possible models. We can use validation data to compare them and decide which one appears to perform better.
Based on the validation result, we might choose Model B for further development.
So validation data answers a question like:
Test Data — "How Good Is the Final Model?"
Test data is kept separate until the model development process is finished.
Once we have chosen the model and its important settings, we use the test data for the final evaluation.
Test data answers a different question:
One Simple Example
Imagine you are building a model that identifies whether an email is spam.
You have 10,000 labeled emails.
The model learns patterns associated with spam and normal emails.
We use validation results to help choose and improve the model.
We check the finished model on emails it did not use during development.
The three groups contain similar types of information, but their purpose is different.
The Exam Analogy
One of the easiest ways to remember the difference is to think about learning for an exam.
Learn the subject from lessons and practice material.
Check progress and decide what needs improvement.
Measure the final result using questions not used during preparation.
This analogy is not a perfect technical description, but it is a useful way to remember the purpose of each data split.
The Biggest Difference
The biggest difference is not the type of data. It is what we use the data for.
The Complete Picture
Now we can put everything together.
Train the Model. Use Validation to Develop It. Use Test Data to Judge the Final Model.
Training, validation, and test data are not three different types of machine learning. They are three different roles that portions of the dataset can play during the model-building process.
Can You Identify Each One?
1. The model needs examples to learn patterns.
2. You want to compare two models and decide which one to continue developing.
3. The model is finished and you want one final evaluation using data that was kept separate.