Test Data
Test data is the data kept aside until the model has been developed. It is used at the end to get a more independent idea of how well the model performs on data it has not seen during training or model selection.
Test data is the final exam for the model.
The model learns from training data. We use validation data while developing and choosing the model. Once those decisions are finished, test data gives us a final check using examples that were kept separate.
What Is Test Data?
Test data is a separate portion of the original dataset that is not used to train the model or make development decisions.
After the model has been trained and the important choices have been made, we use the test data to measure how well the final model performs.
Example: House Price Prediction
Suppose we have information about 10,000 houses and want to build a model that predicts house prices.
We divide the data into separate portions. For example, some records can be used for training, some for validation, and some for final testing.
The model learns relationships between house features and prices.
We compare choices and improve the model.
We evaluate the final model on data kept aside.
The test houses should not have been used to train the model or choose between different versions of the model.
Test Data Is Like a Final Exam
Think about a student preparing for an exam.
Learn the subject.
Check progress and improve.
Measure the final performance.
The important part is that the final exam contains questions the student did not use while preparing.
Test data serves a similar purpose for a machine learning model.
Why Should Test Data Stay Separate?
Suppose you repeatedly check your model on the test data and change the model whenever it performs poorly.
After enough changes, you are indirectly using the test results to make development decisions.
Use training and validation data for development.
Test results start influencing model decisions.
If the test results influence the model repeatedly, the test data is no longer a truly independent final check.
What Does the Test Set Tell Us?
After the model is finalized, we make predictions on the test data and compare those predictions with the actual results.
We can then use an appropriate evaluation metric to summarize how well the final model performed.
The exact metrics depend on the type of machine learning problem. We will study those metrics later.
Test Data Should Represent New Data
The goal of testing is to estimate how the model may perform when it encounters data from the same real-world problem that it has not seen before.
For example, if we build a model to classify emails as spam or not spam, the test set should contain separate email examples that were not used during model development.
This is why keeping the test set separate matters. We want the final measurement to give us useful information about generalization to new data.
Test Data Is Used for the Final Evaluation.
Training data teaches the model. Validation data helps us develop and choose the model. Test data is kept separate and used at the end to evaluate the final model on unseen examples.
Which Data Should Be Used?
You have finished training your model and have finished choosing its settings. You now want one final, independent measurement of its performance.
Test data should be used for the final evaluation because it was kept separate from the model's development process.
Don't Keep Tuning the Model Using the Test Set.
If you repeatedly use test results to change the model, the test set starts becoming part of the development process. That weakens its purpose as a final independent evaluation.