Test Data

What Is Test Data?

Test data is a portion of the original dataset that is kept aside and not used to train the model.

It is also not normally used to repeatedly make development decisions.

Instead, we use it near the end to estimate how well the final model performs on unseen examples.

Training, Validation, and Test

Why Keep the Test Data Separate?

Imagine you build a model and keep checking its performance against the test set.

Every time the test result influences a change to your model, you are indirectly using information from the test set during development.

A House Price Example

Suppose we have 10,000 historical house records.

We could create a simple example split:

The model learns from the training data.

We use the validation data while developing and choosing the model.

Finally, we evaluate the selected model on the test data.

The Test Data Should Be Unseen

When we say the test data is unseen, we mean the model should not have used those examples as training examples.

More importantly, we should avoid repeatedly making model-development decisions based on the test results.

What Does the Test Result Tell Us?

Suppose the final model achieves an accuracy of 92% on the test dataset.

That gives us evidence about how the model performs on data that was kept separate from the training and development process.

The exact metric depends on the Machine Learning problem. Accuracy is only one possible metric.

Test Performance Can Reveal Generalization

One of the important things we care about is whether the model learned useful patterns rather than simply fitting the training examples.

The difference between training and unseen-data performance can provide useful information about the model.

Don't Keep Rechecking the Test Set

Consider this workflow:

This turns the test set into another development dataset.

A better workflow is:

Test Data Is Not Always a Perfect Representation

A test score is useful, but it does not magically guarantee real-world performance.

If the test data is poorly collected or does not represent the environment where the model will be used, a strong test score can still be misleading.

Test Data in Python

In a typical workflow, the test set is separated before training and development decisions are made.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

model.fit(X_train, y_train)

predictions = model.predict(X_test)

Here, the model is fitted using the training data and predictions are then generated for the separate test data.

In a full training/validation/test workflow, the exact splitting strategy can be more involved.

The Final Exam Analogy

The Three Roles

Which Dataset Should Be Used for the Final Evaluation?

You have trained several models and used validation data to select the final approach.

Which dataset should you use now to estimate the final performance?