Why Do We Split the Data?
Before training a machine learning model, we usually divide the available data into different parts. This helps us train the model and then check whether it can make good predictions on data it has not seen before.
We don't want the model to simply memorize the data.
We want the model to learn useful patterns that it can use when it receives new data. Splitting the dataset gives us a way to train the model and then check how well it works on examples that were not used during training.
Imagine You Are Studying for an Exam
Think about how a student prepares for an exam.
A student studies examples and exercises before the actual exam. If the teacher gives exactly the same questions in the exam, the student may simply remember the answers.
The student learns from these examples.
This checks whether the student really understands the subject.
Machine learning works with a similar idea.
The model learns from training examples, but we also need examples it has not seen to check whether it has actually learned useful patterns.
What Happens If We Use the Same Data?
Suppose we have 1,000 house records and we use all 1,000 records to train the model.
Then we test the model using those same 1,000 records.
The model has already seen these examples during training.
Therefore, a very good result does not necessarily mean that the model will perform well when it encounters completely new houses.
Testing on the same data used for training can give us an overly optimistic idea of how well the model actually performs.
We Want to Test on Unseen Data
Instead of giving all the data to the model for training, we keep some data aside.
The model learns from the training portion.
Later, we can use the held-out portion to see how well the model performs on data that was not used during training.
Example: House Price Prediction
Suppose we have information about 10,000 houses.
We want to build a model that predicts the price of a house based on its features.
Historical house information and prices.
Some data is used for learning and some is kept aside.
The model learns patterns from one portion of the houses.
We then check how well those learned patterns work on houses that were not used during that training step.
Don't ask the model to prove that it remembers the examples it already saw. Ask it to perform on examples it has not seen.
Splitting Helps Us Detect Memorization
A model can sometimes learn the training examples too closely instead of learning general patterns.
This is called overfitting. We will study overfitting in much more detail later.
The model knows these examples because it learned from them.
The model struggles when the examples are different from what it saw before.
Comparing performance on training data with performance on unseen data helps us identify this kind of problem.
Why Not Just Use More Training Data?
More training data can often be useful, but it does not solve the basic problem of knowing whether the model works on unseen examples.
If every available example is used for training, we have no independent examples left to properly check the model's performance.
We need both learning data and data that helps us judge whether the learning was successful.
This Is Why We Split the Dataset
The main reason for splitting data is simple:
In machine learning, we care about how well a model works on new data—not just how well it remembers old data.
Split the Data So We Can Test What the Model Really Learned.
The model learns from some examples and is evaluated using examples that were not used for that learning. This gives us a more realistic idea of how the model will perform on new data.
Which Approach Is Better?
You have 10,000 house records.
Approach A: Train the model using all 10,000 records and evaluate it using the same 10,000 records.
Approach B: Use part of the data for training and keep another part aside so the model can be evaluated on examples it has not seen.
Approach B gives us a better indication of whether the model can generalize to new data.
The exact roles of the different portions—training, validation, and test data—are covered in the next topics.