How Do We Know If It's Good?
Building a Machine Learning model is only half of the problem. We also need a reliable way to determine whether its predictions are actually good.
A model is useful only if its predictions perform well on data it has not seen before.
We need to compare the model's predictions with the actual outcomes and measure how well they match.
A Prediction Can Be Wrong
Machine Learning models do not automatically produce perfect answers.
Suppose a model predicts that a house will cost $400,000, but the actual selling price is $430,000.
The prediction is not exactly correct. We therefore need a way to measure how far the prediction was from the actual value.
Predicted price: $400,000
Actual price: $430,000
Difference: $30,000
We Need a Measurement
A model's quality cannot be judged by simply saying "it looks good."
We need a measurable way to describe how well the model performs.
This measurement is called an evaluation metric.
Different Machine Learning problems use different metrics because different types of predictions have different meanings.
Numerical Prediction
Suppose we are predicting house prices.
The model might make several predictions:
- Actual: $300,000 → Predicted: $310,000
- Actual: $400,000 → Predicted: $390,000
- Actual: $500,000 → Predicted: $530,000
We can measure how far the predictions are from the actual prices.
Smaller prediction errors generally mean the model is performing better for this type of problem.
Classification
Classification is different because the output is a category rather than a numerical value.
Suppose our model predicts whether an email is Spam or Not Spam.
- Actual: Spam → Predicted: Spam
- Actual: Not Spam → Predicted: Not Spam
- Actual: Spam → Predicted: Not Spam
Here we can measure how many predictions were classified correctly and how many were incorrect.
Accuracy
One simple classification metric is accuracy.
Accuracy tells us what percentage of predictions were correct.
For example, if a model correctly classifies 90 out of 100 emails, its accuracy is 90%.
A model can have high accuracy while still doing a poor job on an important category. The right evaluation metric depends on the problem.
Test the Model on Unseen Data
There is a major problem if we evaluate a model only using the examples it already learned from.
A model could perform extremely well on its training examples simply because it has effectively memorized them.
What we really care about is whether the model can make useful predictions for new examples.
Training data helps the model learn. Unseen test data helps us determine whether the learned patterns actually generalize.
Why Can't We Just Test on the Training Data?
Imagine a student memorizes every question from a practice sheet.
If you give the student the exact same questions, they may get everything correct.
But that does not prove that the student understands the subject.
Machine Learning has a similar problem. A model that performs well on data it has already seen may not perform well on new data.
Generalization
The ability of a model to perform well on new, previously unseen examples is called generalization.
This is one of the central goals of Machine Learning.
We do not want a model that simply remembers the training examples. We want a model that learns useful patterns that continue to work on new data.
Training Performance vs Real Performance
Consider two models.
- Model A performs extremely well on training data but poorly on new data.
- Model B performs slightly worse on training data but performs well on new data.
Model B may be the more useful model because our goal is not to memorize old examples. Our goal is to make good predictions for future examples.
The Metric Depends on the Problem
There is no single metric that is perfect for every Machine Learning problem.
For numerical prediction, we may care about the size of prediction errors.
For classification, we may care about correct and incorrect categories, and sometimes about specific types of mistakes.
Later in the course, we will study these evaluation metrics in much more detail.
What Does "Good" Actually Mean?
A model is not automatically good just because it has a high metric score.
The result needs to be useful for the actual business or real-world problem.
For example, a fraud detection system might need to prioritize catching fraudulent transactions, while a house-price system may care about keeping numerical prediction errors small.
Always ask: "Good for what?" before deciding that a model is good.
Evaluate Models on the Right Data With the Right Metric
We judge a Machine Learning model by how well it performs on appropriate unseen data using metrics that match the actual problem.
Which Model Is More Useful?
Suppose two house-price models produce these results:
- Model A: excellent performance on training data, poor performance on new houses.
- Model B: slightly worse training performance, but much better predictions for new houses.
Model B is generally more useful because the real goal is to predict prices for houses the model has not already seen.