Prediction Probabilities

What Is a Prediction Probability?

In classification, the model can estimate the probability that an input belongs to each class.

Suppose our model predicts whether a student will Pass or Fail.

The model is saying that, based on the information it has learned, Pass is more likely than Fail.

predict() vs predict_proba()

Scikit-learn provides two useful methods for classification.

A Simple Example

Imagine our classifier receives a new student who studied for 6 hours.

The model might produce probabilities such as:

Because the probability for Pass is much higher, the classifier predicts Pass.

Getting Probabilities With Python

We can use predict_proba() to get the probabilities from a trained classifier.

probability = model.predict_proba([[6]])

print(probability)

The result may look similar to:

[[0.10 0.90]]

These two numbers represent the probability of each class.

Understanding model.classes_

How do we know which probability belongs to which class?

Scikit-learn provides classes_ so we can see the order used by the model.

print(model.classes_)

For our example, the output may be:

[0 1]

We defined:

Therefore:

Why Are Probabilities Useful?

A class prediction alone can hide important information.

Compare these two predictions:

Both students are classified as Pass, but the model is much more confident about Student B.

Probability Is Not the Same as Certainty

A probability from a Machine Learning model should not automatically be interpreted as absolute certainty.

If the model returns:

The model predicts Pass because 65% is higher than 35%. But this is not a very strong separation.

In contrast:

Here the model's probabilities are much more separated.

Prediction Probability and the Threshold

For binary classification, we often use a threshold such as 0.5 to convert probability into a class.

So if the model gives:

Pass probability = 0.80

and our threshold is 0.50:

0.80 >= 0.50

Prediction = Pass

Complete Python Example

Now let's combine training, prediction, and probability into one small example.

from sklearn.linear_model import LogisticRegression

# Training data
X = [
    [1],
    [2],
    [3],
    [5],
    [6],
    [7]
]

# Labels
y = [
    0,
    0,
    0,
    1,
    1,
    1
]

# Create the model
model = LogisticRegression()

# Train the model
model.fit(X, y)

# New student
new_student = [[6]]

# Predict the class
prediction = model.predict(new_student)

# Get probabilities
probability = model.predict_proba(new_student)

print("Classes:", model.classes_)
print("Prediction:", prediction)
print("Probabilities:", probability)

The important part is understanding the three outputs:

A Real-World Example

Prediction probabilities become especially useful when the decision is important.

Instead of treating every prediction as simply Yes or No, probabilities can provide more information about the model's output.

Check Your Understanding