Missing Data

Create a DataFrame With Missing Values

Let's create a small dataset where some values are missing.

import pandas as pd

students = pd.DataFrame({
    "Name": ["Raj", "John", "Sarah", "David", "Emma"],
    "Age": [29, None, 27, 30, None],
    "Score": [85, 90, None, 92, 65],
    "City": [
        "Hyderabad",
        "London",
        None,
        "Hyderabad",
        "London"
    ]
})

print(students)

    Name   Age  Score       City
0    Raj  29.0   85.0  Hyderabad
1   John   NaN   90.0     London
2  Sarah  27.0    NaN        NaN
3  David  30.0   92.0  Hyderabad
4   Emma   NaN   65.0     London

Pandas represents missing numeric values as NaN.

None was used when creating the data, but Pandas converts the missing values into its missing-data representation.

Find Missing Values With isna()

The first thing you should do with a dataset is often check whether missing values exist.

print(students.isna())

    Name    Age  Score   City
0  False  False  False  False
1  False   True  False  False
2  False  False   True   True
3  False  False  False  False
4  False   True  False  False

True means the value is missing. False means the value exists.

Count Missing Values

Usually, you don't want to inspect every True and False manually. You want a count.

print(students.isna().sum())

Name     0
Age      2
Score    1
City     1
dtype: int64

This tells us:

Age      → 2 missing values
Score    → 1 missing value
City     → 1 missing value
Name     → 0 missing values

This is one of the most useful commands when inspecting a new dataset.

Find Values That Are Not Missing

Sometimes you want the opposite: values that are present.

Use notna().

print(students.notna())

Here:

isna()
→ Is this value missing?

notna()
→ Is this value available?

Filter Rows With Missing Values

You can use isna() as a filtering condition.

For example, find students whose score is missing.

result = students[
    students["Score"].isna()
]

print(result)

    Name   Age  Score City
2  Sarah  27.0    NaN  NaN

This is useful when you want to inspect the rows that need attention.

Remove Rows With Missing Data

One simple solution is to remove rows containing missing values.

result = students.dropna()

print(result)

    Name   Age  Score       City
0    Raj  29.0   85.0  Hyderabad
3  David  30.0   92.0  Hyderabad

Pandas removed every row containing at least one missing value.

In our dataset, only Raj and David have complete information.

Remove Columns With Missing Data

Sometimes the problem is a column rather than a row. You can remove columns containing missing values.

result = students.dropna(axis=1)

print(result)

    Name
0    Raj
1   John
2  Sarah
3  David
4   Emma

axis=1 means operate on columns.

axis=0
→ rows

axis=1
→ columns

Be careful with this approach. Removing a whole column just because it contains a few missing values can throw away useful information.

Fill Missing Values With fillna()

Instead of deleting missing data, we can replace it.

For example, replace missing ages with 0.

result = students["Age"].fillna(0)

print(result)

0    29.0
1     0.0
2    27.0
3    30.0
4     0.0
Name: Age, dtype: float64

The missing values were replaced with 0.

But there is a problem: does an age of 0 actually mean "unknown age"?

Usually, no. This is why blindly filling missing values is a bad habit.

Fill Missing Values With the Mean

For numeric data, one common approach is to replace missing values with the column's mean.

mean_age = students["Age"].mean()

students["Age"] = students["Age"].fillna(
    mean_age
)

print(students)

The mean is calculated from the available ages.

29 + 27 + 30
─────────────
      3

= 28.67

The missing ages are then replaced with approximately 28.67.

This is more meaningful than replacing an unknown age with zero.

Fill With the Median

Another common option is the median.

median_age = students["Age"].median()

students["Age"] = students["Age"].fillna(
    median_age
)

The median is the middle value when the values are sorted.

27
29
30

Median = 29

Median can be useful when the data contains extreme values because it is less affected by outliers than the mean.

Fill Missing Text Values

For text columns, using a numeric mean makes no sense. You can use a meaningful text value instead.

students["City"] = students["City"].fillna(
    "Unknown"
)

print(students)

The missing city becomes:

Unknown

This explicitly tells us that the city is not known.

Forward Fill

Pandas can also fill a missing value using the previous available value.

result = students["Score"].ffill()

print(result)

For example, if the data looks like:

85
90
NaN
92

forward fill uses the previous value:

85
90
90
92

This can be useful for time-series data where the previous value is a reasonable replacement.

Backward Fill

Backward fill does the opposite. It uses the next available value.

result = students["Score"].bfill()

print(result)

For example:

85
NaN
90

backward fill produces:

85
90
90

Check Missing Data Before Training

Before sending a dataset into a Machine Learning model, it is useful to check how much data is missing.

missing = students.isna().sum()

print(missing)

You can also calculate the percentage of missing values.

missing_percentage = (
    students.isna().mean() * 100
)

print(missing_percentage)

This tells you not only how many values are missing, but how large the missing-data problem is relative to the entire column.

Real-World AI Example

Imagine a customer dataset:

Customer    Age    Income    Purchases
Raj         29     50000        12
John        NaN    60000         8
Sarah       31     NaN          15
David       42     80000        20

Before training an AI model, you need to decide what to do with the missing Age and Income values.

Age
→ Could use median age

Income
→ Could use median income

Purchases
→ No missing values

The important point is that there is no universal "correct" way to handle missing data. The correct strategy depends on what the column means and why the data is missing.

Simple Mental Model

Dataset

Name     Age     Score
Raj      29       85
John     ???      90
Sarah    27       ???
David    30       92

1. Remove the row

2. Fill the missing value

3. Keep it and handle it later