Filtering Data

Start With a DataFrame

We will use a simple student dataset throughout this lesson.

import pandas as pd

students = pd.DataFrame({
    "Name": ["Raj", "John", "Sarah", "David", "Emma"],
    "Age": [29, 32, 27, 30, 25],
    "Score": [85, 90, 78, 92, 65],
    "City": [
        "Hyderabad",
        "London",
        "New York",
        "Hyderabad",
        "London"
    ]
})

print(students)

    Name  Age  Score       City
0    Raj   29     85  Hyderabad
1   John   32     90     London
2  Sarah   27     78   New York
3  David   30     92  Hyderabad
4   Emma   25     65     London

Basic Filtering

Suppose we want only students whose score is greater than 80.

high_scores = students[students["Score"] > 80]

print(high_scores)

    Name  Age  Score       City
0    Raj   29     85  Hyderabad
1   John   32     90     London
3  David   30     92  Hyderabad

Pandas checks the Score column for every row.

85 > 80  → True
90 > 80  → True
78 > 80  → False
92 > 80  → True
65 > 80  → False

Only rows with True are returned.

Understand the Condition

This expression:

students["Score"] > 80

creates a Boolean Series containing True and False.

print(students["Score"] > 80)

0     True
1     True
2    False
3     True
4    False
Name: Score, dtype: bool

Then Pandas uses that Boolean result to select the matching rows.

students[condition]

This is the foundation of Pandas filtering.

Comparison Operators

You can use different comparison operators when filtering.

# Greater than
students[students["Score"] > 80]

# Less than
students[students["Score"] < 80]

# Greater than or equal
students[students["Score"] >= 80]

# Less than or equal
students[students["Score"] <= 80]

# Equal
students[students["Score"] == 80]

# Not equal
students[students["Score"] != 80]

Be careful with ==.

score = 80       # assign 80

score == 80      # ask: is score 80?

Filter Text Values

Filtering is not limited to numbers. We can also filter text columns.

Suppose we want students from Hyderabad.

hyderabad_students = students[
    students["City"] == "Hyderabad"
]

print(hyderabad_students)

    Name  Age  Score       City
0    Raj   29     85  Hyderabad
3  David   30     92  Hyderabad

Pandas keeps only rows where the City value is exactly "Hyderabad".

Filter With Multiple Conditions

Real-world questions often contain more than one condition.

For example:

Show students who scored more than 80 AND are from Hyderabad.

result = students[
    (students["Score"] > 80) &
    (students["City"] == "Hyderabad")
]

print(result)

    Name  Age  Score       City
0    Raj   29     85  Hyderabad
3  David   30     92  Hyderabad

The & means AND.

AND vs OR

Pandas provides operators for combining conditions.

# AND
condition1 & condition2

# OR
condition1 | condition2

AND

Both conditions must be true.

students[
    (students["Score"] > 80) &
    (students["Age"] > 28)
]

OR

At least one condition must be true.

students[
    (students["City"] == "Hyderabad") |
    (students["City"] == "London")
]

Why Parentheses Are Important

When combining conditions in Pandas, put each condition inside parentheses.

students[
    (students["Score"] > 80) &
    (students["Age"] > 28)
]

Do not write it like this:

students[
    students["Score"] > 80 &
    students["Age"] > 28
]

That can produce an error because of Python's operator precedence.

The safe habit is simple: put parentheses around every condition.

Filter a Range of Values

Suppose we want students whose score is between 80 and 90.

result = students[
    (students["Score"] >= 80) &
    (students["Score"] <= 90)
]

print(result)

   Name  Age  Score       City
0   Raj   29     85  Hyderabad
1  John   32     90     London

Both conditions must be true:

Score >= 80
AND
Score <= 90

Filter Multiple Values With isin()

Suppose we want students from either Hyderabad or London.

We could use |, but Pandas also provides isin().

result = students[
    students["City"].isin(
        ["Hyderabad", "London"]
    )
]

print(result)

    Name  Age  Score       City
0    Raj   29     85  Hyderabad
1   John   32     90     London
3  David   30     92  Hyderabad
4   Emma   25     65     London

isin() asks: "Does this value exist in the list?"

Filtering Missing Values

You can also filter rows based on whether a value is missing or not missing.

For example, to find rows where Score is not missing:

result = students[
    students["Score"].notna()
]

To find rows where Score is missing:

result = students[
    students["Score"].isna()
]

Missing data will be covered in more detail in the upcoming Missing Data lesson.

Filter Rows and Select Columns

Sometimes you don't need every column from the matching rows.

For example, find students with a score above 80 but show only their names and scores.

result = students[
    students["Score"] > 80
][["Name", "Score"]]

print(result)

    Name  Score
0    Raj     85
1   John     90
3  David     92

First we filter the rows, then we select the columns.

Real-World AI Example

Imagine you have a dataset containing customer information:

Customer    Age    Purchases    Country
Raj         29       12         India
John        35       4          UK
Sarah       31       15         USA
David       42       20         India

You might want to find customers who:

Purchases > 10
AND
Country == "India"

The Pandas code would be:

customers[
    (customers["Purchases"] > 10) &
    (customers["Country"] == "India")
]

This is exactly the type of filtering you will perform before analyzing data or preparing it for an AI model.

Simple Mental Model

DataFrame
    ↓
Check every row
    ↓
Does it satisfy the condition?
    │
    ├── Yes → Keep row
    │
    └── No  → Remove row
    ↓
Filtered DataFrame

Score > 80

85 → Keep
90 → Keep
78 → Remove
92 → Keep
65 → Remove