Preparing a Real Dataset

Start With Raw Data

Imagine an e-commerce company gives us this customer dataset.

At first glance, the dataset looks simple. But there are several problems hidden inside it.

This is why we should not immediately send raw data into a machine learning model.

Step 1 — Find Problems in the Data

First, inspect the dataset and identify problems.

Typical things to check include:

Step 2 — Handle Missing Values

Our dataset contains a missing Income value.

Depending on the situation, we might remove the row, replace the value with a suitable statistic such as the median, or use another appropriate method.

The important point is that the missing value should not simply be ignored without considering how it affects the model.

Step 3 — Remove Duplicate Data

Our raw dataset contains the same customer record twice.

If these really represent the same record, keeping both can give that observation extra influence.

We should identify and handle duplicates before training the model.

Step 4 — Handle Incorrect Values and Outliers

Our dataset contains:

An age of 250 is almost certainly an incorrect value. We should investigate and correct or remove it.

Remember the distinction from the previous topic: an unusual value is not automatically wrong. We need to investigate before removing an outlier.

Step 5 — Encode Categorical Data

Our City column contains text:

These are categorical values. Depending on the situation, we can encode them so that the machine learning model can use them.

Step 6 — Scale Numerical Features

Suppose our final numerical features include:

These features have very different numerical ranges.

For algorithms that are sensitive to feature scale, we may apply an appropriate scaling method.

Step 7 — Split the Dataset

After deciding how to prepare the data, we need to keep separate data for evaluation.

The exact split depends on the problem and dataset. A common example is 80% training and 20% testing.

Most importantly, test information should not leak into the training process.

The Complete Preparation Process

Now we can see how the individual preparation steps fit together.

A Simple Python Example

In Python, a real data-preparation workflow might look conceptually like this:

import pandas as pd
from sklearn.model_selection import train_test_split

# Load the dataset
data = pd.read_csv("customers.csv")

# Remove duplicate rows
data = data.drop_duplicates()

# Separate features and label
X = data.drop("purchased", axis=1)
y = data["purchased"]

# Split the data
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

This is only a simplified example. Real datasets may require additional cleaning, encoding, missing-value handling, scaling, and other preprocessing steps.

What Should We Do With This Dataset?