Pandas: DataFrames for Machine Learning

You get a CSV file of customer orders.

You open it. It's a disaster:

• Half the phone numbers are blank.
• Dates are strings like "15/03/2024", not numbers.
• The customer names live in a different file.
• Your model can only eat numbers (from topic 25: a NumPy array is one dtype, no strings).

So the question becomes

That is exactly what pandas is for.

A Series is

A DataFrame is

Carry this analogy through: a DataFrame is your exam-results spreadsheet. Each row is a student. Each column is a subject. The row index is the roll number. You can find any cell by saying "row Ada, column score" — that is exactly what .loc does.

The key difference from NumPy (topic 25): each column has its own dtype. So a DataFrame can hold numbers, strings, booleans, dates and missing values (NaN) side by side. That mixed-type flexibility plus the label-based index is what makes pandas ideal for real, messy data — whereas NumPy arrays stay strictly homogeneous.

import pandas as pd

df = pd.DataFrame({
    "name":   ["Ada", "Lin", "Raj"],
    "score":  [92.5, 78.0, 88.5],
    "passed": [True, False, True],
})

df.head()          # first rows
df.describe()      # numeric summary
df.info()          # dtypes + non-null counts

Here is the DataFrame from the code above, drawn as a grid:

code
Copy
index    name    score   passed
  0      Ada      92.5    True
  1      Lin      78.0    False
  2      Raj      88.5    True

• The leftmost column is the index — a label for each row (here 0, 1, 2 — but it could be names or dates).
• Each other column is a Series with its own dtype (object, float64, bool).
• Together they form the DataFrame.

Working through a tiny example: "select only the students who scored above 85."

The mask picks rows 0 and 2 (Ada, Raj) and drops row 1 (Lin, 78.0). That is all filtering is.

pandas offers two main accessors:

• .loc — label-based: "give me the row whose label is Ada."
• .iloc — position-based: "give me the first row."

Plus boolean masks for filtering by condition. Getting these straight avoids the classic chained-indexing pitfalls that trigger SettingWithCopyWarning.

df["score"]                        # one column (Series)
df[["name", "score"]]              # several columns (DataFrame)
df.loc[df["score"] > 85]           # rows where score > 85
df.iloc[0:2, 0:2]                  # first 2 rows, first 2 columns
df.loc[df["passed"], "score"] = 100  # safe in-place assignment

The split-apply-combine pattern underpins groupby.

Think of it like a teacher grading by class section:

1. Split — separate the rows into groups by a key (e.g., "region").
2. Apply — compute an aggregation on each group (e.g., mean revenue).
3. Combine — stack the per-group results back into one table.

Here is a tiny example:

code
Copy
orders:                     groupby("region").mean():
region   revenue             region   mean_revenue
North       200              North        150
North       100              South         80
South        80

North's mean = (200 + 100) / 2 = 150. South has one row, so mean = 80.

merge joins two DataFrames on shared keys, mirroring SQL JOIN. If you have an orders table and a users table, merge glues them on user_id — just like a VLOOKUP in a spreadsheet.

These two operations cover the bulk of feature preparation on tabular data — computing per-customer averages, rolling windows, or joining a lookup table.

df.groupby("region")["revenue"].agg(["mean", "sum", "count"])

users = pd.read_csv("users.csv")
orders = pd.read_csv("orders.csv")
joined = orders.merge(users, on="user_id", how="left")  # LEFT JOIN

Real data is full of NaN. pandas treats missingness explicitly: isna counts gaps, fillna imputes, and dropna removes rows or columns.

astype and the constructors to_datetime / to_numeric fix the string-vs-number problems that arise straight out of a CSV — e.g., turning "15/03/2024" into a real date so you can extract month or day-of-week.

This connects directly to the Data Preprocessing topic later in this phase.