Missing Values and Messy Data
Real data has holes in it
Last lesson, df.isna().sum() came back all zeros. That was the dataset being unusually kind to you, not what to expect. A tenant leaves the "furnished" field blank on a listing form. A field agent forgets to note the floor. A rent gets typed into the wrong box and lands in the wrong column entirely, leaving this one empty. Real data almost always has gaps, and scikit-learn refuses to train on a column that contains one: it throws an error and stops, every time.
So this is not optional cleanup. You cannot train a model until every missing value has been dealt with, one way or another. This lesson is about the real ways to deal with them, and which one to reach for.
Building a messier copy to practise on
To practise on something realistic, take the flats data and knock out three values on purpose: an area, a furnished status, and a rent.
1import pandas as pd
2
3df = pd.read_csv("https://cod-e.vercel.app/data/pune-flats.csv")
4messy = df.copy()
5messy.loc[2, "area"] = None
6messy.loc[9, "furnished"] = None
7messy.loc[15, "rent"] = None
8
9messy.isna().sum()area 1 bhk 0 size_sqft 0 floor 0 furnished 1 rent 1 dtype: int64
Tip:df.copy() matters here. Without it, messy would just be another name for the same table, and blanking out values in messy would quietly blank them out in df too.
Three missing values out of a possible 126 cells (21 rows times 6 columns), spread across three different columns. That is a realistic shape: missingness is rarely one tidy column, it is scattered wherever the data was recorded by hand.
The idea: figure out why, not just where
Before picking a fix, ask why a value might be missing. Sometimes it is pure accident, a sensor glitch, a form field that got skipped by mistake, nothing to do with the flat itself. Sometimes it is not an accident at all: on a real listings site, expensive flats are more likely to have "furnished" left blank because the owner did not bother filling every field for a flat they expect to rent fast. That second kind is more dangerous, because the missingness itself is a clue, and quietly filling it in throws that clue away.
You will not always be able to tell which kind you have. But asking the question first, instead of jumping straight to a fix, is what separates a considered decision from a reflex.
Option 1: drop the rows
1messy.dropna().shape(18, 6)
dropna() removes every row that has a missing value in any column. Three rows had a gap somewhere, so 18 remain out of 21. This is the simplest option, and it is often the right one, but only when you have enough data to spare and the missing rows are not concentrated in one group you care about. Here, dropping 3 of 21 rows, 14% of the whole dataset, over one blank cell each is a lot to throw away. In a dataset of 50,000 rows, dropping 3,000 scattered ones barely matters.
Option 2: drop the column
If a single column is missing in most of its rows, the column itself may not be worth keeping. Nobody would trust a column that is 90% empty to teach a model anything reliable. Check the fraction missing per column before deciding:
1(messy.isna().mean() * 100).round(1)area 4.8 bhk 0.0 size_sqft 0.0 floor 0.0 furnished 4.8 rent 4.8 dtype: float64
Each affected column here is under 5% missing, nowhere near the point of dropping the whole column. If furnished had been blank on 19 of 21 rows instead of 1, dropping the column with df.drop(columns=["furnished"]) would genuinely beat trying to guess 19 values out of thin air.
Option 3: fill it in
This is called imputing, filling in a reasonable guess instead of throwing the row or column away. What counts as "reasonable" depends on the column.
For a numeric column like rent, fill with the mean or the median of the values you do have:
1filled = messy.copy()
2filled["rent"] = filled["rent"].fillna(filled["rent"].mean())
3filled.loc[13:16, "rent"]13 30000.0 14 13500.0 15 26950.0 16 54000.0 Name: rent, dtype: float64
Row 15 was the one you blanked out. It now reads ₹26,950, the average of the other 20 rents, computed automatically because .mean() skips missing values on its own. That number is a guess, not a fact, which is exactly why the next lesson's scaling and the model after it should never be graded on how well they predict a row you invented the answer for.
Tip:Prefer the median over the mean when a column has extreme outliers, like a handful of flats that rent for ten times the rest. The median ignores how far away the extreme values are; the mean gets pulled toward them.
For a text column with a clear "most common answer", like furnished, fill with the mode, whichever value shows up most:
1filled["furnished"] = filled["furnished"].fillna(filled["furnished"].mode()[0])
2filled.loc[8:10, "furnished"]8 yes 9 yes 10 yes Name: furnished, dtype: object
"yes" appears 14 times among the 20 known values against 6 for "no", so row 9 gets filled with "yes". mode() can return more than one value if there is a tie, which is why [0] takes the first.
For area, there is no safe guess. Baner, Kothrud and Wakad are all tied for the most common area here, and guessing wrong tells the model a flat is somewhere it never was. When no guess is honest, say so instead of guessing: add an explicit category rather than picking one that already exists.
1filled["area"] = filled["area"].fillna("Unknown")
2filled.loc[1:3, "area"]1 Hinjewadi 2 Unknown 3 Baner Name: area, dtype: object
"Unknown" is now a real category the model can see and treat differently, rather than a silent guess dressed up as a fact.
Where people get confused
Key Points
- "Just fill everything with 0." 0 is a real value, not a stand-in for missing. Filling a missing rent with 0 tells the model that flat is free, which is worse than the gap it replaced.
- "dropna() is always safest." It is safest for accuracy on the rows that remain, but it throws away real data. On a small dataset, or when the missing rows share something in common, that is a real cost.
- "Filling a value makes it as good as a real one." It does not. It is a considered guess, useful for keeping a row usable, not a fact about that flat.
- "Check isna() once and you are done." Run it again after any join or reshape. New gaps show up in places you did not expect.
Quick check
Why does filling every missing numeric column with 0 usually cause more harm than good?
0 is a real, meaningful value in most columns, so filling with it tells the model something false, like a rent of ₹0, instead of honestly saying the value is unknown.
A column is missing in 92% of its rows. What should you seriously consider, and why?
Dropping the column entirely. With that little real signal, whatever you fill in is mostly a guess, and the column is unlikely to help the model more than it misleads it.
Why might mean and median give very different fill values for the same column?
The mean shifts toward extreme values (outliers), while the median does not. A column with a few very large or very small values will have a mean pulled away from where most of the data actually sits.
Recap
Key Points
- Missing values are normal in real data, and scikit-learn will not train on a column that has any.
- Before fixing a gap, ask whether it is random or whether the missingness itself means something.
- dropna() removes affected rows. Reasonable when the fraction missing is small and you have data to spare.
- Drop a whole column when it is missing in most of its rows, since there is little real signal left to fill in.
- fillna() imputes a guess: mean or median for numbers, mode for a clear-favourite category, an explicit placeholder like "Unknown" when no guess is honest.
The flats data is clean again, filled in, and ready. But area and furnished are still text, and a model only understands numbers. Turning categories into numbers, properly, is next.
Theia
Can you explain this concept?
Practise Missing Values and Messy Data with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.