Train/Test Split
A model that has seen every flat can just memorise
Chapter 4 trains a real model on this data. Once it is trained, the honest question is not "does it fit these 21 flats", it is "would it be right about a flat it has never seen." Train it on all 21 rows and ask it to predict rent for those same 21 rows, and a flexible enough model can simply memorise the answers, the way a student who has seen the exact exam paper beforehand looks brilliant without having understood a thing.
The fix is the last thing to set up before any model touches this data: hold some rows back, out of sight, and only ever use them to check the model afterwards.
The idea
Split the dataset into two pieces before training anything.
Key Points
- Training set: the rows the model is allowed to learn from. Usually 70 to 80% of the data.
- Test set: the rows the model never sees during training. Used only afterwards, to check how it does on flats it was never shown.
How well the model does on the training set tells you almost nothing on its own, since it had every chance to memorise those exact rows. How it does on the test set is the real answer to "does this model generalise", the same word from lesson 1.5's lifecycle: does it work on new cases, not just the ones it was trained on.
Why the obvious way to split is a trap
The simplest possible split is the first 80% of rows for training, the last 20% for testing. Try it on the flats data:
1import pandas as pd
2from sklearn.model_selection import train_test_split
3
4df = pd.read_csv("https://cod-e.vercel.app/data/pune-flats.csv")
5train_df, test_df = train_test_split(df, test_size=0.2, shuffle=False)
6train_df.shape, test_df.shape((16, 6), (5, 6))
shuffle=False keeps the original row order, so train_df is simply the first 16 rows and test_df is the last 5. Look at which areas end up in that test set:
1test_df["area"].tolist()['Viman Nagar', 'Hinjewadi', 'Kothrud', 'Kothrud', 'Wakad']
Baner and Hadapsar, two of the six areas in this dataset, do not appear in the test set at all. You would have zero way of telling whether the model's predictions for those areas are any good, because it is never checked on a single flat from either of them. This did not happen because the data was recorded badly; it happened purely because of the order the rows happened to be in. That is the trap: an unshuffled split can quietly hide whole categories from the one check that is supposed to catch problems like this.
The fix: shuffle before splitting
train_test_split shuffles the rows before splitting by default, which is exactly what you want almost every time. But a random shuffle picks different rows each time you run it, which makes it impossible to reproduce a result or compare two models fairly. random_state fixes that: it seeds the shuffle so it is random once, then exactly the same every time you rerun the cell.
1train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
2train_df.shape, test_df.shape((16, 6), (5, 6))
Same shapes, 16 training rows and 5 test rows, but now which rows land in which set is properly randomised, not just whatever order they happened to be recorded in. Run this cell again with random_state=42 and you get the exact same 16 and 5 rows every time, which is what lets you rerun an experiment tomorrow and trust that any difference came from a real change, not from a different random split.
Tip:random_state=42 is not special, it is a running joke among ML folks (from The Hitchhiker's Guide to the Galaxy), and you will see it everywhere. Any fixed number works the same job; what matters is that it is fixed, not which number it is.
Splitting features and target together
In practice you usually split X and y at the same time, so a flat's features and its true rent always stay matched to the same set:
1X = df[["bhk", "size_sqft", "floor"]]
2y = df["rent"]
3
4X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
5X_train.shape, X_test.shape, y_train.shape, y_test.shape((16, 3), (5, 3), (16,), (5,))
X_train and y_train go into training the model in Chapter 4. X_test and y_test stay untouched until after training, when you finally ask the model to predict rent for those 5 flats and compare its answers to the real y_test values it was never shown.
What not to do with the test set
Two mistakes undo everything this lesson is for.
Key Points
- Fitting anything, a scaler, an encoder, a model, on the full dataset before splitting. From lesson 3.5: fit a StandardScaler on all 21 rows and it already knows the mean and spread of the test rows, a small but real leak of test information into training.
- Looking at test performance while still tweaking the model, then tweaking again based on what you saw. Do that enough times and you have effectively trained on the test set by hand, just slower. Chapter 5's cross-validation exists specifically to let you tune a model honestly without burning the one true test set for that.
Treat the test set as a single, final exam you get to sit once, not a practice paper you can peek at while studying.
Where people get confused
Key Points
- "More test data is always better." A bigger test set gives a more reliable check, but every row in it is a row the model did not get to learn from. 70 to 80% training is a common balance, not a law.
- "random_state=42 makes the split representative." It only makes the shuffle reproducible. On a very small or very imbalanced dataset, even a shuffled split can occasionally come out lopsided, which is one reason real projects often use several splits at once, covered under cross-validation in Chapter 5.
- "I can scale or encode the whole dataset, then split it." Fit anything that learns from the data, a scaler, an encoder's categories, a model, only on the training set. Applying it to the test set afterwards is fine; fitting it on the test set, or on both together, is not.
Quick check
Why does a model scoring perfectly on its training data tell you almost nothing about how good it is?
A flexible enough model can simply memorise the training rows it was shown, the way a student memorises answers to a specific exam paper. That says nothing about how it handles a flat it has never seen.
What went wrong with the shuffle=False split on the flats data, specifically?
Because the split just took the last 20% of rows in their original order, the test set ended up with zero flats from two of the six areas (Baner and Hadapsar), so the model was never checked on either area at all.
Why is it wrong to fit a StandardScaler on the whole dataset before splitting into train and test?
The scaler would learn the mean and spread using the test rows too, which leaks information about the test set into a step that is supposed to happen before the model ever sees it.
Recap
Key Points
- Split data into a training set and a test set before training anything. Only the test set tells you the truth about generalisation.
- A naive, unshuffled split can hide entire categories from the test set purely because of row order. Shuffle before splitting.
- train_test_split(test_size=0.2, random_state=42) gives a random but reproducible split, the standard pattern.
- Fit any scaler, encoder or model on the training set only. Never let the test set influence anything the model learns from.
- Treat the test set as a check you run once, not a paper to keep peeking at while tuning.
Every idea in this chapter, encoding, scaling, engineering, splitting, exists for one reason: to hand a model data it can actually learn something true from. Chapter 4 is where that finally pays off. Time to train your first real model.
Theia
Can you explain this concept?
Practise Train/Test Split with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.