Data Leakage: When the Answer Sneaks Into the Features
A model can cheat without anyone asking it to
Data leakage happens when information that would not be available at prediction time influences training or evaluation. The model appears accurate because a clue related to the answer has crossed the boundary into the input.
Leakage is dangerous because it often looks like success. A suspiciously excellent validation score can make a broken pipeline appear ready for launch.
Key Points
- Ask what was knowable at the exact moment the prediction would be made.
- Ask whether any feature was created using the target or a future event.
- Ask whether train and validation rows share an entity or time information.
- Treat a surprising score as a reason to audit the data path, not only celebrate.
Tip:The cleanest leakage question is: “Would this value exist, in this form, before the model had to make the prediction?”
Target leakage uses a future consequence
Suppose the target is whether a customer will miss a payment next month. A feature called collection_call_made may strongly predict the target, but the call might happen only after the customer has already shown trouble. It is a consequence, not a prediction-time clue.
Key Points
- Post-outcome status fields can reveal the answer.
- Resolution codes, final invoices, cancellation reasons, and later diagnoses need a timestamp check.
- A feature can be stored before training and still be invalid if it was created after the prediction event.
- Remove or redesign the feature rather than trusting a high score.
Train-test contamination leaks through preprocessing
A transformer can leak information even when the target is not directly present. If you calculate the global mean and standard deviation using train and validation rows together, the validation rows influence the representation used to fit the model.
The effect may be small for scaling, but it is still the wrong information flow. Imputation, feature selection, target encoding, and dimensionality reduction can create larger leaks.
Key Points
- Fit preprocessing on training rows only.
- Apply the fitted transformation to validation and test rows.
- Keep feature selection inside the training or cross-validation loop.
- Use a pipeline so fitting and transforming happen in the correct order.
1train = [10, 12, 14]
2validation = [100]
3
4global_mean = sum(train + validation) / len(train + validation)
5train_mean = sum(train) / len(train)
6
7print(f"Mean using all rows: {global_mean:.1f}")
8print(f"Mean using train only: {train_mean:.1f}")Mean using all rows: 34.0 Mean using train only: 12.0
The validation value of 100 changes the global mean dramatically. Even if the transformer never reads the target column, using that validation row to calculate training statistics lets information cross the evaluation boundary.
Time leakage looks like a very good forecast
For a time-based prediction, future rows must not influence past predictions. Randomly splitting a dataset can place next month’s information in training while evaluating a prediction for an earlier month.
Key Points
- Use timestamps to define what was available before each prediction.
- Prefer forward-looking splits for forecasting and many operational workflows.
- Watch for features updated after the outcome, even if their row timestamp looks old.
- Backfill processes and historical corrections can silently change what was knowable at the time.
Random split
Trained on its own future
Test months sit in between training months, so the model saw March before being scored on February. In production nothing comes from the future.
Split by time
Trained only on the past
Everything before the cut trains, everything after is scored. It reads worse on paper and it is the only number worth reporting.
Duplicate entities can cross the split
If the same patient, device, seller, or household appears in both train and validation, the model may benefit from recognising that entity’s history. The score can look strong while performance on a genuinely new entity is poor.
Key Points
- Decide whether deployment predicts future rows for known entities or first-time entities.
- Group related rows when the evaluation question is about new entities.
- Do not treat duplicate removal as the only solution. Preserve valid historical information for the scenario you actually serve.
- Document the unit of generalisation: row, customer, device, location, or time period.
Leakage can enter through feature engineering
A feature created from a whole dataset can carry future or target information even if its final column name looks harmless. Examples include target means calculated across all rows, counts that include future events, and “number of complaints” measured after the prediction date.
Key Points
- Define the feature timestamp and source events.
- Calculate aggregates using only records available at prediction time.
- For target encoding, calculate mappings within training folds.
- Keep feature generation code reproducible so it can be audited later.
A leakage audit follows the prediction boundary
1features = [
2 {"name": "signup_age_days", "available_before_prediction": True},
3 {"name": "support_calls_after_outcome", "available_before_prediction": False},
4 {"name": "train_scaled_income", "available_before_prediction": True},
5]
6
7for feature in features:
8 status = "KEEP" if feature["available_before_prediction"] else "REMOVE"
9 print(f"{status}: {feature['name']}")KEEP: signup_age_days REMOVE: support_calls_after_outcome KEEP: train_scaled_income
The third feature is safe only if “train_scaled_income” means the transformation was fitted on training data and then applied consistently. A name alone cannot prove a pipeline is clean.
A clean pipeline has one direction of information
Key Points
- 1. Define the prediction timestamp and target availability time.
- 2. Build each feature from events known before that boundary.
- 3. Split by the deployment scenario before fitting learned transformations.
- 4. Fit preprocessing and feature selection inside training folds.
- 5. Train the model and evaluate on untouched future or held-out rows.
- 6. Audit features again when the data refresh process changes.
Leakage prevention is a data-pipeline property. A strong algorithm cannot compensate for information arriving from the future.
A suspicious score deserves questions
A validation score that is nearly perfect can be real in a narrow task, but it should trigger an audit. Compare against a simple baseline, inspect the feature definitions, check timestamps, look for duplicates, and test a split that matches deployment.
Key Points
- Ask whether any feature is a post-outcome status.
- Rebuild preprocessing inside the split or pipeline.
- Compare random and time-aware or group-aware evaluation when relevant.
- Remove suspicious features and measure the effect.
- Document why every retained feature is available at prediction time.
Where people get confused
Key Points
- "Leakage means the target column is in X." That is one obvious form. Future events, shared entities, and preprocessing statistics can leak without a target-named column.
- "A timestamp column is always leakage." A timestamp can be valid if it is known before prediction and used consistently. The event timing matters.
- "Random splitting prevents leakage." It can create leakage when rows are related by entity or time.
- "Removing all correlated features fixes leakage." Correlation is not the definition. The key question is information availability and evaluation separation.
- "A pipeline library guarantees clean data." It helps with transformation order, but feature timestamps and entity boundaries still require domain reasoning.
Quick check
What is data leakage?
It is information flow from the target, the future, the evaluation rows, or related entities into training features or decisions when that information would not be available at prediction time.
Why can fitting a scaler before splitting be leakage?
The scaler learns statistics from validation or test rows. Those rows then influence the representation used to fit the model, making the evaluation less independent.
Why can random row splitting be wrong for a time-series problem?
It can place future observations in training while evaluating an earlier prediction. A time-aware split should preserve the direction of information.
What should you ask about every feature?
Ask what event created it, when it became available, whether it uses the target or future information, and whether it is calculated separately inside the training process.
Chapter 5 recap
Key Points
- Overfitting learns training details that do not generalise; underfitting misses useful structure.
- Bias describes systematic error, while variance describes sensitivity to the training sample.
- Ridge and Lasso regularisation control coefficient complexity in different ways.
- Validation and cross-validation support model choices without spending the final test set.
- Hyperparameter tuning must use a deployment-relevant metric and an honest search design.
- Data leakage makes evaluation look better by letting unavailable information cross the prediction boundary.
- A reliable model is built from an honest data path, not only a clever algorithm.
You now have a way to diagnose model failure, control complexity, select settings, and audit information flow. The next chapter changes the prediction target from a number to a category and introduces classification.
Theia
Can you explain this concept?
Practise Data Leakage: When the Answer Sneaks Into the Features with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.