Validation Sets and Cross-Validation: Choosing Honestly
A model needs data for learning and data for decisions
If you train a model and evaluate it on the same rows, the score is too close to the learning task. The model has already seen the answers. To choose between models or hyperparameters, reserve data that does not participate in fitting.
Key Points
- Training data fits model parameters.
- Validation data supports choices such as model family, degree, or regularisation strength.
- Test data is held back for a final estimate after choices are finished.
- The split must reflect how new data will arrive.
Tip:Validation is not a ceremonial extra split. It is the evidence used to make choices without spending the final test set.
Three sets, three jobs
A common workflow uses train, validation, and test sets. The model learns weights on train. You compare candidate configurations on validation. After locking the approach, you use test once for a final report.
Key Points
- Train: learn parameters such as regression weights.
- Validation: choose hyperparameters and compare candidate pipelines.
- Test: estimate final performance on an untouched sample.
- If data is scarce, cross-validation can use the training portion more efficiently while the test set remains untouched.
If you repeatedly change the model after looking at the test score, the test set is quietly becoming validation data. At that point, a new untouched test set is needed for a trustworthy final estimate.
A simple validation split
A validation split takes one part of the available development data and holds it out. For a time-ordered problem, the latest period may be the validation set. For randomly arriving independent rows, a shuffled split may be reasonable.
1rows = ["r1", "r2", "r3", "r4", "r5", "r6", "r7", "r8"]
2
3train = rows[:6]
4validation = rows[6:]
5
6print("Train:", train)
7print("Validation:", validation)
8print("Rows used for choices:", len(train) + len(validation))Train: ['r1', 'r2', 'r3', 'r4', 'r5', 'r6'] Validation: ['r7', 'r8'] Rows used for choices: 8
The exact proportion is a design choice. A tiny validation set can make the score noisy. A validation set that is too large leaves too little data for fitting. The important rule is that the validation rows are not used to fit each candidate model.
K-fold cross-validation rotates the validation role
In K-fold cross-validation, split the development data into K folds. Train on K minus 1 folds and validate on the remaining fold. Rotate the validation fold until every row has been used for validation once, then summarise the scores.
Key Points
- Each fold gives one train-validation split.
- Every development row is validation data exactly once in ordinary K-fold cross-validation.
- The mean score describes typical performance across folds.
- The spread shows how sensitive the model is to the particular fold.
- Cross-validation is a process for model choice, not permission to inspect the final test set repeatedly.
Calculate a small K-fold summary
1fold_mae = [8.0, 10.0, 9.0, 13.0, 10.0]
2
3mean_mae = sum(fold_mae) / len(fold_mae)
4lowest = min(fold_mae)
5highest = max(fold_mae)
6
7print(f"Fold MAE: {fold_mae}")
8print(f"Mean MAE: {mean_mae:.1f}")
9print(f"Range: {lowest:.1f} to {highest:.1f}")Fold MAE: [8.0, 10.0, 9.0, 13.0, 10.0] Mean MAE: 10.0 Range: 8.0 to 13.0
The mean is 10 minutes, but the range tells us that one fold is harder. Report both when stability matters. A candidate with mean 10 and a narrow range may be operationally preferable to one with mean 9 and a very unstable range.
The folds must match the data-generating process
Random folds are not automatically honest. If the same customer appears in several rows, rows from one customer can land in both train and validation. The model may recognise the customer rather than generalise to a new customer.
Key Points
- Grouped data may need group-aware splits.
- Time-ordered data often needs a forward-looking split rather than random shuffling.
- Repeated measurements from one person, device, or household need special care.
- The validation question should resemble the deployment question.
Random split
Trained on its own future
Test months sit in between training months, so the model saw March before being scored on February. In production nothing comes from the future.
Split by time
Trained only on the past
Everything before the cut trains, everything after is scored. It reads worse on paper and it is the only number worth reporting.
Cross-validation improves data use, but it cannot repair a fold design that lets future information travel into the past or lets the same entity appear on both sides.
Preprocessing belongs inside the validation loop
Scaling, imputation, feature selection, and polynomial expansion can learn information from data. If you fit them once on every row before cross-validation, the validation fold influences the transformation used to train the candidate model. That makes the score optimistic.
Key Points
- For each fold, fit preprocessing on that fold’s training rows.
- Apply the fitted preprocessing to the fold’s validation rows.
- Fit the model after preprocessing the training rows.
- Evaluate on transformed validation rows without refitting the transformer.
- Use a pipeline abstraction when the library provides one.
Do not tune forever on the same validation scores
Every comparison gives the validation set another chance to influence your choices. After many rounds of manual tuning, the selected model may fit quirks of that validation sample even though no parameter was directly trained on it.
Key Points
- Keep the final test set untouched during exploration.
- Record the search space and selection metric before comparing many candidates.
- Use cross-validation to reduce dependence on one validation split.
- If the project has many adaptive experiments, consider a new holdout or a nested evaluation design.
Choose a model from validation evidence
1candidates = [
2 {"name": "line", "cv_mae": 14.2, "cv_spread": 1.1},
3 {"name": "degree_2", "cv_mae": 10.8, "cv_spread": 1.5},
4 {"name": "degree_8", "cv_mae": 9.9, "cv_spread": 8.4},
5]
6
7for candidate in candidates:
8 print(
9 f"{candidate['name']}: mean={candidate['cv_mae']}, "
10 f"spread={candidate['cv_spread']}"
11 )line: mean=14.2, spread=1.1 degree_2: mean=10.8, spread=1.5 degree_8: mean=9.9, spread=8.4
The degree-eight candidate has the lowest mean in this toy table but much higher variation. The degree-two model may be the more dependable choice if unstable errors are costly. Model selection is a decision, not a race to the smallest displayed mean.
Where people get confused
Key Points
- "Validation and test mean the same thing." Validation supports choices; test is reserved for the final estimate.
- "More folds always gives a better answer." More folds changes compute, dependence between training folds, and score variation. Choose a sensible design.
- "Random K-fold is safe for every dataset." Groups, time order, duplicates, and leakage can make random folds optimistic.
- "Scaling before cross-validation is harmless." If scaling statistics use validation rows, the score has seen information it should not have.
- "The lowest mean score is always the best choice." Stability, cost, interpretability, and error consequences also matter.
Quick check
What job does a validation set perform?
It provides data for comparing models and choosing hyperparameters without fitting those choices to the training rows or spending the final test set.
How does K-fold cross-validation use the development data?
It rotates the validation fold so each development row is validated once while the remaining folds are used for fitting that candidate.
Why must preprocessing be fitted inside each fold?
Preprocessing can learn means, scales, selected columns, or other information. Fitting it on validation rows lets those rows influence the model indirectly.
When might random folds be inappropriate?
When rows share an entity, follow time order, or contain repeated measurements. The split should represent the way future predictions will be requested.
Recap
Key Points
- Train data fits parameters, validation data supports choices, and test data estimates final performance.
- K-fold cross-validation rotates the validation role across development rows.
- Fold design and preprocessing must respect groups, time, and information flow.
- Report mean performance and variation instead of hiding instability behind one average.
- The next lesson treats model choices as hyperparameters and builds a disciplined tuning search.
Theia
Can you explain this concept?
Practise Validation Sets and Cross-Validation: Choosing Honestly with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.