Hyperparameters and Tuning: Choosing the Model Settings
Parameters are learned; hyperparameters are chosen
A linear model learns coefficients from training data. A polynomial model’s degree, Ridge’s penalty strength, the number of neighbours in a nearest-neighbour model, and a tree’s maximum depth are settings chosen before or around training. These are hyperparameters.
Key Points
- Parameters are fitted by the learning algorithm.
- Hyperparameters control the model family, capacity, or training process.
- Hyperparameters should be selected with validation evidence.
- The final test set should not be used as a steering wheel.
Tip:The exact boundary can vary by library, but the practical question is stable: was this value learned from the fitting rows, or did a person or search procedure choose it?
Examples of hyperparameters
Key Points
- Polynomial degree controls how many powers are created.
- Ridge or Lasso lambda controls regularisation strength.
- A decision tree maximum depth controls potential complexity.
- A learning rate controls the size of Gradient Descent updates.
- The number of neighbours controls how local a nearest-neighbour prediction is.
Each setting changes the kind of model you are willing to fit. A tuning process should connect each candidate range to the data size, domain, compute budget, and cost of errors.
Grid search is a small planned experiment
A grid search tries a list of specified combinations. For one hyperparameter, the grid may be simple. For several hyperparameters, the combinations can multiply quickly, so keep the grid purposeful.
1degrees = [1, 2, 3]
2regularisation = [0.0, 0.1]
3
4combinations = [
5 (degree, strength)
6 for degree in degrees
7 for strength in regularisation
8]
9
10print("Candidates:", combinations)
11print("Number of candidates:", len(combinations))Candidates: [(1, 0.0), (1, 0.1), (2, 0.0), (2, 0.1), (3, 0.0), (3, 0.1)] Number of candidates: 6
A grid is easy to explain and reproduce. It can miss a good value between the chosen points, and a large multi-dimensional grid can waste compute on combinations that domain knowledge already makes unlikely.
Random search can cover wide ranges
Random search samples candidate combinations from chosen distributions or lists. It can explore a large space more efficiently when only a few hyperparameters strongly affect the score. The random seed should be recorded for reproducibility.
Key Points
- Grid search gives systematic coverage of the selected points.
- Random search can spend trials across a wider space.
- Both methods are only as sensible as their ranges and evaluation design.
- A search budget is a project constraint, not a reason to skip a baseline.
Tune against the right objective
The search needs a scoring rule. For a delivery estimate, MAE might reflect the typical customer-facing miss. For a safety-sensitive forecast, RMSE may make large misses more visible. The tuning metric should reflect the decision, not just the metric that is easiest to print.
Key Points
- Choose the primary metric before looking at the candidate results.
- Track secondary metrics and important group errors.
- Use the same evaluation procedure for every candidate.
- Do not select a model from a score that hides the failure you care about.
Select from validation results
1results = [
2 {"degree": 1, "alpha": 0.0, "cv_mae": 14.2},
3 {"degree": 2, "alpha": 0.0, "cv_mae": 10.8},
4 {"degree": 2, "alpha": 0.1, "cv_mae": 9.7},
5 {"degree": 3, "alpha": 0.1, "cv_mae": 10.1},
6]
7
8best = min(results, key=lambda result: result["cv_mae"])
9print("Selected degree:", best["degree"])
10print("Selected alpha:", best["alpha"])
11print("Validation MAE:", best["cv_mae"])Selected degree: 2 Selected alpha: 0.1 Validation MAE: 9.7
This selection is justified only relative to the candidates and scoring procedure shown. It is not evidence that degree two and alpha 0.1 are universal settings.
A tuning pipeline has several layers
Key Points
- Define the prediction question and the primary metric.
- Create a split or cross-validation design that matches deployment.
- Fit preprocessing inside each training fold.
- Search a documented hyperparameter space.
- Select using validation evidence and inspect tradeoffs.
- Refit the chosen pipeline using the allowed development data.
- Evaluate once on the untouched test set.
Saving the selected settings, data version, random seed, metric, and code turns tuning from a one-off lucky run into an experiment another person can reproduce.
Do not overreact to tiny score differences
If two candidates have validation MAE values of 9.70 and 9.72, the difference may be smaller than the uncertainty from another split or another month of data. A slightly simpler or more stable candidate may be the better operational choice.
Key Points
- Inspect fold-to-fold variation.
- Consider training time, memory, interpretability, and maintenance.
- Check group-level errors and important thresholds.
- Prefer a meaningful improvement over a cosmetic decimal change.
Keep the test set out of the search
If you try ten configurations, inspect the test scores, and keep the one with the best test score, the test set has become part of the tuning process. The final number is then optimistic because the choice was influenced by those rows.
Key Points
- Use train and validation or cross-validation during exploration.
- Lock the pipeline and settings before the final test evaluation.
- If the test result changes the model, record that the test was used for development and reserve a new holdout.
- Treat experiment logs as part of evaluation hygiene.
Where people get confused
Key Points
- "The algorithm learns hyperparameters automatically." A search procedure can choose them, but that is a separate layer from fitting model parameters.
- "More candidates guarantees a better model." It can increase search overfitting, compute cost, and opportunities to chase noise.
- "Grid search is always better than random search." The choice depends on the space, budget, and prior knowledge.
- "The best validation score is always the production choice." Stability, groups, cost, and interpretability may change the decision.
- "Testing many models on the test set is harmless." Repeated test inspection turns it into a tuning set.
Quick check
What is the difference between a parameter and a hyperparameter?
A parameter is learned by fitting the model to training data. A hyperparameter controls the model or training process and is selected outside that parameter-fitting step.
Why can a grid search become expensive?
The number of combinations multiplies as more hyperparameters and candidate values are added. Cross-validation then repeats the fitting work for each combination.
What should determine the primary tuning metric?
The prediction decision and the consequences of different errors should determine it. The metric should represent what the model needs to do, not merely what is convenient.
Why should the test set remain untouched during tuning?
Looking at test scores can influence the selected settings, making the test estimate optimistic. The test set is most useful as a final check after choices are locked.
Recap
Key Points
- Hyperparameters control model capacity, penalties, or training behaviour.
- Grid and random search are ways to compare documented candidate settings.
- Use a deployment-relevant metric and validation design.
- Record the search space, scores, seeds, and selected pipeline.
- The final lesson explains data leakage, the quiet failure that can make every score look better than reality.
Theia
Can you explain this concept?
Practise Hyperparameters and Tuning: Choosing the Model Settings with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.