Regularisation: Ridge and Lasso
A flexible model needs a reason to stay simple
Overfitting happens when a model uses its flexibility to chase noise. Regularisation adds a cost for large or unnecessary coefficients, so the training objective rewards fit and simplicity together.
The model is still allowed to use features. It simply has to earn that complexity by improving the data-fit part of the objective enough to justify the penalty.
Key Points
- Data loss measures how far predictions are from actual targets.
- A regularisation penalty measures coefficient size or complexity.
- The training objective combines both parts.
- The penalty strength is a hyperparameter selected using validation evidence.
Tip:Regularisation is not a substitute for removing target leakage or fixing a bad split. It controls a model after the data pipeline is made honest.
Ridge adds squared coefficient cost
Ridge Regression adds the sum of squared weights to the objective. A simplified form is data loss + λ × Σwⱼ². The symbol λ, often called alpha in libraries, controls how strongly large weights are discouraged.
Because a large weight is squared, Ridge steadily pushes coefficients toward zero. It usually keeps all features in the model, but makes their contributions smaller and often more stable when features are correlated.
Key Points
- λ = 0 gives the unregularised objective.
- Larger λ puts more pressure on small coefficient magnitudes.
- The intercept is commonly left unpenalised.
- Ridge usually shrinks coefficients close to zero without making many exactly zero.
L1
Switches weights off
Three features are gone entirely. The pull is the same strength however small a weight already is, so it can reach zero and hold there.
L2
Turns weights down
Every feature survives, all of them smaller. The pull fades as a weight shrinks, so nothing quite arrives at zero.
Lasso adds absolute coefficient cost
Lasso Regression adds the sum of absolute coefficient values. Its simplified objective is data loss + λ × Σ|wⱼ|. The absolute-value penalty can push some coefficients exactly to zero.
That makes Lasso useful when a sparse set of features is desirable. A zero coefficient can act as a form of feature selection, but correlated features make the choice less stable: Lasso may select one clue and drop another similar clue.
Key Points
- Ridge uses squared weights.
- Lasso uses absolute weights.
- Both can reduce overfitting by discouraging unnecessary coefficient size.
- Lasso can produce exact zeros, but a zero coefficient is not proof that a feature has no real-world importance.
See how the penalty changes
1weights = [3.0, -2.0, 0.5]
2
3ridge_penalty = sum(weight ** 2 for weight in weights)
4lasso_penalty = sum(abs(weight) for weight in weights)
5
6for strength in [0, 0.1, 1.0]:
7 ridge_cost = strength * ridge_penalty
8 lasso_cost = strength * lasso_penalty
9 print(f"lambda={strength}: ridge={ridge_cost:.2f}, lasso={lasso_cost:.2f}")lambda=0: ridge=0.00, lasso=0.00 lambda=0.1: ridge=1.33, lasso=0.55 lambda=1.0: ridge=13.25, lasso=5.50
This cell only calculates the penalty part. A real training objective adds it to the prediction loss. The same λ has different numerical effects for Ridge and Lasso because their penalty shapes differ.
Scaling matters before regularisation
A penalty sees coefficient values, not the meaning of a feature. If one feature is measured in rupees and another in years, the coefficients can have very different magnitudes simply because of units. Penalising them equally before scaling can be misleading.
Key Points
- Scale numeric features before comparing coefficient penalties.
- Fit the scaler on training data only.
- Reuse the training mean and spread for validation, test, and future rows.
- Interpret coefficients with the transformation and units in mind.
Standardisation often makes a one-unit change mean one standard deviation of the original feature. That makes regularisation pressure more comparable across columns, though it does not remove correlation or leakage.
Regularisation changes the objective, not just the result
An unregularised model chooses coefficients to minimise prediction loss. Ridge or Lasso chooses coefficients that balance prediction loss against coefficient size. A regularised model may have slightly worse training MSE and better validation performance.
Key Points
- Do not compare only training loss when deciding whether regularisation helped.
- The reported evaluation metric can be unregularised MAE, MSE, or RMSE on held-out data.
- The penalty is part of the training objective, not necessarily the business metric.
- Choose λ using a validation process rather than the test set.
Compare the two tools
1results = [
2 {"model": "unregularised", "train_mse": 4.0, "validation_mse": 19.0, "non_zero_weights": 8},
3 {"model": "ridge", "train_mse": 6.0, "validation_mse": 11.0, "non_zero_weights": 8},
4 {"model": "lasso", "train_mse": 7.0, "validation_mse": 12.0, "non_zero_weights": 3},
5]
6
7for result in results:
8 print(
9 f"{result['model']}: validation MSE={result['validation_mse']}, "
10 f"non-zero weights={result['non_zero_weights']}"
11 )unregularised: validation MSE=19.0, non-zero weights=8 ridge: validation MSE=11.0, non-zero weights=8 lasso: validation MSE=12.0, non-zero weights=3
In this example Ridge has the lowest validation MSE, while Lasso gives a sparser model. The choice depends on whether predictive stability, a compact feature set, or another operational constraint matters more.
A zero coefficient is not a causal conclusion
Lasso can set a coefficient to zero because another correlated feature already carries similar information, because the penalty is strong, or because the feature adds little predictive value in this sample. That is different from proving the feature has no causal relationship with the target.
Key Points
- Coefficient selection is model- and sample-dependent.
- Correlated features can compete for representation.
- Repeat the evaluation or inspect domain evidence before making a feature policy decision.
- Keep important domain features visible even if a regularised model does not use them directly.
A safe regularisation workflow
Key Points
- 1. Split the data according to how future examples arrive.
- 2. Fit preprocessing on the training portion only.
- 3. Try a meaningful range of λ values for Ridge or Lasso.
- 4. Select λ with validation or cross-validation.
- 5. Compare held-out metrics and coefficient stability.
- 6. Refit the selected approach on the allowed training data and evaluate once on the untouched test set.
Regularisation is one part of a model-selection process. It works best when paired with an honest validation strategy and a clear explanation of what the coefficients are being used for.
Where people get confused
Key Points
- "Regularisation makes training error lower." It usually adds a penalty, so the regularised training objective can rise even when held-out prediction improves.
- "Ridge removes features." Ridge usually shrinks coefficients but keeps most features active.
- "Lasso always finds the true features." Lasso selects features useful under its objective and sample; correlated columns can make the choice unstable.
- "A larger lambda is always safer." Too much penalty can create underfitting.
- "Penalty size is comparable before scaling." Coefficient units can make that comparison unfair.
Quick check
What does regularisation add to a training objective?
It adds a penalty for coefficient size or complexity to the prediction loss, encouraging a model to fit the data without using unnecessarily large parameters.
What is the practical difference between Ridge and Lasso?
Ridge uses squared coefficient magnitudes and usually shrinks weights toward zero. Lasso uses absolute magnitudes and can set some weights exactly to zero.
Why should features usually be scaled before applying a coefficient penalty?
Coefficient magnitude depends on feature units. Scaling makes the penalty more comparable across columns instead of treating unit choice as evidence of complexity.
Why should lambda be selected with validation rather than the test set?
Using the test set repeatedly turns it into a tuning set and makes its final score optimistic. Validation data is for choices; the test set is for the final check.
Recap
Key Points
- Regularisation balances prediction fit with coefficient simplicity.
- Ridge uses a squared penalty and usually shrinks all coefficients.
- Lasso uses an absolute penalty and can create sparse coefficients.
- Scaling, correlated features, validation, and coefficient interpretation all matter.
- The next lesson explains how validation sets and cross-validation support honest model choices.
Theia
Can you explain this concept?
Practise Regularisation: Ridge and Lasso with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.