Measuring a Regression Model: MAE, MSE, RMSE, R²
A prediction needs a measurement plan
A model can produce a number for every row and still be unhelpful. To judge it, compare each prediction with the actual target and summarise the residuals with a metric. The metric should match the decision: explaining typical error, protecting against large misses, or comparing with a simple baseline.
There is no single score that answers every question. MAE, MSE, RMSE, and R² look at the same prediction errors from different angles. A careful report names the metric, the dataset, and the units before interpreting the number.
Key Points
- Actual value: what happened.
- Prediction: what the model estimated.
- Residual: actual minus prediction for one row.
- Metric: a summary of many residuals.
- Evaluation set: the rows used to judge performance, ideally unseen during fitting.
Tip:Always say whether a score came from training, validation, or test data. The same number can mean very different things depending on how the data was used.
Start with residuals
Suppose the actual delivery times are 30, 45, 60, and 80 minutes. A model predicts 28, 50, 54, and 86 minutes. The residual is actual minus prediction: 2, −5, 6, and −6 minutes.
Key Points
- A positive residual means the prediction was too low.
- A negative residual means the prediction was too high.
- The absolute residual describes the size of the miss without its direction.
- The sign pattern can reveal systematic underprediction or overprediction.
1actual = [30, 45, 60, 80]
2predicted = [28, 50, 54, 86]
3
4residuals = [real - guess for real, guess in zip(actual, predicted)]
5absolute_errors = [abs(error) for error in residuals]
6
7print("Residuals:", residuals)
8print("Absolute errors:", absolute_errors)Residuals: [2, -5, 6, -6] Absolute errors: [2, 5, 6, 6]
MAE answers the typical-distance question
Mean absolute error, or MAE, takes the absolute value of every residual and averages those distances. The formula is MAE = (1 ÷ n) × Σ|y − ŷ|. It reports the average miss in the original target units.
For the delivery example, the absolute errors are 2, 5, 6, and 6 minutes. Their total is 19, so MAE is 19 ÷ 4 = 4.75 minutes. You can explain that as: the model misses by about 4.75 minutes on average under this summary.
Key Points
- MAE treats each absolute miss in a direct, easy-to-explain way.
- MAE is less dominated by one extreme error than squared metrics.
- MAE has the same units as the target.
- MAE does not show whether the model tends to be too high or too low. Inspect signed residuals separately.
1actual = [30, 45, 60, 80]
2predicted = [28, 50, 54, 86]
3
4mae = sum(abs(real - guess) for real, guess in zip(actual, predicted)) / len(actual)
5print(f"MAE: {mae:.2f} minutes")MAE: 4.75 minutes
MSE makes large misses count more
Mean squared error, or MSE, squares each residual before averaging. The formula is MSE = (1 ÷ n) × Σ(y − ŷ)². Squaring removes the sign and gives a much larger contribution to a large miss.
In the delivery example, the squared residuals are 4, 25, 36, and 36. Their total is 101, so MSE is 101 ÷ 4 = 25.25 minutes squared. The squared units make MSE useful for optimisation and comparison, but less natural to explain to someone asking how many minutes the model misses.
Key Points
- MSE strongly penalises large errors.
- MSE is smooth and useful as a training objective for many models.
- MSE is measured in squared target units.
- A single unusual error can move MSE much more than it moves MAE.
1actual = [30, 45, 60, 80]
2predicted = [28, 50, 54, 86]
3
4mse = sum((real - guess) ** 2 for real, guess in zip(actual, predicted)) / len(actual)
5print(f"MSE: {mse:.2f} minutes squared")MSE: 25.25 minutes squared
RMSE returns to the target units
Root mean squared error, or RMSE, takes the square root of MSE. The formula is RMSE = √MSE. For the delivery example, RMSE is √25.25, which is about 5.02 minutes.
RMSE keeps MSE’s sensitivity to large errors but returns to the original unit. That makes it easier to discuss than MSE while preserving the influence of unusually large misses.
Key Points
- MAE: average absolute distance from the target.
- MSE: average squared distance from the target.
- RMSE: square root of MSE, expressed in target units.
- RMSE is at least as large as MAE for the same set of errors, except when all absolute errors are equal.
1import math
2
3mse = 25.25
4rmse = math.sqrt(mse)
5print(f"RMSE: {rmse:.2f} minutes")RMSE: 5.02 minutes
R² asks how much better than a baseline
R², called R-squared or the coefficient of determination, compares the model with a simple baseline that predicts the average target for every row. The formula is R² = 1 − (model squared error ÷ baseline squared error).
If the model makes much less squared error than the mean baseline, R² is close to 1. If it matches the baseline, R² is 0. If it is worse than the baseline on the evaluation data, R² can be negative.
Key Points
- The baseline predicts the mean of the evaluation targets for every row.
- R² is a relative comparison, not a percentage of individual predictions that are correct.
- R² can be negative on new data.
- A high R² does not prove causation, fairness, or that every prediction is close.
- R² does not replace MAE or RMSE when stakeholders need error in target units.
1actual = [10, 12, 14, 16]
2predicted = [9, 13, 13, 15]
3mean_target = sum(actual) / len(actual)
4
5model_squared_error = sum((real - guess) ** 2 for real, guess in zip(actual, predicted))
6baseline_squared_error = sum((real - mean_target) ** 2 for real in actual)
7r_squared = 1 - model_squared_error / baseline_squared_error
8
9print(f"Mean baseline: {mean_target:.1f}")
10print(f"R²: {r_squared:.2f}")Mean baseline: 13.0 R²: 0.80
One bad miss shows the metric tradeoff
Consider two models for four delivery orders. Model A misses by 2 minutes on every order. Model B misses by 0, 0, 0, and 8 minutes. Both have a total absolute error of 8 minutes, so they have the same MAE: 2 minutes.
MSE sees a difference. Model A has squared errors 4, 4, 4, and 4, for MSE 4. Model B has 0, 0, 0, and 64, for MSE 16. If an 8-minute miss is especially harmful, the squared metric makes that risk visible.
Key Points
- MAE can make consistent moderate errors look similar to one isolated large error when the absolute totals match.
- MSE and RMSE emphasise the isolated large error.
- Metric choice is a statement about which mistakes deserve more attention.
- When possible, report more than one view instead of hiding a serious tail behind one average.
Metrics should match the decision
For a delivery ETA shown to a customer, MAE can be easy to explain: “the typical miss is about five minutes”. For a hospital capacity forecast where a very large miss creates serious planning problems, RMSE may deserve attention. For model comparison against a mean baseline, R² can add context.
Key Points
- Use MAE when a typical absolute miss is the clearest story and extreme errors should not dominate as strongly.
- Use MSE when large errors should receive extra weight, especially during training.
- Use RMSE when large-error sensitivity matters but the result should use target units.
- Use R² when you need a relative comparison with a mean baseline.
- Add business thresholds, group checks, or residual analysis when one average cannot represent the decision.
The metric is part of the model design. If the team rewards a model for improving one score while ignoring another important failure, the model may optimise the wrong behaviour.
Training score is not the final report
Calculate metrics on data the model did not use to fit its parameters when you want to estimate performance on new examples. A training score tells you how well the model fits the rows it already saw. A test score gives evidence about a separate sample.
Key Points
- Compare training and test metrics to look for a large generalisation gap.
- Keep the test set for a final check rather than using it repeatedly to choose every detail.
- Use the same preprocessing and feature meanings for both datasets.
- Check whether the evaluation rows represent the situations where the model will actually be used.
- A metric cannot detect a target definition that is wrong for the business question.
Tip:A clean test score is not proof that a model is safe. It is evidence from one evaluation design. Look at errors by time period, geography, price range, or other groups that matter to the decision.
Residuals can tell a story that one score hides
Two models can have the same MAE and still fail in different ways. One may underpredict expensive flats. Another may work well on weekdays but miss weekend delivery times. Grouping residuals and plotting predictions against actual values can reveal these patterns.
Key Points
- Look for residuals that grow as the target grows.
- Look for a consistent positive or negative bias in a subgroup.
- Look for unusual points that dominate MSE or RMSE.
- Check whether errors are concentrated in a time period or range that matters operationally.
- Use the metric to start an investigation, not to end one.
Where people get confused
Key Points
- "An MSE of 25 means the model is off by 25 target units." No. MSE is in squared units. RMSE returns to the target units.
- "R² = 0.8 means 80% of predictions are correct." No. It describes relative squared-error reduction against a mean baseline.
- "The highest R² is always best." Not if another model has unacceptable large misses, group errors, or deployment costs.
- "MAE and RMSE should have the same value." They are equal only in special error patterns. RMSE gives more weight to large misses.
- "A good test score proves the model is fair and causal." Metrics measure prediction error under a chosen evaluation design, not every quality of a system.
Quick check
What does MAE tell you in the target units?
MAE is the average absolute distance between the actual target and the prediction. For delivery time, an MAE of 4.75 means the average absolute miss is 4.75 minutes on that evaluation set.
Why can RMSE be larger than MAE?
RMSE squares residuals before averaging and then takes a square root, so unusually large errors receive extra influence. It is at least as large as MAE for the same errors.
What does an R² of zero mean relative to its baseline?
It means the model has the same total squared error as the baseline that predicts the mean target for every evaluation row. It does not mean every prediction is exactly average.
Why should metrics be calculated on held-out data?
A model can fit its training rows without learning a pattern that generalises. Held-out data provides a separate check of how its predictions behave on rows it did not use to learn parameters.
Chapter 4 recap
Key Points
- Linear regression predicts a numerical target with a weighted line.
- Cost and MSE turn prediction gaps into a score that can guide learning.
- Gradient Descent changes parameters by moving in a cost-reducing direction.
- Multiple Linear Regression combines several feature clues in one weighted sum.
- Polynomial Regression adds powers of an input to represent a controlled bend.
- MAE, MSE, RMSE, and R² answer different questions about prediction quality.
- A trustworthy evaluation names the data, metric, units, baseline, and important error patterns.
You can now frame a numerical prediction problem, build a first regression model, train it, and judge its errors. The next chapter turns to what happens when a model looks good on one set of examples but fails to generalise: overfitting, underfitting, bias, and variance.
Theia
Can you explain this concept?
Practise Measuring a Regression Model: MAE, MSE, RMSE, R² with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.