Polynomial Regression: When the Relationship Bends
A straight line has limits
Linear regression is a useful first model because it asks a simple question: does the target tend to rise or fall at a steady rate as an input changes? Some real relationships do look roughly straight over the range we care about. Others clearly bend.
Imagine the daily temperature in a city rising through the afternoon and then falling after sunset. A single straight line cannot capture both directions at once. A curved model can describe the rise, the turning point, and the fall more naturally.
Key Points
- A straight line has one constant slope.
- A curved relationship can have a slope that changes as x changes.
- The curve should be supported by the data and the problem, not chosen just because it looks more impressive.
- Polynomial regression is one way to add controlled curvature to a regression model.
Tip:Before reaching for a curve, plot the data or inspect grouped averages. A curved model cannot rescue a target that was measured incorrectly or a feature that is unavailable at prediction time.
Create powers of the input
A degree-two polynomial adds the square of the input. Its formula is ŷ = b + w₁x + w₂x². A degree-three version adds x³ as well. These extra columns let the model combine a straight input, a squared input, and possibly a cubed input.
The curve appears when we look at predictions against the original x. Inside the model, the learned parameters still multiply columns and add together. That is why polynomial regression is linear in its coefficients even though its graph can bend.
Key Points
- Degree 1: x, which gives the ordinary straight-line model.
- Degree 2: x and x², which can create one broad bend.
- Degree 3: x, x², and x³, which can create a more flexible bend.
- The degree controls how many powers are available, not how good the model must be.
1def polynomial_features(x, degree):
2 return [x ** power for power in range(1, degree + 1)]
3
4for degree in [1, 2, 3]:
5 print(f"Degree {degree}: {polynomial_features(3, degree)}")Degree 1: [3] Degree 2: [3, 9] Degree 3: [3, 9, 27]
Work through one curved prediction
Suppose the model is ŷ = 10 + 4x − 0.5x². The positive x term makes the prediction rise at first. The negative x² term eventually becomes stronger and bends the curve downward.
Key Points
- At x = 2: ŷ = 10 + 4(2) − 0.5(2²) = 16.
- At x = 4: ŷ = 10 + 4(4) − 0.5(4²) = 18.
- At x = 8: ŷ = 10 + 4(8) − 0.5(8²) = 10.
- The prediction rises from x = 2 to x = 4, then falls by x = 8.
1def predict_curve(x):
2 return 10 + 4 * x - 0.5 * (x ** 2)
3
4for x in [2, 4, 8]:
5 print(f"x={x}, prediction={predict_curve(x):.0f}")x=2, prediction=16 x=4, prediction=18 x=8, prediction=10
The coefficients are not isolated instructions such as “increase forever because 4 is positive”. The x and x² terms act together. Interpretation must consider the whole formula and the range of x values used in the problem.
A curve is still trained against a cost
Polynomial regression does not replace the training ideas from earlier lessons. The model creates transformed features, makes predictions, measures errors with a cost such as MSE, and adjusts the coefficients to reduce that cost.
For degree two, the optimiser learns an intercept, a weight for x, and a weight for x². The extra weight gives the model permission to curve. Gradient Descent can update all three parameters just as it updated several weights in Multiple Linear Regression.
Key Points
- Transforming the input creates new columns before training.
- The target remains one numerical column.
- The cost still compares predictions with actual targets.
- The training data can show a lower MSE simply because the model has more flexibility.
- Held-out data tells us whether the curve learned a useful pattern or memorised the sample.
Compare a line and a curve fairly
A curve with more terms can almost always fit the training points at least as closely as a simpler line. That fact alone is not evidence that it will predict better. Compare models on the same held-out examples, using the same target values and the same evaluation metric.
1actual = [4, 9, 16, 25]
2line_predictions = [5, 8, 17, 24]
3curve_predictions = [4, 10, 15, 25]
4
5def mse(actual, predicted):
6 errors = [real - guess for real, guess in zip(actual, predicted)]
7 return sum(error ** 2 for error in errors) / len(errors)
8
9print(f"Line MSE: {mse(actual, line_predictions):.2f}")
10print(f"Curve MSE: {mse(actual, curve_predictions):.2f}")Line MSE: 1.00 Curve MSE: 0.50
Here the curve has the lower MSE on the examples shown. A careful conclusion is “the curve performed better on this evaluation set under MSE”. A stronger claim about future data needs a separate test or validation result.
High degree means high flexibility
Increasing the degree adds more powers: x⁴, x⁵, and so on. The model gains more ways to bend between the observed points. That can reduce training error, but it also gives the model more ways to follow random noise.
A high-degree curve may wiggle sharply because small coefficient changes get amplified by large powers. It can look excellent on the examples used for fitting and behave strangely just outside them.
Key Points
- Underfitting: the model is too simple to capture a real bend.
- Useful fit: the model captures the broad pattern without chasing every point.
- Overfitting: the model follows noise and becomes unreliable on new examples.
- Choose degree using validation evidence and domain reasoning, not training MSE alone.
Tip:A curve that passes through every historical point can be a poor forecasting tool. The goal is useful predictions for new inputs, not a perfect drawing of old noise.
Polynomial features can grow quickly
With one input, degree five creates five power columns. With several inputs, polynomial feature generation can also create combinations such as x₁x₂. The number of columns can grow quickly, making training slower and interpretation harder.
Key Points
- More generated columns mean more parameters to estimate.
- Powers can become numerically large when the original input is large.
- Different generated columns can be strongly correlated.
- The resulting model can overfit even when the original dataset had only a few columns.
Scaling the original feature before creating powers can help keep numbers manageable. Always record the transformation steps so that new inputs receive exactly the same treatment as training inputs.
Extrapolation is where curves become risky
Interpolation asks for a prediction inside the range of inputs the model has seen. Extrapolation asks for a prediction outside that range. Polynomial curves can behave reasonably between known points and then turn sharply outside them.
If the temperature data covered 10°C to 35°C, a prediction at 22°C is closer to interpolation. A prediction at 70°C is extrapolation. The second request needs stronger domain checks because the learned curve has little evidence there.
Key Points
- Check the training range before accepting a prediction.
- Flag inputs far outside the observed range.
- Use domain knowledge to decide whether the relationship should continue in the same direction.
- Do not present a precise-looking polynomial output as certainty.
Interpret a curve with care
A straight-line coefficient has one simple unit interpretation. Polynomial coefficients are harder to explain separately because x, x², and possibly higher powers change together as x changes. The useful question is often how the complete prediction changes over a realistic range.
Key Points
- Show predicted values or a plot across a meaningful input range.
- Explain where the curve rises, turns, and falls.
- Report the observed range and the evaluation data.
- Avoid saying that the x² coefficient alone proves a real-world effect.
- Keep the transformation pipeline available so another person can reproduce the prediction.
Where people get confused
Key Points
- "Polynomial" does not mean the target must be a polynomial in reality. It is a chosen approximation over a range.
- A curved graph can still come from a model that is linear in its learned coefficients.
- A higher degree is not automatically better. It trades simplicity for flexibility.
- A lower training error does not prove better future predictions.
- Extrapolation outside the observed range is usually more fragile than interpolation inside it.
- The coefficients of x and x² should not be interpreted as independent real-world effects without considering the full formula.
Quick check
How does degree-two polynomial regression create a curve?
It adds x² as a feature and learns a prediction of the form b + w₁x + w₂x². When predictions are plotted against the original x, the squared term allows the slope to change.
Why can a higher-degree polynomial have a lower training error but worse future predictions?
Extra powers give the model more flexibility. It may use that flexibility to follow random noise in the training examples instead of learning a pattern that continues on new data.
What is the difference between interpolation and extrapolation?
Interpolation predicts inside the input range represented by the training data. Extrapolation predicts outside that range, where the learned curve has less evidence and can behave unexpectedly.
Why is a polynomial coefficient difficult to interpret by itself?
The original input and its powers change together. The prediction depends on the complete combination of terms, so the effect is clearer when examined across realistic input values.
Recap
Key Points
- Polynomial regression adds powers of an input so a model can represent a bend.
- It remains linear in its learned coefficients even though its graph is curved.
- More degree adds flexibility, which can help underfit data and can also cause overfitting.
- Evaluate the curve on held-out data, inspect its realistic range, and be cautious with extrapolation.
- Interpret the complete prediction pattern rather than treating one polynomial coefficient as a standalone causal fact.
We now have several ways to make a numerical prediction: one line, several feature contributions, and a controlled curve. The final lesson in this chapter is about judging those predictions with regression metrics that answer different questions.
Theia
Can you explain this concept?
Practise Polynomial Regression: When the Relationship Bends with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.