Gradient Descent: Walking Downhill
The model knows what better means, but not how to get there
In the last lesson, MSE gave us a score for a line. A smaller MSE meant that the line made smaller squared mistakes on the examples. That solved one problem: we could tell which of two lines was better.
But a model still has to find its line. It has to choose values for w, the slope, and b, the intercept. Trying every possible pair would be slow. There are infinitely many possible values, and real models can have thousands or millions of parameters.
Gradient Descent is a repeated search strategy. It checks which direction would reduce the cost, takes a step in that direction, checks again, and keeps walking until the cost stops improving much.
Tip:Gradient Descent is not the definition of learning. It is one method for finding parameter values that make the chosen cost smaller.
Picture the cost as a valley
Imagine a smooth valley. Your position on the ground represents the current parameter values. The height represents the cost. At the top, the model is making large mistakes. Near the bottom, its cost is lower.
If you are standing on a slope, the ground tells you which way is uphill. The opposite direction is downhill. You take a small step downhill, look at the new slope, and repeat. You do not need to see the entire valley at once.
For a line, the valley is not a physical place. It represents possible combinations of w and b. Each combination has a cost calculated from the predictions. Gradient Descent moves through those combinations.
The lowest point you reach is a minimum of the cost. For the simple linear-regression problems in this chapter, the MSE surface is bowl-shaped, so a sensible downhill walk can approach its bottom. More complicated models can have more complicated surfaces.
See the downhill direction with real numbers
Take one teaching example: the input x is 2 and the actual target y is 6. Keep b at zero for a moment. A weight w of zero predicts zero, so its squared error is 36. We can try a few weights and see which direction makes that error smaller.
Key Points
- w = 0: prediction is 0 × 2 = 0. Squared error is (0 − 6)² = 36.
- w = 1: prediction is 1 × 2 = 2. Squared error is (2 − 6)² = 16.
- w = 2: prediction is 2 × 2 = 4. Squared error is (4 − 6)² = 4.
- w = 3: prediction is 3 × 2 = 6. Squared error is zero.
- w = 4: prediction is 4 × 2 = 8. Squared error rises again to 4.
Starting at w = 0, increasing w is the downhill direction. Starting at w = 4, decreasing it is downhill. There is no rule that weights should always increase or always decrease. The useful direction depends on where you are now.
Trying a handful of values works for this tiny example. A gradient gives us a local direction without trying every weight. Local means the slope right around the current parameter values, rather than a map of every possible value.
The gradient tells us which way is uphill
A derivative is the slope of a function. If the cost rises when a parameter increases, the derivative is positive in that direction. If the cost falls when the parameter increases, the derivative is negative.
For several parameters, we collect these slopes into a gradient. The gradient points toward the direction where the cost increases fastest. We want the opposite direction, so the update subtracts the gradient.
For the line ŷ = wx + b, we calculate a separate cost slope for w and for b. The weight slope uses each error multiplied by its input. The intercept slope uses the errors alone, because increasing b raises every prediction by the same amount.
Key Points
- Weight gradient: ∂MSE/∂w = (2 ÷ n) × Σ((ŷ − y) × x).
- Intercept gradient: ∂MSE/∂b = (2 ÷ n) × Σ(ŷ − y).
- x is the input feature, y is the actual target, and ŷ is the prediction.
- n is the number of training examples. Σ means add the result for every example.
- ∂MSE/∂w means the slope of MSE when w changes while b stays fixed. ∂MSE/∂b means the slope when b changes while w stays fixed.
- The factor 2 comes from the slope of the squared-error calculation. We use this result without deriving it here.
You do not need to memorise these formulas yet. Follow what they do: use the current prediction errors to calculate a direction for each parameter. The code below expresses the same steps with averages.
Tip:In Lesson 4.2 we used actual minus predicted to describe an error. Here the formula uses predicted minus actual. Both conventions give the same squared error, but the gradient formula and update must use a consistent sign. In the code, errors always means predictions minus targets.
The sign matters. A gradient is not a score saying whether the model is good or bad. It is a direction signal for the next update.
The learning rate controls the step size
The learning rate is a positive number that multiplies the gradient in each update. A smaller rate takes smaller steps for the same gradient. A larger rate can move faster, but it can also jump past the useful region.
The basic update has this shape: new parameter = old parameter − learning rate × gradient The minus sign means move opposite to the uphill direction. The learning rate decides how much of that direction to use.
Key Points
- Learning rate too small: cost may decrease, but training can take many updates.
- Learning rate in a useful range: cost decreases steadily and the parameters approach a low-cost region.
- Learning rate too large: the model can jump across the valley, oscillate, or make the cost grow instead of shrink.
1def walk_towards_three(learning_rate, steps=10):
2 w = 0.0
3 for _ in range(steps):
4 # Cost for this tiny example is (w - 3)^2.
5 gradient = 2 * (w - 3)
6 w = w - learning_rate * gradient
7
8 return w, (w - 3) ** 2
9
10for rate in [0.01, 0.1, 1.1]:
11 w, cost = walk_towards_three(rate)
12 print(f"Learning rate {rate}: w={w:.3f}, cost={cost:.3f}")Learning rate 0.01: w=0.549, cost=6.008 Learning rate 0.1: w=2.678, cost=0.104 Learning rate 1.1: w=-15.575, cost=345.038
This toy cost has its minimum at w = 3. After ten steps, the rate 0.01 is still moving slowly. The rate 0.1 has moved close to the minimum. The rate 1.1 is too large for this curve and has jumped away from the answer.
One update by hand
Return to x = 2 and y = 6, but now allow both parameters to change. Start with w = 0 and b = 0. The prediction is ŷ = wx + b = 0. The error is 0 − 6 = −6. The model is predicting too low.
Key Points
- Prediction: ŷ = 0 × 2 + 0 = 0.
- Error: ŷ − y = 0 − 6 = −6.
- Gradient for w: 2 × (−6) × 2 = −24.
- Gradient for b: 2 × (−6) = −12.
- With learning rate 0.1, new w = 0 − 0.1 × (−24) = 2.4.
- With learning rate 0.1, new b = 0 − 0.1 × (−12) = 1.2.
Both gradients are negative, so subtracting them increases w and b. That makes sense: the first prediction was too low. The next prediction is 2.4 × 2 + 1.2 = 6.0, which is exactly right for this one example.
Getting it right in one step is a coincidence of these numbers and this learning rate, not a promise about training. One point also cannot uniquely determine a line: many pairs of w and b would predict 6 when x is 2. We need more examples to constrain the relationship.
Real training uses many examples together. One example can pull the line in one direction while another pulls it in a different direction. The average gradient gives the update a direction that considers the full training set.
Train a line with repeated updates
Use a small teaching dataset where the target follows y = 2x + 1. Inputs 1, 2, and 3 have targets 3, 5, and 7. These small numbers let you inspect the arithmetic. Start with w = 0 and b = 0: every prediction is zero, so the MSE is large.
For every update, the code calculates predictions, errors, the two gradients, and a new w and b. The cost is measured before the update so we can watch whether the walk is going in the right direction.
1x = [1.0, 2.0, 3.0]
2y = [3.0, 5.0, 7.0]
3n = len(x)
4
5w = 0.0
6b = 0.0
7learning_rate = 0.1
8
9for epoch in range(101):
10 predictions = [w * value + b for value in x]
11 errors = [prediction - target for prediction, target in zip(predictions, y)]
12 mse = sum(error ** 2 for error in errors) / n
13
14 if epoch in [0, 1, 2, 10, 50, 100]:
15 print(f"Epoch {epoch}: w={w:.3f}, b={b:.3f}, MSE={mse:.4f}")
16
17 if epoch == 100:
18 break
19
20 # Calculate both gradients from the SAME current predictions.
21 dw = 2 * sum(error * value for error, value in zip(errors, x)) / n
22 db = 2 * sum(errors) / n
23 w = w - learning_rate * dw
24 b = b - learning_rate * dbEpoch 0: w=0.000, b=0.000, MSE=27.6667 Epoch 1: w=2.267, b=1.000, MSE=0.3319 Epoch 2: w=2.018, b=0.893, MSE=0.0053 Epoch 10: w=2.035, b=0.921, MSE=0.0009 Epoch 50: w=2.013, b=0.970, MSE=0.0001 Epoch 100: w=2.004, b=0.991, MSE=0.0000
The values do not become exactly 2 and 1 after one update. They get closer over many updates. That is normal. The cost is the useful signal: it falls from 27.6667 to almost zero, and the learned line approaches y = 2x + 1.
This code uses plain Python so each operation is visible. The expression inside square brackets calculates a value for each example. zip pairs a prediction with its matching target, or an error with its matching input. len gives the example count; sum adds the values.
dw and db are names for the cost slopes of w and b. We calculate both before changing either parameter. If we changed w and then recalculated the gradient for b, the two updates would no longer describe the same starting line.
Tip:Epoch 0 is the initial state, before any updates. Epoch 100 shows the result after 100 updates, then the loop stops. MSE rounds to 0.0000 in that last printout; its unrounded value is still a small positive number.
An epoch means one complete pass through the training examples. An iteration means one parameter-update step. Here, each update uses all three examples, so one epoch and one update correspond. This is called batch, or full-batch, Gradient Descent.
Some training methods update after a small group of examples or after each example. Then several updates can happen in one epoch. We will study those variants later. For now, keep the two ideas separate: an epoch counts data passes, while an iteration counts updates.
Bring the idea back to the flat-rent model
In the Pune example, w describes how much rent changes with size, and b shifts every predicted rent. The same loop could adjust those parameters. But flat sizes measured in hundreds of square feet produce much larger numbers than the inputs 1, 2, and 3 above.
The weight gradient includes the input x. Changing the scale of x changes the size of that gradient. So 0.1 is not a universal learning rate that you can copy into every rent dataset. Feature scaling, introduced in the Data chapter, can help keep updates manageable.
There is another detail from Lesson 4.1: scikit-learn LinearRegression does not normally run this Gradient Descent loop for ordinary unconstrained dense data. It solves a least-squares problem using numerical linear algebra. Both approaches seek a line with low squared error, but they find it differently.
Tip:You do not need another scikit-learn estimator to understand this lesson. The manual loop makes the search visible. Later you can choose a solver or training method based on the size and structure of the problem.
What if the learning rate is wrong?
A training run should not be judged by its final parameters alone. Watch the cost over time. A healthy run usually moves toward a lower cost. If the cost barely changes, the steps may be too small. If it jumps up and down or becomes enormous, the steps may be too large.
1def costs_for_rate(learning_rate, steps=8):
2 w = 0.0
3 costs = []
4
5 for _ in range(steps):
6 cost = (w - 3) ** 2
7 costs.append(round(cost, 3))
8 gradient = 2 * (w - 3)
9 w = w - learning_rate * gradient
10
11 return costs
12
13for rate in [0.01, 0.1, 1.1]:
14 print(rate, costs_for_rate(rate))0.01 [9.0, 8.644, 8.301, 7.973, 7.657, 7.354, 7.062, 6.783] 0.1 [9.0, 5.76, 3.686, 2.359, 1.51, 0.966, 0.618, 0.396] 1.1 [9.0, 12.96, 18.662, 26.874, 38.698, 55.726, 80.245, 115.553]
The rate 0.01 is learning, but slowly. The rate 0.1 reduces the cost at every step. The rate 1.1 makes the cost larger because each step overshoots the valley. This is why the learning rate is a hyperparameter: it is chosen before training and is not learned from the same update.
Epochs, convergence, and stopping
You can stop after a fixed number of epochs, such as 100. You can also choose a small improvement threshold, called a tolerance, and stop if the cost stays within it for several updates. Convergence means the process is settling toward a stable result; it does not necessarily mean the exact minimum has been reached.
Key Points
- An epoch is one complete pass through the training data.
- A training curve is the cost recorded across epochs.
- A falling curve usually means the current training process is reducing the chosen cost.
- A flat curve can mean the model has converged, the learning rate is too small, or the implementation has a problem.
- A rising or unstable curve can mean the learning rate is too large or the data needs attention.
A low training cost is not the whole story. Later lessons will show why a model can fit its training examples too closely. For this lesson, focus on the optimisation question: are the parameter updates reducing the cost we asked them to reduce?
For ordinary linear regression with MSE, a stationary point at the bottom is a global minimum: no other parameter values have lower cost. Some datasets allow several equally good solutions. More complicated models can have local minima and flat regions, so do not carry this simple bowl picture into every model without checking.
Where people get confused
Key Points
- "Gradient Descent always moves straight to the best answer." No. Its path depends on the starting values, the learning rate, the data, and the cost surface.
- "A bigger learning rate always learns faster." No. A bigger step can overshoot and make the cost grow.
- "An epoch is one parameter update." Not in this full-data example. An epoch is one pass through the dataset. How many updates happen inside it depends on the training style.
- "The gradient is the cost." No. The cost is a score. The gradient is a direction signal that helps choose the next parameter update.
- "More epochs always make a model better." More updates can reduce training cost, but later we must check how the model behaves on data it did not train on.
Quick check
Why does Gradient Descent subtract the gradient instead of adding it?
The gradient points toward the direction where the cost increases. Subtracting it moves the parameters in the opposite, downhill direction.
What does the learning rate control?
It controls the size of each parameter update. A very small rate can make learning slow, while a very large rate can overshoot and make the cost unstable.
What is an epoch?
An epoch is one complete pass through the training examples. After an epoch, the model may have updated its parameters one or more times depending on the training style.
If training MSE falls, what has Gradient Descent actually shown?
It has shown that the current parameter updates are reducing MSE on the training data. It has not yet proved that the model will work well on new data.
Recap
Key Points
- Gradient Descent repeatedly changes parameters in the direction that should reduce the cost.
- The gradient is a direction signal. The learning rate controls the size of the step.
- For a line, Gradient Descent updates both w, the slope, and b, the intercept.
- An epoch is one complete pass through the training data. Watching cost across epochs shows whether training is improving.
- A falling training cost is useful evidence about optimisation, but it is not proof that the model generalises to new data.
The model can now adjust one line with one feature. Next, we will give the model more than one input, such as size, number of bedrooms, and distance from the station. That is Multiple Linear Regression.
Theia
Can you explain this concept?
Practise Gradient Descent: Walking Downhill with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.