Pipelines: Keep Preprocessing and the Model Together
The model learned from one representation, then received another
You trained a rent model on standardised flat sizes. During serving, someone passes raw square feet directly to the estimator. The code runs and returns a number, but the number uses the wrong feature meaning.
This mismatch is a form of training-serving skew: training and serving do not prepare inputs the same way. Keeping preprocessing and prediction together makes the intended order explicit.
See the mistake with a small calculation
Suppose the training mean was 800 square feet and its standard deviation was 200. A 1,000-square-foot flat becomes (1,000 − 800) ÷ 200 = 1. A model expecting this coordinate predicts ₹28,000. Feeding it 1,000 instead gives ₹3,025,000.
1area = 1000
2train_mean, train_std = 800, 200
3scaled_area = (area - train_mean) / train_std
4
5def fitted_estimator(value):
6 return 25000 + 3000 * value
7
8print(f"With training scale: ₹{fitted_estimator(scaled_area):,.0f}")
9print(f"With raw input: ₹{fitted_estimator(area):,.0f}")With training scale: ₹28,000 With raw input: ₹3,025,000
Both calls use the same estimator formula. The error came from a missing transformation. A model file without its fitted transformations can therefore be misleading even when loading succeeds.
A Pipeline records the sequence
A scikit-learn Pipeline chains named steps. Intermediate steps transform the data; the final step is the estimator. Calling fit learns the intermediate transformations from the training rows and fits the final model on their output.
Fit: learn from training rows
Training rows → fit scaler → transform rows → fit estimator
Only the training side teaches the scaler and model. Cross-validation repeats this within each fold.
Predict: reuse the fitted state
New rows → stored scaler → stored estimator → predictions
A new flat does not redefine the training mean. Save both fitted steps as one pipeline.
Example: stored mean 800, spread 200. Area 1,000 becomes 1, not 1,000, before the estimator.
Calling predict applies the stored transformations and then predicts. It does not calculate a new mean from the single incoming flat. That consistency is the main reason to use the pipeline.
Fit and predict with the same object
1from sklearn.pipeline import Pipeline
2from sklearn.preprocessing import StandardScaler
3from sklearn.linear_model import LinearRegression
4
5X = [[500], [700], [900], [1100]]
6y = [18000, 24000, 30000, 36000]
7pipeline = Pipeline([
8 ("scale", StandardScaler()),
9 ("model", LinearRegression()),
10])
11pipeline.fit(X, y)
12prediction = pipeline.predict([[750]])
13
14print("Step order:", list(pipeline.named_steps))
15print(f"Stored area mean: {pipeline.named_steps['scale'].mean_[0]:.0f}")
16print(f"Predicted rent: ₹{prediction[0]:,.0f}")Step order: ['scale', 'model'] Stored area mean: 800 Predicted rent: ₹25,500
The library remembers the training mean. Scaling is not necessary for this ordinary one-feature least-squares fit, but it makes the transformation boundary easy to see. Other estimators, such as KNN or a regularised model, can be much more sensitive to scale.
Different columns need different preparation
Flat area is numeric; neighbourhood is categorical. A ColumnTransformer lets you apply different transformations to selected columns, then combines their outputs. It can sit inside a Pipeline before the estimator.
Key Points
- Numeric columns can use imputation and scaling.
- Categorical columns can use an encoder fitted on training categories.
- A new category needs an explicit policy, such as an error or an ignored category indicator.
- Column names, order, units, and missing-value rules still need an input contract.
A pipeline does not automatically convert acres to square feet or detect that two columns were swapped. Those are schema and feature-definition problems. The API lesson makes that contract explicit.
Cross-validation must receive the whole pipeline
If you scale every row first and then cross-validate only the estimator, the validation rows influenced the scale. Pass the complete unfitted pipeline to cross-validation so each fold fits its own preprocessing using only that fold’s training rows.
1import numpy as np
2from sklearn.pipeline import Pipeline
3from sklearn.preprocessing import StandardScaler
4from sklearn.linear_model import LinearRegression
5from sklearn.model_selection import KFold, cross_val_score
6
7X = [[500], [700], [900], [1100], [1300], [1500]]
8y = [18000, 24000, 30000, 36000, 42000, 48000]
9pipeline = Pipeline([
10 ("scale", StandardScaler()),
11 ("model", LinearRegression()),
12])
13folds = KFold(n_splits=3, shuffle=True, random_state=7)
14scores = cross_val_score(pipeline, X, y, cv=folds, scoring="neg_mean_absolute_error")
15print("Folds:", len(scores))
16print(f"Mean MAE: {-np.mean(scores):.2f}")
17print("Original pipeline fitted:", hasattr(pipeline.named_steps["model"], "coef_"))Folds: 3 Mean MAE: 0.00 Original pipeline fitted: False
scikit-learn maximises scores, so error scorers use a negative sign; we negate the mean to report MAE. cross_val_score fits cloned pipelines for the folds, leaving the original unfitted. The zero error comes from deliberately exact toy data, not a realistic rent forecast.
Save the fitted pipeline, not only its last step
1from pathlib import Path
2from tempfile import TemporaryDirectory
3import joblib
4from sklearn.pipeline import Pipeline
5from sklearn.preprocessing import StandardScaler
6from sklearn.linear_model import LinearRegression
7
8pipeline = Pipeline([("scale", StandardScaler()), ("model", LinearRegression())])
9pipeline.fit([[500], [700], [900], [1100]], [18000, 24000, 30000, 36000])
10
11with TemporaryDirectory() as directory:
12 path = Path(directory) / "rent-pipeline.joblib"
13 joblib.dump(pipeline, path)
14 loaded = joblib.load(path)
15 print("Loaded steps:", list(loaded.named_steps))
16 print(f"Raw-input prediction: ₹{loaded.predict([[750]])[0]:,.0f}")Loaded steps: ['scale', 'model'] Raw-input prediction: ₹25,500
The new process can receive the expected raw representation and call predict once. Keep the artifact, schema, and environment versions together as in the previous lesson.
Tuning names refer to pipeline steps
A search can select settings inside a pipeline. A parameter name such as model__alpha means “alpha on the step named model”. The double underscore connects the step name to the setting.
Key Points
- Search over the complete pipeline, not a globally transformed table.
- Choose fold design to respect time and groups.
- After selection, fit the chosen pipeline on the allowed development data.
- Use the untouched test set only for the final estimate.
Where people get confused
Key Points
- Pipeline.predict reuses learned transformations; it does not learn new ones from each request.
- A Pipeline cannot fix a feature created from future information before entering the pipeline.
- Cross-validation of only the estimator can still leak globally fitted preprocessing.
- A pipeline is a fitted sequence, not a replacement for input validation or package compatibility.
Quick check
What does a Pipeline keep together?
It keeps the ordered preprocessing steps and final estimator in one object with coordinated fitting and prediction.
Why reuse training scaling for new rows?
The estimator learned relationships in that transformed representation. Recomputing scaling from new rows changes the meaning of its inputs.
Why pass the whole pipeline to cross-validation?
Each fold must learn its preprocessing only from the training side. Passing the whole pipeline lets the library fit a separate clone per fold.
What can a pipeline not fix automatically?
It cannot infer wrong units, repair swapped feature meanings, or remove leakage already built into the source features. Those require a valid schema and feature-generation process.
One prediction path
Key Points
- Fit preprocessing and the estimator together.
- Reuse the same fitted steps during inference and save them together.
- Validate the full pipeline and its input meaning.
- Next, we expose that prediction path through an API.
Theia
Can you explain this concept?
Practise Pipelines: Keep Preprocessing and the Model Together with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.