PCA: Keep the Directions with the Most Variation
Two columns may repeat almost the same information
A dataset records maths and physics marks. Students who score high in one often score high in the other. The dots form a diagonal band rather than filling the whole two-dimensional chart.
You might describe most of that spread with one coordinate along the band, plus a smaller coordinate across it. Principal Component Analysis, or PCA, finds these directions from the data. Keeping fewer directions reduces the number of coordinates while losing some information.
Tip:PCA compresses the representation. It does not automatically find the features that best predict a label.
Centre the points first
Use three points: (1, 1), (2, 2), and (3, 3). The average of each column is 2. Subtracting the means gives (−1, −1), (0, 0), and (1, 1). Now the origin is the centre of the data cloud.
1rows = [(1, 1), (2, 2), (3, 3)]
2means = [sum(row[j] for row in rows) / len(rows) for j in range(2)]
3centred = [tuple(value - mean for value, mean in zip(row, means)) for row in rows]
4print("Column means:", means)
5print("Centred rows:", centred)Column means: [2.0, 2.0] Centred rows: [(-1.0, -1.0), (0.0, 0.0), (1.0, 1.0)]
Centring is not the same as standardising. Standardising also divides by each column’s spread. scikit-learn PCA centres inputs automatically but does not automatically put differently scaled columns on comparable units.
Rotate the view instead of deleting a column
For these points, the main direction runs along the diagonal. Project a centred point onto that direction and you keep all its variation. Project it onto the perpendicular direction and every point lands at zero.
Keep the long direction, discard the smaller spread
PC1 follows the diagonal spread; PC2 is perpendicular. Dotted projection lines show the variation lost when only PC1 is retained. Schematic, not a fitted dataset.
A principal component is a new axis formed from a weighted combination of original features. PC1 captures the largest available variance. PC2 captures the largest remaining variance subject to being perpendicular to PC1. Later components follow the same rule.
Calculate a projection with real numbers
A unit-length diagonal direction is (1 ÷ √2, 1 ÷ √2). For centred point (1, 1), multiply each coordinate by its direction weight and add: 1 ÷ √2 + 1 ÷ √2 ≈ 1.414. This is the point’s new one-dimensional coordinate.
For one component, score = Σ(xⱼ − meanⱼ) × vⱼ. xⱼ is the original value of feature j, meanⱼ is its training mean, vⱼ is the component weight, and Σ adds across features. The calculation is called a projection. You do not need matrix algebra to follow this example.
1import math
2
3direction = (1 / math.sqrt(2), 1 / math.sqrt(2))
4centred_rows = [(-1, -1), (0, 0), (1, 1)]
5scores = [sum(value * weight for value, weight in zip(row, direction)) for row in centred_rows]
6print("One-dimensional scores:", [round(score, 3) for score in scores])One-dimensional scores: [-1.414, 0.0, 1.414]
Fit, transform, and reconstruct
1import numpy as np
2from sklearn.decomposition import PCA
3
4X = np.array([[1, 1], [2, 2], [3, 3]], dtype=float)
5model = PCA(n_components=1, svd_solver="full")
6scores = model.fit_transform(X)
7reconstructed = model.inverse_transform(scores)
8print("Compressed shape:", scores.shape)
9print("Explained variance ratio:", model.explained_variance_ratio_.round(3).tolist())
10print(f"Reconstruction MSE: {np.mean((X - reconstructed) ** 2):.3f}")
11print("New-row shape:", model.transform([[4, 4]]).shape)Compressed shape: (3, 1) Explained variance ratio: [1.0] Reconstruction MSE: 0.000 New-row shape: (1, 1)
The exact component sign can flip without changing the information. Here one component reconstructs the data perfectly because all three points lie exactly on one line. Real marks data usually has some spread across the band, so one component loses information.
Explained variance is not accuracy
An explained variance ratio of 0.90 means a component accounts for 90% of the total input variance used in this PCA fit. It does not mean 90% of classifications are correct, nor that 90% of all useful signal was preserved.
Key Points
- Sum ratios for retained components to see cumulative variance retained.
- More components preserve more information but compress less.
- Reconstruction error measures lost input information.
- For prediction, validate the downstream model as well as the retained variance.
A small-variance direction could distinguish two classes. Removing it may hurt a classifier even when the retained variance is very high. PCA does not see the target labels when selecting directions.
Scaling changes which direction wins
Income in rupees may have vastly larger numerical variation than age in years. PCA can then spend its first component mostly on income. Standardisation makes each column’s spread comparable if that is appropriate for your question.
Do not scale blindly. Some units encode meaningful relative variation. Decide whether you want large natural variation or equalised feature spread to drive the representation, and explain the choice.
Keep fitted transformations on the training side
Key Points
- Fit a scaler and PCA on training rows when they are part of a prediction workflow.
- Transform validation and test rows with the same fitted values.
- Fit both steps inside each cross-validation fold.
- Choose the component count using validation evidence without spending the final test set.
For a purely descriptive study, you can fit PCA to the collection you intend to describe. That is a different claim from reporting generalisation of a classifier trained after PCA.
When PCA is useful and when it is not
PCA can reduce correlated numeric dimensions, speed up later models, or provide a simple plot. It can also reduce interpretability because components combine several features rather than keeping the original columns.
Key Points
- PCA is a linear projection, so curved structure may need a different representation.
- High-variance outliers can pull component directions.
- All principal components together rotate the centred data; information is lost when components are discarded.
- Use original-space or suitable transformed-space validation to check what matters.
Quick check
What is a principal component?
It is a new direction made from a weighted combination of the original features. Components are ordered by the input variance they capture and are perpendicular to one another.
Does PCA automatically standardise features?
No. scikit-learn PCA centres features but does not automatically divide by their standard deviations. Scaling is a separate choice.
Does 95% retained variance mean 95% prediction accuracy?
No. It measures input variation preserved by the projection. Predictive signal may lie in a low-variance direction.
Why fit PCA inside a cross-validation pipeline?
PCA learns directions and means from data. Fitting it on validation rows lets those rows influence the representation used to train the model.
Keep the purpose clear
Key Points
- PCA changes coordinates and can compress data.
- Variance is a property of inputs, not an answer-quality metric.
- Inspect scaling, reconstruction, and the downstream decision.
- Next, t-SNE makes a nonlinear visual map, with different interpretation limits.
Theia
Can you explain this concept?
Practise PCA: Keep the Directions with the Most Variation with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.