Support Vector Machines: Find a Boundary with Room Around It
A classifier needs a boundary
A Support Vector Machine, or SVM, searches for a decision boundary that separates classes. In a simple two-feature picture, the boundary is a line. With more features, it is a hyperplane that we cannot draw directly.
Key Points
- The boundary separates positive and negative regions.
- The margin is the space between the boundary and the closest training points.
- SVM prefers a wide margin when the data allows it.
- The closest points are the support vectors.
The margin is a safety gap
Two boundaries can classify every training point correctly. SVM prefers the one with more room before the nearest points. A wider margin can make the boundary less sensitive to small changes in the training sample.
Key Points
- Large margin can improve generalisation.
- A perfectly separating boundary is not automatically best.
- Some overlap may be allowed with a soft-margin SVM.
- The cost parameter C controls the tradeoff between margin width and training violations.
A score gives side and distance information
1def classify(score):
2 return 1 if score >= 0 else 0
3
4scores = [-2.5, -0.1, 0.4, 3.0]
5print("Labels:", [classify(score) for score in scores])
6print("Distance-like scores:", scores)Labels: [0, 0, 1, 1] Distance-like scores: [-2.5, -0.1, 0.4, 3.0]
The sign gives the predicted side of the boundary. The magnitude gives a distance-like confidence in the linear SVM score, but it is not automatically a calibrated probability.
Soft margins handle imperfect separation
Real data often contains overlap or noisy labels. A soft-margin SVM allows some points to be inside the margin or misclassified. C controls how costly those violations are.
Key Points
- Large C penalises training violations strongly and can produce a narrower, more complex boundary.
- Small C allows more violations in exchange for a wider margin.
- C is a hyperparameter selected with validation data.
- The best C depends on scale, noise, and the evaluation goal.
Kernels represent curved boundaries
A linear SVM uses the original feature space. Kernel methods calculate similarities that act as if the data had been mapped into a richer feature space. An RBF kernel can create curved boundaries without explicitly writing every transformed feature.
Key Points
- The kernel controls the notion of similarity.
- RBF gamma controls how local the influence of a point is.
- Large gamma can create very wiggly boundaries.
- Small gamma creates smoother, broader influence.
- Kernel SVMs can be expensive on large datasets.
Scale features before using a distance-like margin
SVM margins and kernels are sensitive to feature scale. A feature measured in rupees can dominate one measured in years if columns are not scaled. Fit the scaler on training data and apply it consistently.
Key Points
- Scaling is usually important for SVMs.
- Keep the transformation in a pipeline.
- Tune C and kernel parameters after the feature representation is fixed.
- Interpret weights in the scaled feature units.
Compare margin settings
1settings = [
2 {"C": 0.1, "gamma": 0.1, "validation_f1": 0.74},
3 {"C": 1.0, "gamma": 0.1, "validation_f1": 0.81},
4 {"C": 10.0, "gamma": 2.0, "validation_f1": 0.76},
5]
6
7best = max(settings, key=lambda setting: setting["validation_f1"])
8print("Best C:", best["C"])
9print("Best gamma:", best["gamma"])
10print(f"Validation F1: {best['validation_f1']:.2f}")Best C: 1.0 Best gamma: 0.1 Validation F1: 0.81
The third setting is more local and more strongly penalised, but it performs worse on this validation sketch. Settings must be evaluated rather than judged from their size alone.
SVM scores are not automatically probabilities
A margin score tells which side of the boundary a row lies on and how far it is from the boundary in the model’s geometry. If a product needs probability thresholds or expected costs, calibrate scores using a separate validation process.
Key Points
- Use a calibrated method when probability meaning matters.
- Do not compare raw scores from unrelated models as probabilities.
- Threshold selection still belongs to the decision layer.
- Check class weights when positive and negative errors have unequal cost.
Where people get confused
Key Points
- “Support vectors are all training rows.” They are the points that determine the margin and boundary.
- “A larger C is always better.” It can reduce training violations while increasing sensitivity and overfitting.
- “The SVM score is a probability.” It is a margin-related score unless calibrated.
- “Kernels are free.” Kernel calculations can be expensive and sensitive to hyperparameters.
- “SVMs never need scaling.” Margin and kernel calculations usually make scaling important.
Quick check
What is the SVM margin?
It is the gap around the decision boundary before the closest training points. SVM seeks a boundary with a useful margin.
What are support vectors?
They are the training points closest to or inside the margin that strongly influence the fitted boundary.
What does C control?
C controls how strongly the model penalises margin violations compared with keeping a wide margin.
Why are SVM features usually scaled?
The geometry of margins and kernel similarities depends on feature distances and magnitudes. Unscaled units can let one column dominate.
Chapter 6 recap
Key Points
- Classification predicts categories and often probabilities.
- Logistic Regression creates probabilities from a sigmoid score.
- Confusion matrices, precision, recall, F1, imbalance analysis, ROC, and AUC judge decisions from several angles.
- KNN predicts from nearby examples; Naive Bayes combines probability evidence.
- Trees, forests, boosting, and SVMs create different boundaries with different complexity controls.
- The model is only one part of classification. Thresholds, class costs, validation, calibration, and leakage determine whether it is useful.
- The next chapter begins unsupervised learning, where labels are not available.
Theia
Can you explain this concept?
Practise Support Vector Machines: Find a Boundary with Room Around It with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.