ROC Curve and AUC: Ranking Across Thresholds
One threshold is only one operating point
Precision, recall, and accuracy describe predictions after choosing a threshold. A threshold can be useful for deployment, but during model comparison we may want to see behaviour across many thresholds.
Key Points
- A score model ranks examples from more positive-looking to less positive-looking.
- Moving the threshold changes TP, TN, FP, and FN.
- ROC analysis summarises the tradeoff across thresholds.
- The final operating threshold still needs a business decision.
True positive rate and false positive rate
True positive rate is recall: TPR = TP ÷ (TP + FN). False positive rate is FPR = FP ÷ (FP + TN). ROC plots TPR on the vertical axis against FPR on the horizontal axis for many thresholds.
Key Points
- TPR asks how many actual positives were found.
- FPR asks how many actual negatives were incorrectly alerted.
- The ideal top-left point has TPR 1 and FPR 0.
- Moving toward more positive predictions often raises both rates.
Calculate rates for one threshold
1tp, fn, fp, tn = 70, 30, 20, 880
2
3tpr = tp / (tp + fn)
4fpr = fp / (fp + tn)
5
6print(f"TPR: {tpr:.3f}")
7print(f"FPR: {fpr:.3f}")TPR: 0.700 FPR: 0.022
This model finds 70% of positives and alerts on about 2.2% of negatives at the selected threshold. Whether that is useful depends on the cost and capacity of the application.
Build a small ROC table
1points = [
2 {"threshold": 0.9, "tpr": 0.20, "fpr": 0.01},
3 {"threshold": 0.6, "tpr": 0.70, "fpr": 0.02},
4 {"threshold": 0.3, "tpr": 0.90, "fpr": 0.15},
5]
6
7for point in points:
8 print(
9 f"threshold={point['threshold']}: "
10 f"FPR={point['fpr']:.2f}, TPR={point['tpr']:.2f}"
11 )threshold=0.9: FPR=0.01, TPR=0.20 threshold=0.6: FPR=0.02, TPR=0.70 threshold=0.3: FPR=0.15, TPR=0.90
The threshold of 0.3 catches more positives but also alerts on more negatives. A deployed system may choose 0.6 or 0.3 based on the cost of each error.
AUC summarises ranking quality
AUC means area under the ROC curve. It can be interpreted as the probability that a randomly selected positive receives a higher score than a randomly selected negative, with ties handled by convention.
Key Points
- AUC near 1 means strong positive-versus-negative ranking.
- AUC near 0.5 is roughly random ranking for balanced sampling assumptions.
- AUC below 0.5 can indicate reversed ranking, though the score direction may be flipped.
- AUC does not choose the deployment threshold or report probability calibration.
Compute a simple trapezoid area
When ROC points are ordered by increasing FPR, the trapezoid rule estimates area between neighbouring points. Real libraries calculate the curve with careful threshold handling.
1roc_points = [(0.0, 0.0), (0.02, 0.70), (0.15, 0.90), (1.0, 1.0)]
2
3auc = 0.0
4for (left_fpr, left_tpr), (right_fpr, right_tpr) in zip(roc_points, roc_points[1:]):
5 width = right_fpr - left_fpr
6 height = (left_tpr + right_tpr) / 2
7 auc += width * height
8
9print(f"Approximate AUC: {auc:.3f}")Approximate AUC: 0.918
ROC AUC can hide rare-class workload
When positives are very rare, a model can have a respectable ROC AUC while producing many false positives compared with the number of real positives. A precision-recall curve can focus more directly on positive-alert quality in that situation.
Key Points
- Use ROC AUC to compare ranking across thresholds.
- Inspect precision-recall behaviour when the positive class is rare.
- Report a chosen operating point and its workload.
- AUC is not the same as accuracy, precision, recall, or calibration.
AUC is not a business metric by itself
A model with higher AUC may be worse at the threshold your team can operate. Ranking is useful, but the final choice should consider capacity, intervention cost, false-negative harm, and whether probabilities are trustworthy.
Key Points
- Choose threshold on validation data.
- Report confusion counts or rates at that threshold.
- Check performance by relevant groups and time periods.
- Use calibration analysis if probability values drive decisions.
Where people get confused
Key Points
- “AUC is the accuracy.” AUC measures ranking across thresholds; accuracy uses one threshold.
- “AUC of 0.8 means 80% of predictions are correct.” It has a ranking interpretation instead.
- “A high AUC chooses the threshold.” It does not include capacity or error-cost choices.
- “ROC is always the best curve for imbalance.” Precision-recall views can be more revealing when positives are rare.
- “AUC proves probabilities are calibrated.” Ranking and calibration are different properties.
Quick check
What is TPR?
True positive rate is recall: the fraction of actual positives that receive a positive prediction at a threshold.
What is FPR?
False positive rate is the fraction of actual negatives that receive a positive prediction: FP divided by FP plus TN.
What does ROC AUC mainly measure?
It summarises how well the model ranks positives above negatives across thresholds.
Why can ROC AUC be insufficient for a rare positive class?
It can hide the number of false alerts relative to the small number of positives. Precision-recall and the operational threshold may be more informative.
Recap
Key Points
- ROC curves show TPR against FPR across thresholds.
- AUC summarises ranking quality, not one final operating decision.
- Rare-class problems need precision-recall and workload checks too.
- The next lesson introduces K-Nearest Neighbours, which predicts from nearby examples.
Theia
Can you explain this concept?
Practise ROC Curve and AUC: Ranking Across Thresholds with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.