Anomaly Detection: Unusual Does Not Automatically Mean Wrong
A strange UPI payment needs a question, not a verdict
A customer usually sends ₹200 to ₹800, then sends ₹60,000 at 2 a.m. The event is unusual compared with their history. It may be fraud, a hospital bill, or a valid business payment. An anomaly detector can prioritise it for review, but cannot decide the truth from rarity alone.
Anomaly detection looks for observations that differ from a reference pattern. In this chapter, the reference can be learned without labelled anomaly examples. When reliable fraud labels exist, supervised classification may be a better or complementary approach.
Define what normal is relative to
A ₹60,000 transfer may be rare for a student and routine for a wholesale shop. A late-night transfer may be ordinary for a night-shift worker. The baseline population, time window, and context determine whether a score has useful meaning.
Key Points
- Point anomaly: one row is unusual in the reference population.
- Contextual anomaly: a row is unusual for its time, customer, or situation.
- Collective anomaly: a sequence or group is unusual even if individual rows look ordinary.
- Choose the unit of review before choosing a detector.
A simple threshold teaches the score idea
Suppose comparable clean measurements have a mean of 10 and a standard deviation of 2. A value of 11 is half a standard deviation above the mean. A value of 18 is four standard deviations above it.
A z-score is z = (x − mean) ÷ standard deviation. x is the observed value; the mean and standard deviation describe the reference measurements. A large absolute score suggests a distant value under that reference, not a proven bad event.
1reference_mean, reference_std = 10, 2
2for value in [11, 18]:
3 z = (value - reference_mean) / reference_std
4 print(f"value={value}: z={z:.1f}, flagged={abs(z) > 3}")value=11: z=0.5, flagged=False value=18: z=4.0, flagged=True
A three-standard-deviation threshold is only a teaching rule. Skewed transaction values and heavy tails can make it inappropriate. Compute reference statistics from the permitted history, not from the future events you want to detect.
Separate a score from an alert
1. Score
Compare each event with an appropriate reference.
Unusual is a model relationship.
2. Flag
Apply a threshold that fits review capacity and error costs.
An alert is a request for attention.
3. Investigate
Check context and evidence before deciding what happened.
A verdict needs more than rarity.
A score orders observations by unusualness. A threshold turns that ranking into alerts. A reviewer or downstream process then decides whether to act. Keeping those steps separate lets you change review capacity without pretending the scoring model proved misconduct.
Key Points
- Know whether larger or smaller scores mean more unusual for the chosen library.
- Choose a threshold from review capacity, costs, and validation evidence.
- Keep the original score for later analysis.
- Audit false alarms, not just confirmed anomalies.
Isolation Forest isolates unusual rows quickly
Isolation Forest repeatedly makes random feature splits. A point far from the usual data often needs fewer splits to become isolated. Averaging across many such trees provides an unusualness score.
You do not need the full score formula here. The intuition is different from a classifier: the trees do not learn fraud labels. They measure how easy each row is to separate from the reference distribution.
1import numpy as np
2from sklearn.ensemble import IsolationForest
3
4reference = np.array([[i / 10] for i in range(-10, 11)])
5model = IsolationForest(n_estimators=100, contamination="auto", random_state=7)
6model.fit(reference)
7query = [[0.0], [20.0]]
8scores = model.score_samples(query)
9print("Rows scored:", len(scores))
10print("Far point has lower score:", bool(scores[1] < scores[0]))
11print("Model labels:", model.predict(query).tolist())Rows scored: 2 Far point has lower score: True Model labels: [1, -1]
Here lower score_samples values are more abnormal. predict uses 1 for an inlier and −1 for an outlier. contamination affects the fitted decision cutoff; a numeric value is an assumption about the flagged fraction, not a known fraud rate.
Other methods ask different unusualness questions
Key Points
- DBSCAN noise identifies rows outside fitted dense regions.
- Local Outlier Factor compares local density with nearby rows; a locally sparse point can stand out even inside a varied dataset.
- PCA reconstruction error can highlight a row poorly represented by retained components.
- Simple rules can identify domain failures, such as an impossible negative duration.
A value can be globally rare but locally normal. A detector’s assumptions therefore matter as much as the algorithm name. Default outlier fitting finds odd rows in a supplied collection; novelty detection fits a reference and then scores new rows. Check that the estimator supports the workflow you intend.
Evaluate with the evidence you actually have
If reviewers provide trustworthy labels, estimate alert precision, missed events where observable, and workload on later data. If labels are sparse, inspect samples across score bands, include some unflagged rows, and record how representative the review sample is.
Reviewing only flagged rows cannot tell you how many anomalies were missed. Synthetic unusual examples can test failure handling, but do not establish real-world recall.
1reviewed_alerts = 20
2confirmed_cases = 6
3daily_capacity = 25
4precision = confirmed_cases / reviewed_alerts
5
6print(f"Reviewed-alert precision: {precision:.2f}")
7print("Within daily capacity:", reviewed_alerts <= daily_capacity)
8print("Missed cases known:", False)Reviewed-alert precision: 0.30 Within daily capacity: True Missed cases known: False
The 30% describes these reviewed alerts. It is not a population recall estimate, and biased review sampling can make even precision fail to generalise.
Normal behaviour can change
Diwali shopping, a new payment channel, or a sensor replacement can change the reference distribution. A detector trained on older data can flood the queue with valid new behaviour.
Key Points
- Track the score distribution and alert volume over time.
- Use a reference window appropriate to the decision.
- Investigate changes before automatically retraining on possibly corrupted data.
- Check important customer groups so one valid population is not consistently treated as unusual.
Where people get confused
Key Points
- Rare does not mean fraudulent, unsafe, or erroneous.
- A contamination setting is not a measured ground-truth anomaly rate.
- An alert threshold creates workload; it does not change the historical truth.
- A score direction must be checked in the library documentation.
- Perfect alert precision does not prove that no anomalies were missed.
Quick check
What is the difference between an anomaly score and a verdict?
A score measures unusualness under a reference model. A verdict requires evidence about the event and the decision context.
Why does context matter for a rare transaction?
The same amount or time can be unusual for one person and routine for another. The baseline and available history determine what normal means.
What is the Isolation Forest intuition?
Random splits tend to isolate unusual rows in fewer steps than ordinary rows. Many trees combine that evidence into a score.
Why can reviewed alerts alone not establish recall?
Recall needs the total number of actual anomalies, including missed ones. Reviewing only flagged cases leaves unflagged anomalies unknown.
Chapter 7: what you can now decide
Key Points
- Choose clustering for similar rows and association for co-occurrence.
- Compare centre-based, hierarchical, and density-based grouping assumptions.
- Use PCA for a reusable linear representation and t-SNE for cautious visual exploration.
- Treat anomaly detection as scoring and investigation under a defined reference.
- In every method, feature meaning, scale, stability, and evaluation determine whether a pattern is useful.
The next chapter takes a fitted model beyond the notebook: save it, keep preprocessing with it, serve predictions, and monitor what happens after launch.
Theia
Can you explain this concept?
Practise Anomaly Detection: Unusual Does Not Automatically Mean Wrong with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.