Anomaly Detection

The sim above is the whole idea: learn what normal looks like, score everything by how far from normal it is, then draw a line you can afford. Everything below is optional.

Learning without being taught

Most machine learning you have heard of is supervised: you hand the method thousands of examples labelled fraud and not fraud, and it learns the difference. Anomaly detection is usually unsupervised — it gets one pile of mostly-normal data and no labels at all.

That is not laziness; it is the situation. Fraud is rare, so labels are scarce and lopsided. Labels arrive late — a chargeback lands weeks after the transaction. And the next attack is not in your labelled history by definition, because it hasn’t happened yet. A method that only recognises the frauds it was shown misses the new one; a method that only knows what normal looks like notices anything that isn’t, including a technique nobody has seen before.

The cost is the flip side of the same coin: an unsupervised detector cannot tell you what is wrong, only that something is unusual. Unusual is not the same as bad. That gap is exactly what the review team is for.

The three definitions of normal

The sim’s three buttons are toy versions of three real families. Each answers “is this surprising?” honestly, and they part company on the borderline.

In the sim	What it measures	Real methods it stands in for
Distance	How far from the middle of the data, all measures combined	Mahalanobis distance, k-nearest-neighbour distance
Density	How empty this neighbourhood is compared with its neighbours’	Local Outlier Factor (LOF), DBSCAN’s noise points, Isolation Forest in spirit
Deviation	How far out on the single most unusual measure	z-scores, three-sigma control limits, robust variants using median and MAD

The conference trip is the case that separates them. Sixteen purchases 1,740 km from home, all in one week: far from the middle of the card’s history, so distance and deviation both flag every one of them — but tightly packed together, so the density method reads them as a crowd, and a crowd is normal wherever it sits. Neither answer is wrong. They are answers to different questions.

Real detectors add a fourth family the sim skips: methods that model normal explicitly (Gaussian mixtures, autoencoders) and score a point by how badly the model reproduces it.

The dial is priced, and the prices are yours

It is tempting to look for the correct threshold. There is no universal one — but there is a correct one for you, and finding it starts by admitting what each mistake costs.

Every setting spends the same fixed error budget in a different place. Tighten it and missed fraud turns into stopped customers; loosen it and stopped customers turn back into missed fraud. The count of mistakes barely moves. What moves is the bill — and that is the only thing that can tell you where to stand. A wrongly declined card is a furious phone call, a re-issued card and a sale that did not happen. A missed $1,900 charge is a chargeback and a refund. Somewhere in banking someone has put a number on both, and the line sits where those two numbers balance.

Which makes the honest way to set a threshold not “three sigma” but arithmetic on two prices you had to go and find out. Statistics can tell you how surprising a transaction is. It cannot tell you what a furious customer is worth — that is a business judgement, and swapping it for a round number from a textbook is how a detector ends up tuned for nobody in particular.

Do not expect the cheapest setting to sit still, either. Real fraud is lumpy: a handful of large losses among many small ones, so the bill jumps as the line crosses each big one, and the cheapest point wanders as the mix changes. Which is the deeper reason nobody ships a threshold and walks away.

The same trade shows up wherever detection does, and the balance moves:

• Airport screening leans hard toward catching everything, and pays for it in queues — a false alarm costs a passenger four minutes.
• Account-security checks lean the other way: a step-up challenge rather than a hard block, because locking out a real customer is worse than one extra login to verify.
• Intensive-care alarms are tuned tight, and the cost lands as alarm fatigue — the well-documented failure where staff stop reacting because most alerts are nothing.

What “normal” does next

Two things break a detector that worked on the day it was built.

Normal drifts. People move house, change jobs, start shopping somewhere new; traffic grows; a product launch changes the mix. A model of last quarter’s normal slowly turns every ordinary thing into an anomaly. Detectors are refit on a rolling window for exactly this reason, and the refit itself is a judgement: too short and last week’s incident becomes the new normal, too long and real change reads as fraud for months.

Contamination. The data you learn “normal” from already contains some of the bad thing. Methods take an expected-anomaly rate as a parameter for this reason — and if the attack has been running quietly for a year, it is inside the cluster the detector is learning to trust.

Where this shows up beyond fraud

Same method, different setting. In each, “normal” is a pattern the system already has, and the anomaly is departure from it.

• Server and application metrics. Normal is latency, error rate and CPU tracing their usual daily rhythm — busy mid-morning, quiet at 3 a.m. An anomaly is latency three times its band for this hour, or errors climbing while traffic is flat: a bad deploy or a failing dependency, caught before a human notices.
• Manufacturing quality control. Normal is a part’s dimensions, weight and torque inside an envelope built from thousands of good parts. An anomaly is a part whose combination of measurements falls outside it — each reading individually in spec, but together the signature of a drifting tool or a bad batch.
• Network intrusion detection. Normal is a host’s usual mix of ports, peers, volumes and hours. An anomaly is a workstation opening hundreds of outbound connections to new addresses at midnight — caught without a signature for that specific attack.
• Sensor fleets. Normal is a sensor reading in step with its neighbours and its own history. An anomaly is one going flat, wildly noisy, or steadily diverging from the others measuring the same thing — usually a faulty sensor, and catching it stops bad data poisoning everything downstream.
• Unusual-login detection. Normal is a couple of known devices, a home region, human hours. An anomaly is a login from a new country minutes after one from home. Note where the dial sits here: a challenge, not a block.
• Patient monitoring. Normal is the range a patient’s vitals have held for hours. An anomaly is a coordinated shift across heart rate, blood pressure and oxygen saturation that no single-threshold alarm would catch.

▸ References

• Markus Goldstein and Seiichi Uchida, “A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data” (2016) — the honest comparison of how differently these families behave on the same data.
• Markus M. Breunig et al., “LOF: Identifying Density-Based Local Outliers” (2000) — the local-density method the sim’s Density button is a toy of.
• Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou, “Isolation Forest” (2008).
• Varun Chandola, Arindam Banerjee and Vipin Kumar, “Anomaly Detection: A Survey” (2009).