Every Threshold You Guessed Is Wrong in One Direction or the Other

Detection thresholds fail in two directions, and both come from the same cause: somebody chose a number before looking at the data.

Too Low

We wrote a rule for permanent deletion of mail in a mailbox belonging to another account. Delegated deletion. The reasoning seemed sound: destroying mail in somebody else's mailbox is not something people do casually, so any occurrence is worth reporting.

The threshold was set at one.

It fired on every result in the backtest, all of them plainly legitimate. The organization ran on shared mailboxes. Staff with delegated access routinely cleared marketing mail and newsletters from the mailboxes they helped manage. The observed maximum was two deletions in an hour, and the observed normal was one.

A threshold of one on a behaviour whose normal is one is not a detection. It is a report of everyday work with a severity label attached.

Corrected to five, the rule stayed silent through the entire backtest and would still catch somebody clearing out a colleague's mailbox in volume.

Too High

The opposite error is less visible and more dangerous, because a rule that never fires looks like a rule with nothing to report.

A bulk file modification rule was set at forty distinct files within thirty minutes, aimed at ransomware. The reasoning was that encryption moves in hundreds, so forty is conservative.

The measured ceiling for legitimate bulk modification in that tenant was fourteen files in an hour, and the second busiest hour was ten.

Forty is not conservative against fourteen. It is nearly three times the observed maximum, which means the rule has no chance of detecting a slow or partial encryption event, and no chance of detecting an attacker who has read the same public guidance about typical thresholds.

Twenty would have been defensible. Forty was a number chosen by intuition and never checked.

The Story Text Was Wrong Too

The alert narrative for that rule stated that the observed maximum for a single account over the reviewed period was twenty files. The actual measured maximum was fourteen.

Nobody had invented twenty maliciously. It was an approximate recollection that went into the text and was never verified against the query that produced it.

This matters more than it appears. Alert narrative is what an analyst reads at two in the morning, and it is where they get their sense of whether the number in front of them is unusual. A narrative containing a figure that is not in the data is worse than no narrative, because it is confidently wrong.

Every number in an alert should come from a query someone can re-run.

The Method

Before setting any threshold, run the aggregation the rule will run, over a period long enough to include a full business cycle, with no threshold at all. Order by the metric descending and look at the top twenty rows.

That gives three things:

• The ceiling. The busiest legitimate occurrence.

• The shape. Whether the distribution has a long tail or a hard edge, which
• tells you how much headroom the threshold needs.

• The population. Who does this, how often, and whether the busiest account is
• a person, a service account or a synchronisation client that should have been
• excluded entirely.

Set the threshold above the ceiling with enough headroom to absorb a busy week, and no more. Then record the ceiling in the rule documentation, so the next person to review it knows what the number was based on and can see whether it still holds.

Headroom Is A Judgement, Not A Formula

There is no correct multiplier. What there is, is an obligation to be able to answer two questions about any threshold in production:

• What is the observed normal, and when did you last measure it?

• What would an attacker have to do to stay below this number, and would that
• still be worth their while?

If the answer to the second is that they could comfortably operate underneath it, the threshold is decoration.