Where should the alert line go?
In Lesson eight, a fraud model produced two hundred alerts. Investigators must decide what to review before transactions settle. A model score does not make that decision by itself. Where should the team draw the line for an alert?
This is Module one, Lesson nine: Thresholds connect scores to actions. By the end, you should be able to explain how a threshold changes catches and false alerts, calculate a review workload, and say why a curve or a single statistical score does not select the best operating point for an organization.
We'll follow the score into an alert queue, compare thresholds, and then check the capacity and consequences of using that queue.
Turn a score into an action
Many classifiers produce a score. Higher scores may indicate more evidence for the positive class under the model. A threshold turns that score into a classification: at or above the chosen value, issue a fraud alert; below it, do not.
Inspect actual score distributions
Lowering the threshold often catches more actual fraud because more cases qualify. It often also flags more legitimate purchases. Raising the threshold often reduces alerts, including false alerts, but can miss more fraud. These directions depend on the actual score distribution and task; inspect measured outcomes rather than treating them as guarantees for every system.
Evaluate the model and workflow together
A model could rank transactions usefully while the alert threshold is poorly chosen. The model and the action process must be evaluated together. Review cost, deadline, policy and consequences all shape what a useful threshold means.
Calculate a review-work component
Return to the previous lesson's two hundred alerts. Suppose reviewing one takes three minutes. Two hundred times three is six hundred minutes, or ten hours of review work.
Compare work with available capacity
Now suppose the team can provide only four hours before its decision deadline. Under this procedure it cannot finish the queue in time. A recall and precision report alone does not solve that operational problem.
Test operational alternatives
The team could choose another threshold, triage alerts by urgency, increase staffing, automate a justified part of the review, or redesign the task. Each option has costs and risks. If the threshold is raised merely to make the queue shorter, more fraud may slip through. If it is lowered to catch more fraud, reviewers may become overloaded and fail to respond well to the highest-risk cases.
Respect the decision deadline
Capacity is not a footnote. An alert that no one can inspect before it matters may deliver less benefit than its classification score suggests.
Define ROC and precision–recall denominators
A receiver operating characteristic curve, often called an ROC curve, compares true-positive rate and false-positive rate as the threshold changes. Here, the true-positive rate is recall: the share of actual frauds that are caught. The false-positive rate is the share of legitimate transactions that are incorrectly flagged. A precision-recall curve compares how many alerts are right with how many actual positives are caught.
Compare operating points
Those curves show more than one possible operating point. They can reveal that a model catches additional fraud only by creating many more false alerts. With a rare positive event such as fraud, precision-recall analysis can be especially helpful for understanding the alert queue's quality.
Look beyond a global curve summary
A single area-under-curve number summarizes performance across possible thresholds. It may hide the narrow region the team can actually operate in, such as a four-hour daily review capacity. The relevant question is what happens at a proposed threshold in the intended population and workflow.
Use F1 within its scope
F1 combines precision and recall using their harmonic mean, a formula that rewards balance between those two measures. That can help compare systems when balance is the desired statistical summary.
Name what the formula leaves out
But F1 does not include true negatives directly or arbitrary business costs. It does not know whether one missed fraud is far more expensive than several false alerts, whether a customer abandons a purchase after a false flag, or whether the team can process the queue. A higher F1 score can coexist with a worse operating decision for a specific use.
Metric choice makes some facts visible and leaves others outside the formula. Name what a score counts, then connect it to the consequences the organization actually bears.
Compare a lower-threshold example
Suppose lowering a threshold increases recall from eighty percent to ninety percent, but doubles the number of alerts. On the one-hundred-fraud example, that could mean catching ninety rather than eighty frauds, under the measured conditions. The extra catches might be valuable.
Calculate the changed workload and precision
But if the queue doubles from two hundred to four hundred alerts and each still takes three minutes, review work doubles from ten to twenty hours. A four-hour team cannot simply absorb it. In this same hypothetical group of one hundred frauds, ninety true positives among four hundred alerts gives precision of ninety divided by four hundred, or twenty-two point five percent. That is lower than the earlier forty percent. A real deployment needs measured counts from its own population rather than assuming this illustrative trade-off will recur.
Test what follows the alert
Would extra prevented loss exceed review cost and customer friction? Could a triage process identify the highest-risk cases? Do reviewers have enough time to complete useful work? A threshold decision needs answers from both model evaluation and operational testing.
Check your understanding
Does choosing the threshold with the highest statistical score necessarily maximize business value?
No. The score may omit losses, customer consequences and review capacity. Test outcomes at the proposed threshold in the relevant population, including what happens after an alert is issued.
If two hundred alerts each take three minutes, how much review time is required under our simple assumption?
Six hundred minutes, or ten hours. That assumes every alert receives the stated review and ignores breaks or other work, so it is a workload component rather than a complete staffing plan.
Connect score, threshold and workflow
A score ranks or describes cases under a model; a threshold turns scores into alerts. Moving it changes the balance of caught and missed fraud and false alerts. Curves and F1 can organize that trade-off, while review capacity and consequences determine whether a selected point can work.
Continue to probabilities and calibration
In Lesson ten, we'll ask what a score called a probability really means. Even a good ranking can give unreliable probability estimates, and that matters when a team compares expected loss with the cost of investigation.