Classification And The Confusion Matrix

When high accuracy misses the purpose

Imagine a fraud system that is correct on ninety-nine out of every hundred transactions. That sounds excellent, until you learn it flags no fraud at all. How can both statements be true?

This is Module one, Lesson eight: Classification and the confusion matrix. By the end, you should be able to name the four outcomes of a binary classifier, calculate accuracy, precision and recall in our transaction example, and explain why the event's base rate changes what an alert means.

We'll first count outcomes, then compare a no-alert baseline with an alerting model, and finally ask what the numbers mean for the people who must act on the alerts.

Distinguish classification from regression

Our delivery-cost model predicts a numerical quantity. That task is regression. A fraud system may classify a transaction as suspected fraud or legitimate, perhaps by first producing a score. That is classification. A binary classifier has two classes.

Define the positive class

Positive is simply the class designated for the question. Here it means suspected fraud, not something desirable. Negative means legitimate under this task definition. Keep that meaning steady as we count mistakes.

Build the four outcomes

A confusion matrix compares the predicted category with the actual or established category. True means the classification was correct; false means it was wrong. Positive and negative identify the class. Combining those two dimensions yields four kinds of result.

A true positive is an actually fraudulent transaction correctly flagged. A false positive is a legitimate transaction flagged as fraud. A false negative is actual fraud the model missed. A true negative is a legitimate transaction correctly left alone.

Follow alert and investigation outcomes

Picture an alert queue and two later investigation outcomes. An alert that investigators confirm as fraud is a true positive. An alert on an innocent customer is a false positive and may create friction. A fraudulent transaction that went through without an alert is a false negative. A legitimate purchase that proceeds without an alert is a true negative.

Keep outcome labels within their scope

The terms describe classification outcomes under the chosen ground truth. They do not by themselves say whether the alert was reviewed in time, whether fraud was prevented, or whether the ground-truth labels are complete. Lesson two warned that uninvestigated fraud can be missing from a labeled dataset.

Count the population and base rate

Use a hypothetical batch of ten thousand transactions. One hundred are fraudulent; nine thousand nine hundred are legitimate. Fraud's base rate, or prevalence, is one hundred divided by ten thousand: one percent.

Evaluate the no-alert baseline

Now use a model that calls every transaction legitimate. It gets all nine thousand nine hundred legitimate transactions right and misses all one hundred frauds. Accuracy is correct classifications divided by all classifications. Nine thousand nine hundred divided by ten thousand equals ninety-nine percent accuracy.

Define recall

But recall asks how many actual positive cases were caught. It is true positives divided by all actual positives. This model catches zero of the one hundred frauds, so recall is zero percent.

Define precision and the zero-denominator case

Precision asks what fraction of predicted positive alerts really are positive. It is true positives divided by all predicted positives. The no-alert model predicts no positives, so this formula has zero in the denominator. Its precision is undefined under the basic calculation. Some software reports a chosen convention; if it does, state that convention rather than pretending we measured alert quality.

Count a second model on the same cases

Now consider a second model on the same ten thousand transactions. It catches eighty frauds: eighty true positives. It misses twenty frauds: twenty false negatives. It also flags one hundred twenty legitimate purchases: one hundred twenty false positives. The remaining nine thousand seven hundred eighty legitimate purchases are true negatives.

Calculate alert precision

First count alerts. Eighty true positives plus one hundred twenty false positives equals two hundred alerts. Precision is eighty divided by two hundred, or forty percent. Four in ten alerts correspond to fraud under these hypothetical labels.

Calculate fraud recall

There are one hundred actual frauds. Recall is eighty divided by one hundred, or eighty percent. The model catches eight in ten actual frauds.

Calculate accuracy

Accuracy counts correct results: eighty true positives plus nine thousand seven hundred eighty true negatives equals nine thousand eight hundred sixty. Divide by ten thousand and get ninety-eight point six percent. That is lower than the no-alert model's ninety-nine percent, even though this model catches substantial fraud.

Connect metrics to workflow consequences

The comparison shows why one metric can point away from the task's purpose. It does not prove the alerting model is ready to deploy. False alerts cost review time and may disrupt legitimate customers; missed fraud can impose losses. We need the workflow and consequences.

Explain prevalence and the alert mixture

The base rate is only one percent. There are ninety-nine legitimate transactions for every fraudulent one in this hypothetical population. Even a small rate of false alerts among legitimate purchases can therefore generate many false positives in absolute number.

If the shop moves to a population where fraud is rarer, precision can fall even if the model's ability to catch fraud and its false-positive rate within each class stay the same. The alert queue then contains a different mixture of true and false alerts. Conversely, a higher fraud prevalence can change that mixture in the other direction.

State the operating population

State the operating population when reporting precision. A test on a fraud-enriched dataset may be useful for development but can give a misleading impression of real-world alert quality if its base rate differs from deployment.

Check your understanding

Why can a ninety-nine-percent-accurate fraud model be operationally useless in this example?

Fraud is rare enough that calling everything legitimate gets ninety-nine percent accuracy while catching none of the one hundred fraudulent transactions. Recall is zero, and the model provides no alerts for the fraud-review task.

For the second model, two hundred alerts include eighty true frauds. What are its precision and recall?

Precision is eighty out of two hundred alerts, or forty percent. Recall is eighty out of one hundred actual frauds, or eighty percent. Those numbers organize evidence; the decision also needs prevented losses, review capacity and customer effects.

Connect outcomes and denominators

We separated four outcomes: true and false positives, true and false negatives. Accuracy counted all correct classifications. Recall counted caught fraud among actual fraud. Precision counted real fraud among alerts. The no-alert baseline showed why accuracy alone can fail when the positive event is rare.

Continue to thresholds and workload

An alert system joins model scores to human capacity, customer experience and policy. Lesson nine will make that connection explicit. We will move the threshold that turns a score into an alert and see how the operating point changes both catches and workload.