All simulators
Thresholds, precision & ROC
Slide the decision threshold. Trade precision for recall and trace the ROC curve.
Slide the decision threshold. Trade precision for recall and trace the ROC curve.
Score distributions
ROC curve AUC = 0.954
Red dot = your threshold. Dashed line = random guessing (AUC 0.5).
Confusion matrix
Rows are the truth, columns are the decision at threshold t.
A classifier gives each transaction a score; the threshold turns scores into decisions. Lowering it catches more fraud (higher recall) but also flags more genuine payments (lower precision). The ROC curve plots every threshold at once, so its area, the AUC, measures how well the scores separate the classes and does not depend on the threshold you pick.
Class imbalance is the trap. When fraud is rare, even a small false positive rate on the huge genuine group produces more false alarms than true catches, so precision collapses while the ROC curve and accuracy still look great.
You flag 66 transactions; 44 of them are really fraud, and you miss 6 of the 50 frauds.
A classifier gives each transaction a score; the threshold turns scores into decisions. Lowering it catches more fraud (higher recall) but also flags more genuine payments (lower precision). The ROC curve plots every threshold at once, so its area, the AUC, measures how well the scores separate the classes and does not depend on the threshold you pick.
Class imbalance is the trap. When fraud is rare, even a small false positive rate on the huge genuine group produces more false alarms than true catches, so precision collapses while the ROC curve and accuracy still look great.
Things to try