Coinbase reported on Oct. 7 that newer versions of three major AI model families caught fewer fraudulent payments and a smaller share of fraud value in a historical test of payment screening for its Onramp service, even though the decision policy remained unchanged. The findings, detailed in a benchmark report, challenge the assumption that upgrading a model automatically improves an existing payment screener.

The company's evaluation replayed 16,140 transactions across 7,293 users, including 813 confirmed fraudulent transactions. The cohort covered nine weeks before Coinbase's risk agent rolled out and retained all matured fraud cases while sampling legitimate traffic.

Each candidate model reviewed recent transaction behavior under fixed guidance and the same policy for converting risk classifications into decisions. That design isolated the decision model's behavior within the existing setup, rather than comparing redesigned screening systems. For an operator, that is the upgrade scenario that matters in practice: keeping the screening system intact and changing only the model underneath.

Newer models underperformed in a fixed historical replay

Coinbase compared Opus 4.5 with Opus 5, Sonnet 4.6 with Sonnet 5, and GPT-5.4 with GPT-5.6 (sol). Every newer version posted lower recall, a lower combined precision-and-recall score known as F1, and lower dollar-weighted recall. Recall measures the share of fraud cases a model catches, while dollar-weighted recall measures how much of the total fraud value it catches.

Sonnet's recall fell 22.2 percentage points, and its dollar-weighted recall dropped 22.9 points. Opus's recall declined 0.8 points. Both newer models also showed lower precision, meaning a smaller share of the transactions they flagged as fraud were actually fraudulent.

GPT illustrated why a single improving score can mislead. Its precision rose 11.5 percentage points, but recall fell 20.7 points and dollar-weighted recall fell 21.8 points. Its fraud were more accurate, yet more fraud cases and value escaped detection in the replay.

The replay does not establish customer losses from deploying those versions. Coinbase also said it could identify the regressions without determining their cause. That leaves the mechanism behind the pattern an open question in the published work, and one to watch as follow-up analysis may address it.

Coinbase's earlier online experiment compared adding selective LLM review with the existing models and rules alone. That agent-enabled flow recorded 30% fewer fraudulent transactions and 22% less fraud value; it did not compare newer model versions.

In their limitations section, the SR-Fraud researchers said the proprietary dataset cannot be released, restricting independent replication and generalization. Their related payment-fraud study first appeared Sept. 23 and was revised Sept. 30, before the October blogs.

A separate case for a custom model

In an Oct. 8 disclosure, Coinbase reported that a post-trained Qwen3.5-9B model exceeded Opus 4.5 across four fraud-detection metrics. F1 improved 9.6 percentage points and dollar-weighted recall rose 35.4 points. The company specialized the model using historical fraud outcomes and deterministic rewards that balanced fraudulent and legitimate examples.

Separately, production measurements put median end-to-end LLM-request latency at 0.683 seconds, versus 1.515 seconds for Opus 4.5, a 55% relative reduction. The faster inference and the stronger benchmark detection came from different evaluations.

Read together, the two disclosures frame a practical fork for payment teams: keep pace with frontier version bumps and re-validate each one under their own decision setups, or specialize a smaller model on historical outcomes, as Coinbase did.

For payment providers, Coinbase said, the upgrade question is whether a candidate improves fraud coverage under their actual decision setup. The company recommends testing that configuration first, then evaluating changed prompts or thresholds separately, weighing latency, reliability and cost alongside detection quality.