Coinbase has published findings indicating that the latest generation of frontier AI models performs worse than its predecessors at detecting fraudulent transactions — a result that has prompted the cryptocurrency exchange to steer its risk-control operations toward open-weight models refined with domain-specific post-training.

The findings appeared on October 7 in the second installment of Coinbase's “Owning Intelligence” series, in which the company described evaluating six large language models on a fixed benchmark of 16,140 transactions spanning 7,293 users. The dataset was drawn from nine weeks of production data collected prior to the deployment of Coinbase's LLM-powered risk agent, ensuring that the agent's own interventions would not bias observable fraud outcomes. The cohort contained 813 confirmed fraudulent transactions. Because the comparison used a defined historical dataset and a specific evaluation setup, its results speak to the models' performance in that benchmark rather than establishing that newer models perform worse across every fraud-detection application.

Newer Models Regress Across All Three Pairs

The evaluation compared Opus 4.5 against Opus 5, Sonnet 4.6 against Sonnet 5, and GPT-5.4 against GPT-5.6. Across all three pairs, the newer versions recorded lower recall, lower F1 scores, and lower dollar-weighted recall — the share of total fraudulent transaction value successfully identified, a metric Coinbase considers particularly important for payment risk.

The magnitude of the regression varied considerably. Sonnet 5's recall fell by 22.2 percentage points relative to Sonnet 4.6, with dollar-weighted recall dropping 22.9 points; because both precision and recall declined, the model showed a weaker overall ability to distinguish fraud from legitimate activity rather than a simple trade-off between the two. Opus 5 showed a more modest recall decline of 0.8 points. GPT-5.6 followed a different pattern: precision improved by 11.5 points, but recall fell by 20.7 points and dollar-weighted recall by 21.8 — behavior consistent with more conservative classification.

Announcing the results on X on October 12, the Coinbase Developer Platform wrote:

As we showed in part 1 of our Owning Intelligence series, contextual LLMs can strengthen an existing risk stack. But which model should power the agent? Will upgrading to a frontier model improve its decisions? The short answer is no. More in the thread ↓ pic.twitter.com/fgG4jGCdEL

— Coinbase Developer Platform (@CoinbaseDev) October 12, 2026

(Post on X)

Implications for Model Selection and Coinbase's Response

Coinbase cautioned that its benchmark demonstrates a regression without establishing its cause, noting that without access to providers' training data or objectives, any explanation remains speculative. Plausible factors include shifts in decision boundaries and prompt behavior, trade-offs introduced by post-training processes, and changes in training-data composition and task priorities.

The findings reshape how the company approaches model upgrades. Rather than asking which release is newest, Coinbase argues that teams should evaluate whether a candidate improves the overall trade-off between precision and recall, whether observed differences stem from the model itself or from changes to surrounding prompts and decision policies, and whether the model meets the latency, reliability, and cost requirements of live payment flows. In practice, that makes model selection an operational evaluation problem: the relevant comparison includes both detection quality and the constraints of the payment system in which the model runs.

A Shift Toward Model Ownership

The results motivated a strategic move toward model ownership. Coinbase has shifted the risk-control agent for its fiat deposit service, Onramp, toward open-weight models and domain-specific post-training. According to the series' third installment, the company post-trained Qwen3.5-9B using SkyRL on Anyscale, producing a model that outperformed Opus 4.5 on all four fraud-detection metrics while reducing end-to-end serving latency by 55%.

The approach builds on earlier results: in Part 1 of the series, Coinbase reported that layering an LLM review agent onto its existing machine-learning model and rules engine — deployed without replacing the existing risk stack — reduced fraudulent transactions by 30% and fraud value by 22% in an online A/B experiment. Together, the benchmark and A/B results point to two separate evaluation needs: comparing models under a controlled test and measuring how an integrated risk system performs in live use.

Source: Metaverse Post