In a controlled experiment testing the effectiveness of AI-powered peer review, researcher Paul Litvak collaborated with Anthropic's Claude to plant 100 known errors across 10 open-access psychology papers, then ran the modified papers through multiple frontier AI models and two commercial AI review tools. The experiment speaks to a growing interest in AI-assisted review workflows across academic publishing, where overstretched reviewers and rising submission volumes have intensified debates about quality control.

The results revealed significant variation in performance. The best single system identified 71 out of 100 planted errors, while the worst-performing system caught only 30. However, pooling every system's output raised the detection rate to 93 out of 100 errors — a substantial improvement driven by the fact that different models are only partly correlated in the errors they detect. The gains from pooling are consistent with a long-standing principle in machine learning: combining diverse classifiers often outperforms any individual model because uncorrelated errors cancel out. Litvak emphasized that ensembling multiple models is a powerful strategy for identifying issues in academic papers and encouraged researchers to check their manuscripts against several models.

Seven errors eluded every system tested. All seven were omissions — cases where information had been deleted from a paper rather than new mistakes inserted — suggesting that AI tools remain better at detecting fabricated or altered content than spotting missing material. The distinction matters for psychology in particular, a field that has contended with a widely documented reproducibility crisis, where omitted details about methods, analyses, or data can undermine the ability of other researchers to replicate findings.

Among the tools evaluated, Refine.ink contributed more unique catches than any other single system, though Litvak noted it is expensive to use, raising questions about how cost barriers may shape which researchers and institutions can access the most effective AI-assisted review.

Litvak acknowledged two limitations of the study: he did not measure false positive rates, and the distribution of planted errors may not reflect the distribution of errors found in real-world papers. The absence of false-positive data is notable because high false-alarm rates could add to, rather than relieve, reviewer workload if researchers must manually sift through large numbers of flagged but incorrect critiques.

To support further research, Litvak made the papers, planted errors, model outputs, and the full experiment log publicly available on GitHub: https://github.com/Dawes-Institute/ai-peer-review-benchmark. He expressed hope that others will build on the work to create a comprehensive evaluation benchmark across academic disciplines.

The full write-up by Paul Litvak is available here. Notably, the experiment did not use the most recent generation of AI models, suggesting that newer systems may achieve even higher detection rates.

The findings were highlighted on Marginal Revolution.