IBM Research Releases BenchDrift, an Open-Source Tool for Measuring How Prompt Wording Shifts LLM Benchmark Scores
Key Takeaways
- •BenchDrift measures "bidirectional correctness flips," where a model's answer changes from correct to incorrect or the reverse because of rewording alone.
- •IBM Research tested eight models on GSM8K, MMLU, and MATH-Hard and found wording-related drift across all three benchmarks.
- •Weaker models were more likely to benefit from rephrased questions, while stronger models more often lost accuracy when prompts were rewritten.
- •The researchers found that even high-confidence answers can change with small differences in question length or style.
- •IBM released BenchDrift as open source on GitHub with pipeline code, notebooks, and configuration files.
IBM Research has published a paper introducing BenchDrift, an open-source tool designed to quantify how much large language model benchmark scores shift when test prompts are rephrased without any change in meaning. The project puts a hard number on a problem familiar to anyone who has aced an exam only to fail a retake built from slightly reworded questions.
How BenchDrift works
The underlying method is straightforward: take a benchmark question, rewrite it so that its meaning and correct answer are preserved, and check whether the model still answers correctly. BenchDrift applies this process systematically across four distinct axes of variation: linguistic, referential, pragmatic, and structural.
The tool generates rephrased variations at scale and then tracks what the researchers call "bidirectional correctness flips" — instances in which a model's answer switches from correct to incorrect, or from incorrect to correct, based solely on phrasing.
The research team, led by Shailja Thakur, Sungeun An, Chad DeLuca, and Hima Patel at IBM Research, evaluated eight models across three widely used benchmarks: GSM8K (grade-school math), MMLU (massive multitask language understanding), and MATH-Hard (advanced mathematical reasoning). According to the paper, drift appeared across the board, underscoring that a benchmark score can be sensitive to wording even when the underlying task is unchanged.
Stronger models, bigger problems
The pattern was not uniform across capability levels. Weaker models tended to benefit from rephrased questions, picking up correct answers they had previously missed. Stronger models showed the opposite behavior: they lost significantly more accuracy when questions were reworded, even though the underlying meaning remained identical.
The researchers also found that high-confidence answers can deteriorate with minor changes in question length or style, indicating that even when a model appears certain about its answer, that certainty may be anchored to specific phrasing rather than actual comprehension.
Why benchmark fragility matters beyond the lab
BenchDrift adds to a growing body of evidence that current LLM evaluation infrastructure has fundamental weaknesses. IBM's own prior work in this area includes the projects ITBench and BenchmarkCards, both aimed at improving transparency in AI assessment. BenchDrift extends that effort by providing a quantitative framework for what the researchers call the "wording effect" — the measurable gap between what a benchmark score claims and what it actually demonstrates.
That matters for anyone using benchmark results to compare models, track progress over time, or interpret small score differences, because phrasing sensitivity can make those comparisons less stable than they first appear. The tool has been released as open source through the GitHub repository IBM/BenchDrift. The release includes the full pipeline code, Jupyter notebooks, and configuration files needed to generate question variations, validate that those variations preserve meaning, and detect correctness drift.
