The following is an opinion column by Samuel Vance-Law, head of research at the Decentralization Research Center, arguing that if the White House AI accord's audits are going to be worth anything, auditors should be able to choose what they test.

Last week, six of the most powerful artificial intelligence companies in the world promised the White House they would let outsiders check their work.

On Sept. 29, Nvidia, OpenAI, Anthropic, Google, Meta and xAI signed an accord covering the most advanced AI models, an agreement U.S. President Donald Trump called "morally binding". Each company has promised it will "partner with an independent external auditor or evaluator" and have independent board members review the auditors' reports and make sure problems get fixed.

Outside audits are a good idea, and Vance-Law welcomes their inclusion. But the accord, he writes, says nothing about who chooses the auditors, what they can examine or whether any findings are made public — and there is no penalty for ignoring it. As it stands, each company picks its own auditor and decides what that auditor gets to look at.

The arrangement echoes a cautionary episode from corporate history. In 2000, Enron paid the accounting firm Arthur Andersen — once part of the so-called "Big Five" of international accounting firms, now the "Big Four" after Arthur Andersen's implosion — $25 million to audit its books and another $27 million for tax and consulting work. Within two years, both firms had collapsed. Congress responded with the Sarbanes-Oxley Act of 2002, which took the choice of auditor away from management — the very authority the new accord leaves with each AI company — stopped auditors from selling most other services to the companies they audit, and set up a public board to inspect the auditors themselves.

Andersen failed not because of a deficit in expertise, but because it lacked the incentive to look hard at a client paying it $52 million in a single year.

Voluntary pledges in the AI industry do not have a strong track record, either. In 2023, the leading AI companies made a set of voluntary commitments to the White House, including letting outside experts test their systems before release. When researchers at Stanford, Harvard and Brown went back and scored 16 companies against those commitments, the average came to 53%. On the promise to help outsiders find and report problems, eight of the 16 scored zero.

Even when outside testing does happen, the developer sets the terms. METR, a nonprofit that tests AI models, wrote that its testing of OpenAI's o3 model before release was "conducted over only three weeks," with "limited access to information about the model" and no view of the model's step-by-step reasoning. To be fair to OpenAI, it allowed the testing to take place at all — but the episode shows how much rides on what a company chooses to share.

A trained AI model is essentially a very large file of numbers, often costing more than $100 million to produce, and whoever has a copy has the model. Handing copies to outsiders is a security risk. Until recently, that left testers with an awkward choice, which Google DeepMind has described well: either the testers hand over their questions, so the company sees the exam in advance, or the company hands over its model.

In August, DeepMind showed a way around that dilemma. Working with Singapore's AI Safety Institute and three other outside groups, it ran what it called the world's first double-blind test of an AI model, on the same principle as a double-blind drug trial. The test ran on sealed computer hardware, so the outside testers never saw the model and Google never saw their questions. Google says each side can check mathematically that its material stayed private.

The approach also moves the trust onto the hardware itself. Last year, researchers broke into this kind of sealed hardware with less than $1,000 of equipment, although they needed physical access to the server to do it. That is worth taking seriously, but it is a far smaller problem than "we can't let anyone near the model," and it can be managed by being careful about who controls the machines.

The argument draws on the report "Alternatives to Concentrated AI", published this week by the Decentralization Research Center. For the report, Vance-Law and co-author Kyle Bligen interviewed 36 people and mapped nearly 30 working alternatives to concentrated control over AI. Its main finding: outsiders can now scrutinize advanced AI without taking possession of the model or its data.

If the accord's audits are going to be worth anything, Vance-Law argues, auditors should choose what they test, now that they can do so without taking a copy of the model. Companies should also say who their auditors are and what the audits covered, even if detailed findings stay private — public companies have named their auditors for decades without giving away trade secrets. And the board committees reading these reports should answer to someone outside the company. Sarbanes-Oxley created an inspector for the inspectors. The accord has nothing like it, and Congress should decide what fills that gap.

Note: The views expressed in this column are those of the author and do not necessarily reflect those of CoinDesk, Inc. or its owners and affiliates.