Yunita Anwar提出会计师事务所应用AI基准结果的本地试点方法
What AI benchmarks can teach accounting firms
Yunita Anwar建议会计师事务所将AI基准结果转化为本地试点,通过比较同等信息条件与拟议工作条件、预先定义成功标准,判断是否扩大应用。
将两项AI基准研究转化为任务、信息条件和成功标准三维试点框架,便于事务所验证本地场景中的可靠性与复核价值。
A recent article in Accounting Todaycompares two studies of AI performance from Mercor and Digits on accounting tasks. It gives practitioners a useful starting point for a more practical conversation: How can a firm turn benchmark results into evidence for its own decisions?
Processing Content
My recommendation is to treat each study as a guide to the next question worth testing. A benchmark can help a team identify a promising use, and a carefully designed local trial can then establish what that use requires, where it helps, and what still needs attention.
Consider the two studies in that light. Mercor's study involved 12 licensed CPAs and four month-end tasks, split between AI-assisted and unassisted conditions. The tested Claude Opus 5 agent achieved perfect rubric scores across 20 runs, while unassisted accountants averaged about 37%. The researchers also describe constraints involving context, specialization and opportunities to ask questions.
The Digits study, which examined transaction categorization, reports a much narrower gap. Its best one-shot model scored 79.5%, against a human baseline of 79.1%, a difference within the paper's stated sampling error. Its stronger agent results, however, involved access to booking history that the human comparison group did not receive.
Those details are what make the research useful. Read together, they invite firms to examine three things: the task being measured, the information available, and the definition of a successful result. I would use those three questions to structure a small pilot before making a broader operational decision.
The first question calls for a task that can be described precisely. A firm could, for example, ask a system to prepare a proposed expense-coding schedule from an approved set of invoices, receipts and account definitions. The trial should require a source reference for every proposed entry and a separate list of unresolved items, and it should stay away from live posting until its results have been assessed.
Framed this way, the pilot asks a manageable question: Can this particular arrangement produce a schedule that the team can inspect and use? Verification becomes visible as well: A reviewer should be able to trace an entry to the supporting document, see the chosen category and understand why an item was left unresolved.
Once the task is defined, the second question concerns the information each participant will receive. Here I would run two comparisons. The first would give the preparer and the AI system the same approved materials. The second would test the proposed working arrangement, including whatever access to prior records, explanations and follow-up questions the firm intends to provide.
Keeping those results separate matters. The first comparison asks what the system contributes under matched information, while the second asks whether the proposed process works with its intended resources. Both questions deserve an answer, and reporting the conditions alongside the scores makes it easier to decide what to change next.
The third question, what counts as success, should be settled before anyone inspects the results. In the coding trial, success might mean a supported category, an accurate amount, a usable source reference and an appropriate decision to flag missing evidence. Scoring these dimensions separately keeps a high overall average from hiding a weakness the firm considers consequential.
Those scores are only as informative as the cases behind them. The test set should mix straightforward items with deliberate exceptions, such as an unfamiliar vendor, an incomplete description and a receipt that conflicts with the proposed treatment. The firm should decide in advance where asking a question is the best response, so that a justified request for clarification earns credit and no participant feels pressed to fill every blank.
The same logic applies to review, which deserves its own test. The Mercor paper reports that, on these tasks, assisted accountants were slightly less accurate and took longer than the AI agent alone. That result raises a practical question: what exactly should a reviewer add to the work?
To find out, the pilot should preserve the initial answer and record each change, its reason and its supporting evidence. The firm can then examine corrections that improved the output, changes that made it worse and problems the reviewer left untouched. This would give the team something more useful than a count of approvals: an account of where review contributes and where it needs redesign.
That record can also shape training. I would ask staff to explain why they accepted a recommendation, what evidence would change their conclusion and which unresolved issue belongs with a more experienced colleague. An attractive answer should have to survive a clear explanation, and the same expectation should apply to the reviewer's changes.
Refining the process and assessing it should remain separate steps. One set of cases can improve instructions, documentation and review procedures; a second set, kept out of that work, should then test the revised process. Recording the model, settings, materials and date shows the team what produced the result.
It also helps to agree on a stopping rule in advance. If a test reveals an error the team cannot reliably detect or explain, that use should pause while the cause is investigated, with a named person responsible for the decision. A pilot should make the limits of the proposed process as clear as its promising results.
The final step before expansion is to write down the decision the trial supports. Perhaps the system is suitable for proposing entries in a defined category, while unusual items continue through a separate route. Perhaps the next step is improving the supplied documentation. Even a narrowly stated conclusion can be a valuable operational advance.
In that sense, this approach builds on the practical opportunity in Accounting Today's article. Firms can welcome strong AI results and still ask specific questions about how to put them to work. The aim is to learn which combination of technology, information and professional review produces dependable work, and then to expand on the strength of that evidence.

Assistant professor of accounting
Yunita Anwar, Ph.D., is an assistant professor of accounting at the Shenandoah University School of Business in Winchester, Virginia. Read full bio
来源:Accounting Today(美国会计行业媒体) · accountingtoday.com