New research shows why agreement among AI models is a useful signal for human review, but not a substitute for judgment.
More AI models, fewer mistakes?
Imagine asking an AI model whether a law protects people’s data in a particular way. The model says yes and points to a passage in the legislation.
You ask three other models. They agree.
That may feel like four independent checks. But it could also be four versions of the same misunderstanding.
A new study of AI-assisted policy analysis examined exactly this problem. It found that multiple models often missed the same evidence. At the same time, findings produced by only one model were usually wrong. Agreement was nevertheless useful, not because four models necessarily found much more evidence than one thorough model, but because their agreement helped indicate which findings deserved trust, doubt or closer human review.
The implication is practical: the value of using several AI models may lie less in replacing human judgment than in directing it.
What the study tested
The study asked four language models to assess two laws against the 50 core elements of the Health Data Governance Principles:
- China’s 15-page Personal Information Protection Law; and
- Malaysia’s 100-page Personal Data Protection Act.
Each element was assessed across six areas of law, producing 300 possible judgments, or “cells”, for each document. A cell asked whether the law contained at least one provision satisfying a particular requirement within a particular area of law. The task was not to find every relevant passage.
The models received the complete law, the title of each framework element and short descriptions of the six legal areas. They did not receive the framework’s full definition of each element. Each model answered 50 questions per law, with every question covering all six legal areas.
Two human coders independently reviewed 60 sampled cells from each law. Candidate passages identified by the models were presented as unlabeled leads, so the coders did not know which model had proposed a passage or what conclusion any model had reached. The coders were also instructed to search the laws themselves. Because the sample deliberately included more cells on which the models disagreed, the results were weighted to estimate performance across all 300 cells of each law.
The largest blind spots were shared
The usual argument for using several models is that one model will catch what another misses. That works best when their mistakes are different.
In this study, many of the mistakes were shared.
On the Chinese law, the four models combined missed an estimated 22% of the evidence. The most thorough model alone missed approximately 23%. Adding three models therefore produced only a small increase in the amount of evidence found. Had the models missed evidence independently, their combined miss rate would have been close to 1%, rather than 22%.
One example illustrates why.
Article 26 of the Chinese law contains rules governing cameras and identification equipment in public places. All four models cited the page containing the provision, and one model quoted Article 26 for a different requirement. Yet none of their first runs connected it to two framework requirements that the human review found it satisfied.
The models had not simply failed to read the relevant page. They had encountered the provision but failed to connect it to the right questions.
This distinction is important. Giving a model more access to a document does not necessarily solve an error of interpretation or classification.
Additional models were more helpful when the starting model was cautious and reported relatively little evidence. In those cases, adding the other models increased estimated recall by approximately 10 to 27 percentage points. The benefit of a multi-model panel therefore depended heavily on the model used as the starting poin.
Agreement was useful, but not conclusive
Among the human-checked cells across the two laws, 18 of the 19 cells flagged by all four models contained qualifying evidence. By contrast, only 4 of the 26 cells flagged by a single model did.
This is a striking difference, although the sample is too small to treat those figures as universal accuracy rates.
The result suggests a practical role for multi-model agreement: it can help reviewers prioritize their attention.
- Unanimous findings may be suitable for provisional acceptance, subject to spot checks.
- Disputed findings should receive closer human examination.
- Findings produced by only one model should be treated cautiously.
- Questions for which every model reports no evidence still need to be sampled, because shared blind spots will not appear as disagreements.
Agreement is therefore a confidence signal, not proof.
The study included one example in which all four models agreed and all four were wrong. The models treated a security provision in the Malaysian law as evidence for a requirement concerning data quality, availability and accessibility. The human review rejected that interpretation. Protecting data from loss or misuse is not the same as ensuring its quality.
A genuine quotation can still support the wrong answer
Discussions of AI errors often focus on invented facts and fabricated citations. This study identified a quieter problem: real legal passages attributed to the wrong requirement or the wrong area of law.
Most findings produced by only one model passed the study’s quotation check. The text existed in the law, often exactly where the model said it did. What failed was the interpretation.
This means that verifying a quotation is necessary but insufficient.
A reviewer must still ask:
- Does the passage actually do what the framework element requires?
- Does it belong within this particular area of law?
- Is it direct evidence, or does it merely overlap with a related requirement?
The distinction matters in any AI-assisted policy or legal analysis. An accurate quotation can make an incorrect conclusion appear more credible.
The study also tested whether adding more explicit attribution rules to the prompt would solve the problem. The rules improved precision for the model that tended to report too much, but they did not produce a consistent overall improvement for the models that were already more selective. In one case, the stricter instructions reduced recall.
Writing decision rules remains valuable for human reviewers. Giving those rules to an AI model may help, but the effect needs to be tested model by model.
Asking the same model twice can produce different answers
The study also found substantial variation between repeated runs of the same model.
Even when the document, prompt and settings remained unchanged, two runs of one model disagreed on approximately 30% of the cells flagged by either run. Two different models disagreed on approximately 45%. These percentages relate to cells flagged by at least one of the two runs, not to all 300 cells.
A second run even recovered one connection that all four models had missed in their first runs.
That does not establish that repeating one model is better than using an additional model. The study included only two standard runs per model, so it could not determine which approach provides better results for the same cost.
The more immediate lesson is simpler: a single AI response is not necessarily a stable response.
The longer law was not clearly harder
The Malaysian law was approximately 100 pages, compared with 15 pages for the Chinese law. It would be reasonable to expect the longer document to produce more omissions.
The study found no visible loss of recall on the longer law, and models did not produce wrong findings more frequently among cells that contained no evidence. However, the uncertainty around the estimates was substantial, and the comparison involved only two different laws. It therefore cannot show that document length has no effect.
The nature of the task helps explain the result.
Asking whether there is at least one qualifying provision is different from asking a model to identify every relevant provision. For the first task, one valid passage is enough. For the second, the model must search the entire document comprehensively.
The uncited pages in the longer law were also largely front matter, administrative provisions and procedural material. More importantly, the evidence that the models missed was located on pages they had cited for other questions.
Again, the central problem was not simply unread text. It was deciding what the text meant and which requirement it satisfied.
The study did not directly test splitting the laws into smaller sections. Both documents also fit comfortably within the models’ input limits. A document that exceeds a model’s context window, or a task requiring every relevant passage, presents a different problem.
A better workflow for AI-assisted policy analysis
The findings support a workflow in which AI models act as first readers and help direct human attention.
1. Use at least one model that searches broadly
A cautious model may produce a clean-looking result while missing substantial evidence. Including a more thorough model can help establish a higher ceiling for recall.
2. Use agreement to prioritize review
Treat agreement as a graded signal rather than a final verdict. Findings supported by every model may need only spot checks, while disputed and single-model findings deserve closer examination.
3. Audit some unanimous negative results
If every model misses the same evidence, the error will not appear in a disagreement report. Review a sample of cells for which all models found no evidence.
4. Check interpretation, not only quotation accuracy
Confirm that the cited passage genuinely satisfies the specific requirement and belongs in the relevant legal or policy category.
5. Record the decision rules
Document how reviewers distinguish direct evidence from indirect relevance, and how passages are assigned to categories. Apply those rules consistently across documents and reviewers.
6. Verify the inputs before comparing results
Check that every run used the intended document, model and set of framework elements. A technically valid comparison can still be misleading if one run is incomplete or the wrong run has been selected.
These recommendations are practical implications drawn from the study rather than proof of a universally optimal workflow. The research examined two laws, four models and 120 human-checked cells. Larger studies across other policy domains and document types are still needed.
What the study does not establish
The research is deliberately focused.
Both documents were data protection laws read in English. The Chinese law was analyzed through an English translation, while the Chinese text is authoritative. The Malaysian results relate to the Act as reprinted in 2023, before the 2024 amendments introduced provisions including data breach notification, data protection officers and data portability.
Only 60 cells per law were checked by human coders, and some estimates depended considerably on a small number of weighted observations. The coders were shown model-proposed passages as leads, which may have influenced where they looked. Ten of the 120 cells required the author to resolve a disagreement between the two coders using written decision rules.
The study therefore does not establish universal performance rates for AI models. It identifies patterns that should be examined in future applications and larger studies.
The takeaway
Using more AI models does not automatically produce a more complete or more accurate analysis.
Several models can overlook the same provision. A genuine quotation can support the wrong conclusion. A unanimous answer can still be wrong. And the same model can give a different answer when asked again.
Yet multi-model analysis can still be valuable.
Its strongest contribution may be diagnostic: showing reviewers what they can trust provisionally, what they should question and where they need to look more closely.
The most useful question is therefore not simply:
How many models agree?
It is:
What does their agreement tell us to check next?
About the research
This article is based on Shared Misses, Individual Extras: Multi-Model Agreement and Document Length in LLM-Based Policy Evidence Analysis by Ng S. T. Chong, Campus Computing Centre, United Nations University. The research is available through the SSRN paper page.