Home / When References Cite Nothing

When References Cite Nothing

⋅

Ng Chong

A peer-reviewed PNAS study found references that could not be located in about 1 in 435 physics, chemistry, materials science, engineering and computing papers from early 2026, up from 1 in 1,600 in 2023. That puts citation reliability in question. The study measures how common the problem is and how fast it is growing; it does not establish why, and it does not show that AI tools are the cause.

What the audit found

The Columbia-led team, extending its earlier audit of the medical literature, drew a stratified random sample of 500,000 peer-reviewed journal papers, with no preprints, published from January 2023 through March 2026 and indexed in OpenAlex. Its pipeline extracted 26.2 million references and kept the 21.6 million (about 82%) that carried a resolvable DOI. It labeled 1,135 of them, across 619 papers, as fabricated, an operational label for a claimed title that could not be located in any of seven bibliographic and open-web sources.

The share of affected papers rose from 6.3 per 10,000 in 2023 (1 in 1,600) to 15.4 per 10,000 in 2025 (1 in 650) and 23.0 per 10,000 in the first quarter of 2026 (1 in 435), a quarter the paper marks as an incomplete period. That is about a 3.7-fold increase in three years, based on the reported rates.

Rates differ roughly sixfold across fields over the whole period, from 25.6 per 10,000 papers in physics to 4.3 in chemistry. The paper reports that fabricated references appeared in all five fields and rose steadily over the study period; its figure puts physics at 48.6 per 10,000 in the first quarter of 2026.

Topaz et al., PNAS 2026, Table 1 · pooled rates across the 500,000-paper sample, Jan 2023 to Mar 2026

Chemistry is a conservative floor, because OpenAlex files some analytical and physical-chemistry journals under biochemistry, so the true spread may be narrower. The authors suggest several mechanisms that could produce the gradient and probably work in combination: differences in submission-to-publication lag, reliance on preprints, uptake of language models and journal selectivity. They say further work is needed to explain the differences between fields.

Of the 619 affected papers, 565 (91.3%) held only one or two fabricated references. But 27 papers held five or more, and together they accounted for more than a third of all flagged references. Scaling each field’s rate to roughly 7 million OpenAlex papers in these fields, the authors estimate that about 8,500 published papers carry at least one fabricated reference.

How to read the numbers

Unlocatable is not the same as proven fake. A reference was labeled fabricated when its claimed title could not be found in any of seven bibliographic and open-web sources. The paper stresses that this is an operational label that describes verifiability, not author intent. Two independent experts, masked to the pipeline’s output, reviewed a random sample of 400 flagged candidates and confirmed 85.8% as fabricated (95% CI 82.0% to 88.8%), which leaves roughly one flag in seven unconfirmed.

The check is detection-limited and partly automated. The 18% of references without a resolvable DOI were excluded and recall was not estimated, so the authors say the reported prevalence could be biased in either direction. A language-model screening step (Claude 3.5 Haiku, an Anthropic model) separated fabrication candidates from formatting differences. The pooled rate was 12.4 per 10,000 papers, about 0.12%, so the typical paper is unaffected.

The study counts the problem; it does not explain it. The authors say the temporal pattern is compatible with multiple causes and does not establish language-model involvement. They name paper mills, other publication misconduct and uncritical use of AI writing tools as ways fabricated references can arise, and note that submission-to-publication intervals commonly span several months. The timing still invites the AI hypothesis, and the team’s September preprint, not yet peer reviewed and described in the press release, reports that 26 language models fabricated 55% of the medical references they were asked to supply. That makes AI a plausible mechanism, not a demonstrated one for these papers.

It does not show the damage. One or two fabricated references may reflect an isolated citation error, and the authors say the presence of one does not by itself determine whether correction or retraction is warranted. They call for research on the post-publication fate of affected papers, including whether they are cited, corrected or absorbed into evidence syntheses. The team’s next project, on medical papers, will have experts rate how much fabricated references affect the papers that cite them.

What it means for citation reliability

A citation is a promise, and the filter is leaky. Lead author Maxim Topaz describes a reference as “a promise to the reader that a source exists.” Every paper in the sample was published in a peer-reviewed journal, so these references got through. Retraction records say little yet: 617 of the 619 papers (99.7%) had no recorded retraction, correction or expression of concern, and only one of the two retraction notices mentioned references to nonexistent papers. The authors call this a snapshot that does not show what editors or reviewers noticed before publication. A companion medical study found fabricated references about twice as common in medRxiv preprints as in peer-reviewed articles (37.2 versus 16.4 per 10,000), which fits review filtering some of them, though preprints and journal articles differ in other ways too.

Readers can no longer assume the trail leads somewhere. Following a reference to check a claim is how science stays self-correcting. A reference that leads nowhere pushes the verification work onto the reader, and anything built on the citation graph, from literature reviews to bibliometrics, inherits the noise.

Severity is lopsided, so the response should be too. One or two unverifiable references may reflect an isolated citation error. The 27 papers with five or more held more than a third of all flagged references and, in the authors’ view, warrant priority review. They add that a paper with a single fabricated reference differs in kind from one with five or more, and that each paper’s fate needs case-specific publisher judgment.

The overall rate is low, but the direction is not. The pooled rate was 12.4 per 10,000 papers, about 0.12%, so there is no case for writing off the literature. Yet the first-quarter 2026 rate, from an incomplete period, was about 3.7 times the 2023 rate. Trend lines are not forecasts, but a rise like that is reason enough to keep measuring.

AI’s role is a hypothesis, and the fix does not depend on it. The authors say the timing does not establish language-model involvement. The practical response works either way, because a reference check does not care who or what produced the bad citation.

What to do next

We recommend three steps, as the study’s authors do, and the first is the cheapest:

  • Journals: check references automatically at submission. Topaz calls this one of the few integrity problems that can be checked by machine before a paper is published.
  • Publishers and archives: screen the already-published record retroactively.
  • Indexing services and integrity databases: record fabricated references as their own category, separate from plagiarism and data falsification, so they can be tracked and acted on. 

Three further steps follow from the findings, and they are our suggestions rather than the study’s:

  • Editors: triage by severity. Query authors over a single unverifiable reference; investigate papers with clusters of them.
  • Authors: confirm every reference against the source you actually read, especially in any list drafted or tidied with a tool.
  • Readers: open the references a claim rests on before building on it and treat one you cannot locate as a question for the authors.

One further point goes beyond the study. It does not address AI-assisted peer review, but its findings should also inform how AI-powered or agentic review systems are built. These systems should not be designed only to summarize a manuscript or comment on the quality of its argument. They should treat reference verification as a distinct, auditable task: resolving each citation, comparing the claimed metadata with the retrieved record, identifying which claims depend on it, and directing uncertain or higher-risk cases to a human reviewer. 

The study used several steps to identify potentially fabricated references. These included matching DOIs, applying pattern filters, using a language model for screening, searching seven different sources, and asking experts to review the results. The findings also suggest a way to prioritize concerns. The authors explain that one or two fabricated references may be isolated citation errors, while papers containing five or more should receive closer attention. In other words, a single mismatch may simply need clarification, but several unverifiable references could indicate a more serious problem. AI-assisted peer review should therefore help reviewers check and follow the evidence behind a paper, rather than simply generate more fluent commentary.

One caution applies throughout: an automated flag is a signal to investigate, not proof of fabrication. In the study’s expert review of 400 flagged references, roughly 1 in 7 was not confirmed as fabricated, showing that automated checks can produce false positives. The authors stress that these tools can identify references that warrant closer scrutiny, but deciding what action to take on an individual paper requires case-by-case judgment by its publisher. A flag should therefore prompt human review, not determine the outcome. The aim is to make checking references routine for journals and readers alike, not a rare, heroic effort.

Sources