AI Is Inventing Academic Papers That Don’t Exist — And They’re Being Cited in Real Journals
The proliferation of references to fake articles threatens to undermine the legitimacy of institutional research across the board.
As the fall semester came to a close, Andrew Heiss, an assistant professor in the Department of Public Management and Policy at the Andrew Young School of Policy Studies at Georgia State University, was grading coursework from his students when he noticed something alarming.
As is typical for educators these days, Heiss was following up on citations in papers to make sure that they led to real sources — and weren’t fake references supplied by an AI chatbot. Naturally, he caught some of his pupils using generative artificial intelligence to cheat: not only can the bots help write the text, they can supply alleged supporting evidence if asked to back up claims, attributing findings to previously published articles. But, as with attorneys who have been caught generating briefs with AI because a model offered false legal precedents, students can end up with plausible-sounding footnotes pointing to academic articles and journals that don’t exist.
That in itself wasn’t unusual, however. What Heiss came to realize in the course of vetting these papers was that AI-generated citations have now infested the world of professional scholarship, too. Each time he attempted to track down a bogus source in Google Scholar, he saw that dozens of other published articles had relied on findings from slight variations of the same made-up studies and journals.
“There have been lots of AI-generated articles, and those typically get noticed and retracted quickly,” Heiss tells Rolling Stone. He mentions a paper retracted earlier this month, which discussed the potential to improve autism diagnoses with an AI model and included a nonsensical infographic that was itself created with a text-to-image model. “But this hallucinated journal issue is slightly different,” he says.
That’s because articles which include references to nonexistent research material — the papers that don’t get flagged and retracted for this use of AI, that is — are themselves being cited in other papers, which effectively launders their erroneous citations. This leads to students and academics (and any large language models they may ask for help) identifying those “sources” as reliable without ever confirming their veracity. The more these false citations are unquestioningly repeated from one article to the next, the more the illusion of their authenticity is reinforced. Fake citations have turned into a nightmare for research librarians, who by some estimates are wasting up to 15 percent of their work hours responding to requests for nonexistent records that ChatGPT or Google Gemini alluded to.
Heiss also noticed that the AI-generated notes could be convincing to a reader because they included the names of living academics and titles that closely resemble existing literature. In some cases, he found, the citation led him to an actual author, yet the heading of the article and the journal were both fabricated — they just sounded similar to work the author has published in the past and a real periodical that covers such topics. “The AI-generated things get propagated into other real things, so students see them cited in real things and assume they’re real, and get confused as to why they lose points for using fake sources when other real sources use them,” he says. “Everything looks real and above-board.”
Since LLMs have become commonplace tools, academics have warned that they threaten to undermine our grasp on data by flooding the zone with fraudulent content. The psychologist and cognitive scientist Iris van Rooij has argued that the emergence of AI “slop” across scholarly resources portends nothing less than “the destruction of knowledge.” In July, she and others in related fields signed an open letter calling on universities to resist the hype and marketing in order to “safeguard higher education, critical thinking, expertise, academic freedom, and scientific integrity.” The authors claimed that schools have “coerced” faculty into using AI or allowing it in their classes, and they asked for a more rigorous, comprehensive analysis of whether it can have any useful role in education at all.
Anthony Moser, a software engineer and technologist, was among those who foresaw how chatbots could eventually hollow out educational institutions. “I’m imagining an instructor somewhere making a syllabus with ChatGPT, assigning reading from books that don’t exist,” he wrote in a post on Bluesky in 2023, less than a year after the AI model first came out. “But the students don’t notice, because they are asking ChatGPT to summarize the book or write the essay.” This month, Moser reshared that post, commenting: “I wish it had taken longer for this to become literally true.”
Moser tells Rolling Stone that to even claim LLMs “hallucinate” fictional publications misunderstands the threat they pose to our comprehension of the world, because the term “implies that it’s different from the normal, correct perception of reality.” But the chatbots are “always hallucinating,” he says. “It’s not a malfunction. A predictive model predicts some text, and maybe it’s accurate, maybe it isn’t, but the process is the same either way. To put it another way: LLMs are structurally indifferent to truth.”
“LLMs are pernicious because they’re essentially polluting the information ecosystem upstream,” Moser adds. “Nonexistent citations show up in research that’s sloppy or dishonest, and from there get into other papers and articles that cite them, and papers that cite those, and then it’s in the water,” he says, likening this content to like harmful, long-lasting chemicals: “hard to trace and hard to filter out, even when you’re trying to avoid it.” Moser calls the problem “the entirely foreseeable outcome of deliberate choices,” with those who raised objections “ignored or overruled.”
But AI can’t take all the blame. “Bad research isn’t new,” Moser points out. “LLMs have amplified the problem dramatically, but there was already tremendous pressure to publish and produce, and there were many bad papers using questionable or fake data, because higher education has been organized around the production of knowledge-shaped objects, measured in citations, conferences, and grants.”
Craig Callender, a philosophy professor at the University of California San Diego and president of the Philosophy of Science Association, agrees with that assessment, observing that “the appearance of legitimacy to non-existent journals is like the logical end product of existing trends.” There are already journals, he explains, that accept spurious articles for profit, or biased ghost-written research meant to benefit the industry that produced it. “The ‘swamp’ in scientific publishing is growing,” he says. “Many practices make existing journals [or] articles that aren’t legitimate look legitimate. So the next step to non-existent journals is horrifying but not too surprising.”
Adding AI to the mix means that “swamp” is growing fast, Callender says. “For instance, all of this gets compounded in a nearly irreversible way with AI-assisted Google searches. These searches will only reinforce the appearance that these journals exist, just as they currently reinforce a lot of disinformation.”
All of which contributes to a feeling among researchers that they’re being buried in an avalanche of slop, with limited capacity to sift through it. “It’s been incredibly disheartening for faculty, I think fairly universally, especially as fake content gets accidentally enshrined in public research databases,” says Heiss. “It’s hard to work back up the citation chain to see where claims originated.”
Of course, many aren’t even trying to do that — which is why the phony stuff has been so widely disseminated. It’s almost as if the uncritical and naive adoption of AI has made us more credulous and sapped our critical thinking at the precise moment we should be on guard against its evolving harms. In fact, someone may be toiling away on a (real) study of that phenomenon right now.
▻https://www.rollingstone.com/culture/culture-features/ai-chatbot-journal-research-fake-citations-1235485484
#IA #AI #intelligence_artificielle #édition_scientifique #science #ESR #université #citations #chatbot #références #faux #erreurs
–-
ajouté à la #métaliste sur la #publication_scientifique*:
►https://seenthis.net/messages/1036396
