Questionable Research with High-Profile Attention
A Case for Funders’ Due Diligence in Research
There’s a fundamental knot we need to untangle in the world of research: the factors that determine career advancement often clash with the pursuit of making reliable science available to the community. When it comes to applying for grants and securing funding, data like publication volume and journal or attention metrics risk only telling part of a story that would look very different if integrity indicators (i.e., trust markers) were involved.
Pressure to publish drives a surge in research outputs, which doesn’t necessarily result in more knowledge the world can benefit from: in fact, the larger the volume, the greater the risk of polluting scientific literature with unverifiable results and low-quality work. As we’ve seen before, this pollution feeds global disinformation. But there are more tangible consequences, too: when scarce funding supports less innovative or even fraudulent work, the system actively squanders the opportunity to invest in rigorous, ethical science.
The Cost of Manipulation
Demonstrating impact is key to funding opportunities, and attention is often the gold standard for showing that research can influence conversations and decisions with real-world consequences. Research, Attention and Manipulation raised questions about the papers that make it into Wikipedia and news articles. Mentions in policy documents and clinical guidelines stand out even more because they provide a clear pathway from research to real-world impact.
Citations in policy and clinical guidelines signal that the research is influencing actors who can make a real difference: national governments, major intergovernmental organizations, key opinion leaders in medical practice, and more. This kind of attention may take longer to surface than conversations on news and social media platforms, but its impact is far more lasting. Funders, institutions, and businesses actively track it because identifying researchers whose work is attracting policymakers’ attention can uncover collaboration opportunities and provide evidence of returns on their investments in research.
As our previous article, From Nefarious Networks to Legitimate Funding, shows, authors tied to questionable research networks apply for the same opportunities as researchers with legitimate aims and robust practices. The hallmarks of unethical activities can be sophisticated: just like editors of academic research may not recognise them while reviewing papers, the authors of policy and guideline documents from reputable organisations may not spot them while looking for relevant literature to cite.
Questionable Research, Real-World Consequences
To get a sense of whether questionable research can influence impactful documents like policy and clinical guidelines, we’ve used Altmetric, which tracks online attention to research across a wide range of sources.
This post is an exploration, not a conclusion. We began our exploration from authors with ties to a fictional network (Pharmakon Neuroscience Network) and legitimate grants on their CVs. Did any of their papers catch the attention of policy and clinical guideline developers? And was that attention an endorsement or an alarm bell?
We analysed 82 mentions (54 in policy documents, 28 in clinical guidelines) gained by five authors within the fictional network. And here’s what we found. Some major issuing organisations immediately stand out: large government bodies, major medical associations, and international organisations like the Food and Agriculture Organization of the UN, the European Union, and the OECD.
The idea that questionable papers might make it into documents that drive international policy raises questions about the trust markers authors can rely on. Even policymakers who are scrupulous in selecting quotes and citing verifiable sources might inadvertently rely on a flawed paper because of its high visibility. The key takeaway is the attention questionable articles and their authors gain—and the career advantage that comes with it.
The most interesting mentions came from two separate 2024 documents (issued by the US Centers for Medicare & Medicaid Services and the Health and Human Services Department), both calling out one paper (now retracted) for research misconduct. The paper’s mentions from blogs shows Retraction Watch reported on the case, too.
The author at the heart of this case has the lion’s share of clinical guideline mentions in our analysis: 26 out of 28. Ten of these are for one single 2013 paper, which is itself a clinical guideline.
This leaves us with:
- A 2013 clinical practice guideline, gaining visibility as recently as 2024, is influencing recommendations made by major medical associations to inform everyday practice at a global level.
- A misconduct case reported in 2024, surrounding a grant-funded paper (also published in 2013).
- A further 16 papers by the same author, published between 2000-2024, mentioned 31 times in policy and clinical guidelines. None have been retracted, though RetractionWatch points to further issues with other works by the author.
- Six grants were received by the author between 2013 (when the fraudulent paper was published) and 2024 (when the misconduct case was reported).
If questionable papers are a means to game the system and generate powerful attention, that data can be used to build a strong-looking narrative on a CV, giving candidates with illegitimate credentials a serious advantage over researchers with genuine bodies of work. In a system that provides incentives to inflate publication volume and misrepresent contributions, even the best-intentioned impact metrics can become a tool to spread unreliable research.
Building a Sustainable Ecosystem for Integrity
No system of checks can substitute for a culture of trust: fraudulent practices are continuously evolving. The solution is a structural shift that moves responsibility beyond individual choices, and relies on collaboration across fields of research and professional remits. Institutions, funders, and publishers all have a part to play.
The core idea is that financial investment goes hand-in-hand with an investment in quality, and an increased focus on scientific content and rigour. While bibliometric indicators are well-established and recognized in the world of research, and data about different sources of attention can help build a comprehensive impact narrative, quantitative metrics cannot replace expert human judgement: the two should go hand in hand.
Earlier this year, at the 1st National Congress on Research Integrity in Rome (Italy), members of the research community suggested concrete ways to address this balance:
- Building evaluation criteria focused on research integrity principles into funding application requirements.
- Evaluating a limited number of publications, and requiring candidates to submit a statement describing their individual impact and contribution, to move away from rewarding sheer volume.
- Acknowledging authors’ roles by requiring the candidate to appear as first, last, or corresponding author in a minimal percentage of these publications.
- Taking a holistic view of scientific output, paying particular attention to candidates with an incongruously high, suspicious volume of papers.
- Setting firm standards when evaluating CVs, by ignoring or downgrading publications in predatory journals, and rewarding work placed with publishers that actively promote good practices (e.g., raw data availability; strong plagiarism and self-plagiarism checks; publication of reviewer comments and author responses).
Ultimately, this is about creating an environment where good actors feel supported, not penalized, and making sure funding goes where it belongs: to genuinely innovative, trustworthy research.
Disclosure
I work at Digital Science, and looking after Altmetric attention sources is part of my role. This post isn’t about promoting a product; it’s the continuation of a conversation that started when Leslie asked Could I use Altmetric to look at scholarly attention forensically? Taking my product management hat off to spend some time wondering where a research integrity professional might dig is fascinating, and it’s all for the critical purpose of understanding how trust in science is built or broken.