One researcher asked ChatGPT for input on roughly 5,000 publications because to the “increasing interest” in using AI chatbots for peer review.
Unexpected effects are being felt by researchers as a result of the declining number of scientific peer reviewers. Researchers are progressively using ChatGPT to get feedback on their work after receiving insufficient input from peers in the scientific community. They regard ChatGPT to be more useful than traditional human feedback.
Academics must undergo peer review before their work is accepted for publication in reputable scientific publications. Throughout this procedure, experts in the same field review the submitted work and compare it to theories and previously published articles to strengthen—or utterly refute—the authors’ conclusions. Peer review is a long-standing requirement for academic publishing since it is commonly believed to improve researchers’ work and encourage creativity.
But, there is one small issue: The majority of publications rely on volunteer reviewers. This means that only academics who are able to conduct unpaid work can participate in the peer review process, excluding those who lack the resources (or motivation) to do so. This methodology has produced a comparatively tiny pool of peer reviewers—which naturally grows even smaller when researchers require someone from their own field—whether it’s because journals operate on razor-thin margins or because they’re trying to protect the integrity of each review. Peer reviewers are in short supply as a resource.
James Zou, an assistant professor of biomedical data analytics at Stanford University, was interested in seeing if AI chatbots might reasonably take the place of people in the peer review process. Around 5,000 peer-reviewed research articles from the International Conference on Learning Representations and the Nature publishing family were compiled by him and his colleagues (ICLR). Zou’s team compared the chatbot’s responses with those previously given by humans after feeding the PDFs to ChatGPT. On the ICLR publications, there was a little higher overlap between the issues raised by human reviewers and those made by ChatGPT in about one-third of the cases.
Reaching out to the numerous scientists whose publications had been referenced in his experiment was a priority for Zou. More than half (57.4%) of these researchers judged ChatGPT’s input to be “useful” or “very helpful,” according to a poll created by Zou’s team. A sizable majority (82.4%) even said ChatGPT’s evaluations were more valuable than “at least some” reviews from real people. However, the study did not take into account if ChatGPT’s feedback was occasionally inaccurate, which is kind of crucial in the era of AI chatbot misinformation.
Ironically, Zou’s research is only published on the arXiv since it is currently being peer reviewed, but it offers an intriguing conundrum. Is ChatGPT genuinely more effective at providing feedback to researchers than their own peers are? Or is the current non-compensatory peer review paradigm discouraging knowledgeable reviewers from contributing valuable insight?