Guest article by Lisa Hutchinson, former Chief Editor of Nature Reviews Clinical Oncology

Peer review is a system used by scientists and clinicians to decide which primary research or secondary (review/commentary) content is worthy of being published. In theory it is a process of cumulative, collective knowledge that through publications is cascaded from experts to readers. In biomedical research, peer review is intended to provide independent scrutiny by other qualified experts (or peers) before the information is made public, and considered an extension of trust in scientific integrity. It assesses validity, significance, originality or novelty, overall new or confirmatory findings and implications for current or future research endeavours. When optimal, peer review can reduce unwarranted bias towards particular products or companies, and prevent erroneous or inaccurate findings or interpretations, as well as data falsification, plagiarism, and concerning conflicts of interest. So, is ‘obligatory’ peer review trustworthy and does it represent a seal of quality as infamously purported? Unfortunately, it does not. Peer review is time-consuming, arbitrary, and rarely improves the quality of science. It represents more a ‘consensus’ of opinions that stifles creativity and ingenuity, and can be weaponized to usher in censorship. Critics of peer review comment “the purpose of peer review is not to enhance the integrity or reliability of academic publications. Peer review studies turn out to be wrong all the time. It is rather one of many mechanisms, via which academics aim to police their own discourse and exclude outside ideas”. A Cochrane systematic review of 28 studies revealed little empirical evidence in support of peer review as a mechanism to ensure quality of biomedical research.

Richard Horton, the Editor-in-Chief of the Lancet lamented “The mistake of course is to have thought that peer review was more than a crude means of discovering acceptability—not the validity—of a new finding. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish and frequently wrong.” We assume peer review helps to increase the integrity of science, but it rarely achieves these altruistic aims. Moreover, selecting the most appropriate ‘experts’ is highly subjective. In extreme cases, data can be manipulated and referees have been known to ‘scoop’ or steal new data and publish the findings as their own. It is not uncommon for referees to ask for a laundry list of new experiments, which can take months – at great taxpayer expense – to complete but provide little relevance to the original remit of the research. One aim of such nefarious activity is to delay the publication because of competing work, which is unethical conduct that threatens the reputation of biomedical research. A combination of poor peer review and an obsession with ‘research integrity’ and diversity, equity and inclusion (DEI) conformity rather than sound science undermines the entire process. A particularly frustrating aspect is that editorial administrators often ‘hold the fort’ sending authors generic template letters “your manuscript is being assessed and we will be in touch soon”, providing no transparency for how a decision is reached, which is unacceptable. In a recent Twitter exchange, a publisher who invited an expert opined he should consider it because “you might want to publish your work and have us send it to those referees who would like to see it published. Or you might want to see what your colleagues are working on before it’s published given that you have worked in the same area”. This response clearly has disturbing implications. 

To recap, there are three types of peer review: single blind, double blind (rare) and open peer review, with the latter becoming more popular owing to the growing concerns of accessibility and openness of research. In the single-blind system – still the most prevalent system – referees know the identity of the authors but the authors do not know the referees identity. In this system, the referees’ reports are not published alongside the final accepted article, so there is no way to decipher how much the peer-review process has improved an article. Referees might agree to review an article more quickly and might be more candid in their criticisms of a manuscript, including any known author conflict of interests; however, there is an imbalance in terms of baseline identity disclosure, and a lack of accountability, some reviewers are less critical or might abuse of the system (see later). 

In the double-blind system, the referees do not know the authors’ identity and vice versa. This is a fairer system as it reduces publication bias, as the research is assessed on its own merits and not influenced by the identity of the author or their affiliation. Nevertheless, referees may decline to peer review anonymous work, and important details that are concealed through anonymity might prevent certain conflicts from being disclosed, as it might be hard to determine the geographic relevance of findings if the author affiliations are not available. Of concern, one study showed how ‘status bias’ affects peer review, whereby prominent authors receive more-favourable reviews compared to anonymous or junior authors. In a trial experiment of the exact same paper, a junior author was three times more likely to receive a rejection decision than a high-profile author. In the open peer-review system, accountability is more transparent and the willingness to address certain shortcomings are enhanced, especially if referee reports are published alongside the final accepted version of the article. It is possible, however, some referees may not be as candid in their critique if they feel this would compromise future funding or publication opportunities, or they might be more reluctant to agree to review in the first place. 

All three approaches are associated with various challenges. Importantly, the timeframe for peer review and the workload expectations are considerable stumbling blocks. It can take months, sometimes years to get some findings published, and a large aspect of this protracted process is peer review, which stifles benefits for readers, patients, and impacts timeliness of the published findings. Another aspect of academic publishing is the unpopular journal ‘impact factor’ ratings, in which the journal kudos or reputation is the driver for publishing the research. A paper published in a high-impact journal with low citations is often unfairly judged as being more successful than a highly cited piece of work in a lesser known journal. All the while that scientists’ careers are judged on ‘where’ their work is published (prestige) rather than what they publish, this only adds to the perils of the publishing machine. Other grievances are the astronomical profit margins for the publications industry, which incredibly exceed even Big Tech revenues. For instance, Elsevier’s profit margin is 37% with a $3.9 billion revenue, which overshadows Apple (28%), Google (25%) and Microsoft (34%) profit shares. Authors and referees receive zero of this market share. 

Referees are often selected by a journal in-house editorial team or sometimes by an external ‘advisory board’. Both have associated advantages and pitfalls. For example, in-house expertise might be limited and so the ability to select the ‘most appropriate’ expert might be suboptimal and based less on the knowledge of the expert and more on discoverability of their publication history. Referee diversity selection can be influenced differently by advisory board members versus an in-house editorial team. The transparency of how ‘favoured’ experts or reviewers are selected is often unclear. Since a disproportionate number of external editors or advisory board members publish their own research in the journals they edit, this further compounds bias and appropriate peer-reviewer selection concerns. Moreover, expectations for articles to tell the ‘perfect story’ in a predetermined ‘neat’ narrative contradicts the spontaneity and creativity of the experimental journey, hindering academic research. Some reviewers might believe that the research question behind the piece of work was not high priority or relevant, but it is too late to voice such concerns when the funding for the research has already been granted. Earlier appraisal of research hypotheses would be a far better way to determine validity than peer review. The problem with much of scientific and biomedical research is that exploring the validity of the findings is usually obtained after the work has been carried out, so peer review adds limited value. 

How and why has peer review strayed from its original remit and what can be done to improve this situation? To address this question requires scrutiny of the publishing industry. Scientists and clinicians are drowning in data and the ever-increasing plethora of papers being published (over 5 million articles in 2022) and new journal launches exacerbate this problem. In the past year, a huge number of papers have been retracted – more for censorship reasons than because the research is flawed. There is non-existent academic recognition for peer reviewers: their work is often unrecognised, invisible, and they are not financially reimbursed. Open peer review that publishes referee reports partially addresses this issue. However, finding adequate time to review new articles is a constant struggle as it is time consuming to assess new data and methodological scrutiny to the necessary levels. Adding to the problem is that page constraints of journals mean that a lot of important information is relegated to the ‘supplementary material’ published online as an annex to the main paper, which segregates and de-emphasises crucial aspects of research to a ‘lower priority’ section.

Ultimately, open peer review is the fairest and most transparent way to help overcome some of the issues with peer review. Undeniably, the hard work of the referees should be acknowledged, be visible and their efforts potentially financially reimbursed to reflect their contribution to the finalised published work. In some respects, peer review could be seen more as an educational and expertise exchange opportunity as much as a mechanism to ‘validate’ or scrutinise science. While many journals subscribe to the COPE (Committee on Publication Ethics), guidelines and recommendations, established in 1997 to promote good scientific conduct and assist in individual instances of misconduct, there are multiple aspects that need to be addressed before peer review is dramatically changed for the better. 

As the former Chief Editor of Nature Reviews Clinical Oncology, I found it concerning that referees are required to complete two sections on all online peer review submission systems: one text box is labelled ‘remarks to the author’ and a second text box ‘remarks to the editor’. Both sections are mandatory on online systems. We had several instances of reviewers adding relatively minor comments in the ‘author’ section but highly critical, sometimes unprofessional or vitriolic criticisms of a manuscript (or its authors) recommending rejection in the ‘editor’ text box. The remarks to the editor comments are never sent to the authors, as this is for the editor only. This greatly compounds the problem of transparency and accountability, as reviewers can hide behind the anonymity of their reports while condemning an article unfairly behind closed doors. This might explain why authors can be perplexed at ‘reject’ decisions when the comments received are addressable. There are very few instances when remarks to the editor should be undisclosed, so this section should be removed or only reserved for exceptional circumstances; for example, if knowledge of another unpublished or ‘in press’ piece of work might alter the findings or interpretation of a manuscript, thereby necessitating discretion. 

Journals should provide more guidance to reviewers so they focus on the science and not language or journal formatting concerns. This would allow referees to focus only on the scientific method, so they are not distracted by typographical or language issues. Editors could provide more information on why the referee was selected and avoid ‘generic’ invitation letters. In some instances, editing articles upstream of peer review would help referees; however, editors would not want to invest valuable time to improve articles that might be recommended for rejection. Articles can always be ‘polished’ post publication – especially in the digital era where printing and distribution costs are negligible. Continual improvements/updates of a published article would not only increase the ‘shelf life’ of the work, but would enhance the readership and visibility of the research – but this approach  should not be used to artificially bolster impact factors. Rather than publishing a complete manuscript, an update of an existing article could be abridged to the original – saving valuable time for authors and referees. This might also deter publication of new articles that are sound but boring, unhelpful or unnecessary or might even add value to articles that had a more limited remit on original publication.

In a world that seems to honour the merits of transparency and promote the longevity of published work, it seems odd that more is not being considered to catapult peer review and the publication process to be more dynamic and accountable. It is estimated 20% of manuscripts are rejected for publication, so a relatively high proportion of content never makes it to the published domain. Many highly cited articles are originally rejected in the first round of submission to ‘high-profile’ journals. In some cases, highly-cited papers are rejected outright and not sent for peer review by the editorial team. If rejection decisions are based on unsound or poor judgement, this hinders the research ecosystem and may deprive academics and healthcare stakeholders of important findings. Notably, corruption in peer review is increasing. A whistleblower described how a climate change paper was retracted purely because of disagreement not because of poor science or misconduct. It seems that certain activist scientists and journalists teamed up to pressure the publisher (in this case) to retract the paper, and sadly this pressure campaign worked. Differences of opinion in science should be embraced and welcomed and debates and discussions encouraged through the peer review and post-publication processes. Sadly, even well crafted ‘Letters to the editor’ are being rejected. 

If peer review aims to uncover misconduct, why is there a disturbing increase in the number of retractions? Alarmingly, more than 10,000 research papers were retracted in 2023 – way higher than all preceding years. The reasons may include scientific misconduct, but may be purely because the research findings might challenge an ‘official’ narrative, expose data that contradicts government or industry-led guidelines, or offer a completely new outlook on an existing dogma considered ‘settled’. The appalling trends in retractions point to corruption, and it is no coincidence that retracted papers have risen dramatically since covid. But in most cases, there is no independent investigation carried out about why papers were retracted. No thorough statements are published alongside the retraction to say if misconduct or poor scientific judgement that totally invalidates the findings are the cause of the decision to remove papers. One pair of authors had 300 papers criticised over a decade by a tenacious pair of sleuths, with eventually about two thirds of them being retracted. The senior author admitted fabricating results but 107 of his papers are still in print with no warning. Worth a look at Retraction Watch, who have a database tracking retractions as a window into the scientific process.

Also, the copyright, advertising, pay-per-view or APC (article processing charges) revenue streams publishers make from such papers are unsurprisingly never refunded or reimbursed, as occurs in other industries. The thorny issue of paper mills or cheating (businesses that sell bogus work and authorships to scientists) is estimated to be in the hundreds of thousands alone. The true number of bogus papers is likely far higher. 

As public health has been in the spotlight in the past 4 years, this has also inflated the status of peer review as the Holy Grail of research findings and ‘trusting the science’. The public and many academics assume, incorrectly, that if something is peer reviewed it must be trustworthy or of high integrity. Many have criticised peer review in terms of ‘parasitic’ or predatory journals and there have been calls to decentralise peer review. The ‘gatekeeping’ of peer review has been eroding for decades, but seems particularly egregious in recent years. Do we still need peer review and what role should it serve? How can peer review be revised to do justice to reviewers, authors, scientists and the public alike? 

The time has come for a paradigm shift in the mechanism and role of peer review and to recognise the efforts of reviewers. Embracing an ‘open’ peer review system is a first step. Inviting referees to assess only scientific or medical accuracy could be a measure taken before publication, with two reports being sufficient rather than striving for three – which tends to be the ‘standard’. Referees’ competing interests statements should also be included in the final published work. A ‘peer-analysis’ post publication where time constraints are of less concern would foster more academic engagement and might encourage a broader collective approach for follow-up work, forging new collaborations within the biomedical community. Perhaps different ‘tiers’ of peer review should be allocated to genuinely novel blue-sky hypothesis generating work. Furthermore, the issue of reproducibility of academic findings might deserve its own funding and peer review model, distinct from appraisal of more novel work. There should be no place for corrupt peer review and retraction practices.

We will revisit this topic again and offer further solutions to the integrity and misconduct aspects of peer review, which need to be urgently addressed, especially in times of heightened censorship of well-conducted biomedical research.