Peer review

This piece in the Times Higher Education is just too important not to re-post the link here!

I spend a lot of time doing reviews, receiving reviews (including nasty ones), and exploring the process of peer review in workshops with research students. It is something I care passionately about, both for its value and because it is far from perfect.

This piece in the THE is wonderful for the way in which senior, experienced academics show there is nothing to be ashamed of in receiving a nasty review from a peer. I often say that if you ask a senior professor if they have had a really unpleasant, overly harsh, aggressively worded review then (1) they are lying, (2) they are deluded, (3) they must never have published anything that has gone through peer review.

So, I’m happy to declare that I’ve received plenty of nasty reviews. One questioned my professional integrity, accusing me of wasting participants’ time (when all my evidence suggested the contrary was going on). Another unhelpfully criticised me for only providing 8 references, when this was a rule stipulated by the journal for the kind of piece I had submitted. Another suggested my research was half baked and I should come back when it was done. Ouch.

Personally, I love Andrew Oswald’s “strategy that I would recommend to any young scholar, I do not commit nasty comments to memory”. My version of this involves taking a thick black marker pen and blotting out all the unnecessary and unhelpful vitriol. Like the CIA do when they release classified documents.

I also like the idea, when you get a nasty review, of showing it to as many people as possible, being proud: “Hey! Look at this gem I got! It’s even worse than the one you showed me last week! Clearly this referee has some major issues…”

Do I respect the idea that peer review has a place in academia? Absolutely. There is a need to both police and improve knowledge that has the benefits, impact and potential danger of scholarly legitimacy.

Do I recognise that the process has become, perhaps has always been, flawed? Absolutely. Which is why I spent lots of time helping students and early career researchers to ‘read’ reviews in a productive way, to understand the bizarre, messy and unprofessional things that go on, and to respond agentically to it, rather than review being something that is done on or happens to them.

Would paying reviewers help? Probably not. Would publishing their reviews, with their names next to what they wrote? Well that might remove the cloak of anonymity and some of the mischief that goes on. But then people might not say yes so often.

Better editing might also help in some cases. Rather than copying and pasting, or just pressing ‘forward’, good editors should intervene, and mediate the communication between reviewers and authors, and this can extend to not passing on unhelpful, unprofessional, or unethical comments.

We have to remember peer review is not done by the best experts who know most about a particular topic. It is done by who will say yes to doing it (usually for free) when they have lots of other things to do. While journals may have guidelines they ask reviewers to follow, there are few well-developed pedagogies for reviewers (I submit some of the work I’ve been doing with research students as a humble attempt to address this in a small way).

As the THE piece states, peer review is flawed because it is done by human beings. But this is also what makes it possible to improve the knowledge we present to the public and on their behalf.

Advertisements

2 thoughts on “Peer review

  1. Andy S

    Hi Nick,

    A good piece, and a good response. The THE has a tendency to publish fairly one-sided pieces on academic publishing with a radical lean (full disclosure: I am an academic publisher), so I was pleased to see some nuance and richness of opinion in the six interviews. That peer review is flawed, but that most academics accept it as the best worst choice is a truism and is not newsworthy, I have whole books written about its various pros and cons in its various forms, and straight up abusiveness is one of the least serious of its flaws. It is more novel that people are feeling able to open up about their reviews, share them, and perhaps even publish them alongside their articles. I think that’s great (where the anonymity of the reviewer can be preserved). I like your comment about sharing the most heinous examples. We used to have some of our most foaming feedback pinned above the photocopier, and some of it is very foamy indeed.

    Review is mostly constructive, even when downright rude, and making the process of research review in light of peer comments more transparent is something that many early career researchers may find valuable.

    “Would paying reviewers help? Probably not. Would publishing their reviews, with their names next to what they wrote? Well that might remove the cloak of anonymity and some of the mischief that goes on. But then people might not say yes so often.”

    Couldn’t agree more. The process of incentivisation is fraught with unknowns, and there are words of warning in the article: “But, as with all the ventures by academics into quantifiable systems of assessment, it has grown into a monster.” Incentivisation also won’t prevent the problems in the article unless it goes hand-in-hand with quality benchmarks. Yes! Reviews! Of peer reviews! Not terribly sustainable. Published reviews on the other hand should be self-governing, but the ‘cloak of anonymity’ you speak of works in two directions, and has an important role in protecting reviewers. There is a baby we need to look after when throwing out that bathwater. The brief stats I’ve heard on open peer review indicate that around 6-8 reviewers had to be invited to achieve 2 acceptances, as opposed to 4-6 for anonymous review. Only around 50% of review authors in optional systems opt to sign their reviews (presumably the less vulnerable ones). I’ve also heard arguments that point to difficulties in publicly posting criticism among some global cultures. So open review is not a panacea, though it may prevent some of the worst cases of abuse, bias, and poor-quality reviewing. Given the inefficiency, it’s fine for single journals like the BMJ to impose these policies, as the process is perhaps more objective in medicine, and the BMJ has no need to fear for its recruitment of peer reviewers. What’s morally or scientifically robust in these waters is not necessarily what is pragmatic for the scale process which at the heart of academia at all levels – what’s good for the few is not always good for the many.

    “Better editing might also help in some cases. Rather than copying and pasting, or just pressing ‘forward’, good editors should intervene, and mediate the communication between reviewers and authors, and this can extend to not passing on unhelpful, unprofessional, or unethical comments.”

    Agreed. One of the solutions is to have harder line editors acting as enforcers. But again, here come the caveats – how to do blacklist reviewers when the peer group is vanishingly small, and it’s tough to get reviewers anyway? How do you decide what to censor? I’ve talked with academics before who have told me that they would refuse to conduct a review that they thought was going to be edited and selectively passed on to the author. Other systems do exist – conferred peer review is one such system, used by journals like eLife. In this system, reviewers and editors combine to form a joint, single review report. It’s promising, but at the heart of these ‘solutions’ are processes that make peer review harder, more involved, and less efficient, against a market which wants (needs) review to be swifter, cheaper, and more efficient. The ‘best worst’ system is sure to be around for a few years yet.

    By far the most constructive argument I’ve seen around all of this is that we need to invest in much better peer review training – ‘we’ being academia, journals, publishers. How much training did you get on how to peer review? Was it useful? Is enough training done?

    Reply
    1. nickhopwood Post author

      Thanks Andy for the extensive and thoughtful comments. Could you be tempted into penning a guest post for the blog? I think you’re right that there is a lot a need for less academic-centric views, and some more balance that recognises the mutual relationships between academia and commercial publishers.

      I agree with the risks and hestitations you mention, and think that better training for reviewers is key. I realise now that calling for more editorial mediation might also have the risk of centralising selection and control in fewer hands.

      You’re right: peer review might leave people annoy, and upset at times. But there wasn’t much knowledge that got worse as a result of other people providing criticism.

      I do think we need to come clean about what is going on – are we duping the wider public into thinking peer review is done ‘blind’ (pah!) by ‘experts’… both are true and yet both are misleading too, often.

      Geoffrey Walford once wrote “The use of referees sometimes leads to odd decisions”. Quite.

      Reply

Please join in and leave a reply!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s