Let's just get rid of peer review
Two Truths and a Take, Season 2 Episode 19
Quick note: obviously it’s been a really hard and awful last few days, I wish I had something useful to contribute right now. By next week I’ll try to give this all more thought; for the time being I hope this is a nice distraction about a totally unrelated topic.
As you may know already, ten years ago (before I got into the tech world) I lived in the academia world, as a grad student doing neuroscience research. Scientific research, to put it bluntly, needs to be fixed. There are so many smart and well-intentioned people working really hard in university labs and research institutions, but whose best years are getting wasted through no fault of their own. It’s not an easy problem to fix, but we’re going to have to.
Several months ago I wrote a post called Can Twitter save science? which tackled what I see is the heart of the problem: the interconnected relationship between scientific publishing and academic career advancement. If you never read that post, read it first - it’s important context for how I feel about this issue generally, and what I see are the big issues we need to fix.
Since then, something really big happened! Covid happened. And it matters to this issue for two reasons. First, universities everywhere are going to face an enormous budget crunch, all at the same time, and that could provide the coordinated crisis that prompts university libraries to all capitulate on paying expensive journal subscription fees that they can no longer afford. Capitulation like this works best when everyone stops paying all at once, but prior to Covid, it was hard to imagine what single event could possibly coordinate everyone together like this. Well, we found one.
The second reason is similar, but on a bigger scale: really breaking apart and rebuilding the academic publishing and career advancement model is hard to do incrementally, because the existing system is held together by so many feedback loops. If you make a small step in one direction, you’ll get pulled back to centre. But Covid is a huge reset that everyone’s going to experience at the same time. It’s like the Blue Eyes problem you learn in CS: sometimes all you need is to get everybody thinking about the problem all at the same time, which was impossible to pull off, until some outside force shows up. Well, our outside force has shown up.
Right on time, this paper came out the other day proposing a major change to academic publishing that I think makes a lot of sense, and the time is right: get rid of peer review as a precondition of academic publishing.
Is peer review a good idea? | Remco Heesen, Liam Kofi Bright,British Journal for the Philosophy of Science (2020)
This sounds like a crazy radical proposition; peer review is one of the most sacred, enshrined traditions in academia. But this proposal is advocating for a more targeted change:
Our proposal is to abolish prepublication peer review. Scientists themselves will decide when their work is ready for sharing. When this happens, they publish their work online on something that looks like a preprint archive (think arXiv, bioRxiv, or PhilSci-archive, although the term ‘preprint’ would not be appropriate under our proposal). Authors can subsequently publish updated versions that reply to questions and comments from other scientists, which may have been provided publicly or privately. The business of journals will be to create curated collections of previously published articles. Their process for creating these collections will involve (postpublication) peer review, insofar as they currently use prepublication peer review.
The proposal here is to say: journals get to keep their job as curators, but lose their job as gatekeepers. This shouldn’t be that radical a proposal, since there’s nothing actually stopping scientists from doing this right now, aside from journals refusing to publish any work for which they can’t get exclusive rights or a pre-publication embargo. Ordinarily, there would be no way you could actually coordinate an effort to change this rule. But these aren’t ordinary times! Covid may be the moment of opportunity we need to actually pull off an effort like this.
Would it even be a good idea though? The paper above goes through twelve major decision criteria, and what they think would happen. Here’s a combination summary of what they’re arguing, overlapping with what I personally think about it (we mostly agree, although in some situations I actually go a fair bit farther than they do).
Sharing scientific results
There’s an expected norm in the scientific community that you’re supposed to share your work. Beyond the career or funding requirements to publish, or (more importantly) the prestige associated with getting your name on something published first, there’s something more fundamental at work here. Science is a craft, and it’s a life pursuit. Sharing your work is just something you’re supposed to do, for the benefit of science and the community. Scientists take that seriously.
If we got rid of peer review, there’s really no reason to believe that this norm would change. The only thing that would really change would be the speed at which scientists are able (or, perhaps, expected) to share their work. If the norm becomes “scientists are expected to share their work when they feel they are ready”, as is currently the case with conference abstracts anyway, little would likely change.
Time allocation
Under the current peer review system, every time an article comes up for review, a few scientists are assigned to review the article upon submission. This may not seem like a huge time commitment, but it adds up: by the time you get to be a senior scientist, you’re spending a lot of time reviewing papers on behalf of journals. You don’t get paid for this work; you’re simply expected to do it, as a norm in the community. (If you refuse, you’ll be blacklisted in various unpleasant ways.)
If peer review were to become optional (or shifted to post-publication, as is proposed), what would change? Hopefully, senior scientists would feel more free to allocate their time how they feel is best. In a post-publication peer review world, you probably see some sort of reset around the social norm to review papers on behalf of for-profit journals for free, and maybe even in general. If peer reviewers become a genuinely scarce resource, that would be an interesting challenge to the journal model, regardless of how peer review fits into the process.
Publication bias
In tech, as many of you know, you regularly go through a process called Code Review where senior managers go through software development work as it’s submitted and review it for quality and correctness, much like scientific journals. It’s been well-documented, in both cases, that if you’re a woman your work will be graded on a significantly harsher curve than if reviews are done on anonymous work. For a variety of reasons, it’s hard to review work blindly in either case - it matters a great deal who did the work, as some people may deserve more trust or skepticism than others. But if we simply got rid of review entirely, or moved it to a post-gatekeeping role as is proposed here, it’ll be interesting to see to what extent that changes the existing gap in publishing rates between genders in science, or other areas.
Library resources
The current tax on science levied by academic journals gets paid in part by university libraries, who have no choice but to pay whatever the publishing giants like Elsevier charge for their subscription bundles. (Although this may, finally, be changing in fits and starts.) Abolishing pre-publication peer review would necessarily kneecap most of the journals’ pricing power, and would probably put a great deal of them out of business - libraries would no longer really need to subscribe to journals, if any preprint could easily be found online. You still could subscribe to a journal, mind you, if you decide you want whatever value-added services they provide - commentary, perhaps, editorial, or who knows. They’ll have to be creative.
There is one small but important service that journals provide we shouldn’t overlook, though, and that’s proofreading and copy-editing. That’s a job you can’t totally ignore; someone’s got to do it. But who? Well, how about the library systems? It might take one or two percent of the massive budget you’ve just freed up to hire a team of high-quality scientific writers to serve as full-time support for university faculty. Plus, this would be an interesting shot in the arm for reengaging with libraries around a new kind of service model. Libraries are enormously under appreciated resources on campus (and also everywhere), so this might be an interesting experiment.
Scientific careers
This is the big one. This is more important than everything else.
One of my frustrations with this whole debate around journal access is how much of the debate is around the journal profit motive, and how much this model taxes scientific research. I wrote back in February: The real shame in academic publishing, if you ask me, isn’t Elsevier’s 35% profit margin on journal subscriptions. It’s the much larger amount of money, time and influence that is regressively taxed from the young scientists, to the old ones, in exchange for nothing but brand access. So long as journal access remains the yardstick that matters, then ... I doubt that the overall structure of the ecosystem will change that much. It’s bad for science, and by extension, bad for all of us.
Somewhat frustratingly, I was really hoping that this current paper I’m talking about would jump onto the same point, but they didn’t really. They did a good job of summarizing what I’d call the ‘weak thesis’ that peer review hurts career development, which is that peer review is inherently an imperfect judge of quality of work, and if quality of work is an important measuring stick for career advancement, then you’d want it to happen in the broadest, best possible way - by everyone (e.g. citations, shares, recognition in the community) rather than by a small committee.
That is fine, although again, I’d come back to the stronger thesis that I’d articulated a few months ago, which is that the real harm done by journal gatekeepers is that they impose a stranglehold on career development by serving as the mechanism through which established labs can effectively demand tribute from younger scientists. PhD students actually get real training from their supervisors, but postdocs desperate to build their academic brand have no choice but to pay tax to the labs who can get them into the big journals. It’s a regressive tax from old to young, levied at precisely the most creative and important years of a young scientist’s life. It’s terrible. It’s the dominant concern of the people who actually do all of the science.
The power of gatekeepers
This is kind of the same topic as before, although again, the current article I’m talking about actually fails to present a solution to the real problem, which is how to help young scientists advance their careers in some way other than paying tribute for journal access. Here’s where getting rid of pre-publication peer review won’t actually help much (although it doesn’t hurt); the challenge is that you need something else to actively help with this job. Here’s where Twitter comes in. Twitter does this other job really well: if you have something important to say, you’re going to get heard - even if you’re a nobody. This is where the scientific community honestly shines a lot - the culture is really bent towards finding and celebrating that kind of curiosity. Hence my previous post, “Can Twitter save science?”
Quality standards for scientific research
So here’s where we get into the principal reasons in favour of peer review, or really in favour of the peer review ideal. In the old days of academia (the several hundred years prior to the 20th century, as a rough estimate), peer review existed largely to try to elevate the quality of research that was probably going to be published anyway. The research societies held all of the sway and power in the scientific community, and journals were mostly an afterthought for archiving published research that had already been presented. The point of peer review was noble: peers helping each other present and immortalize their work in the best possible way it could be.
Then in the 20th century, as both the volume of research and number of journals skyrocketed, we had a few funny things happen. On the one hand, you had the rise of the Tier 1 journals, like Cell, Nature and Science, who could make your career - so peer review at those journals became a real kingmaker. Then you had all of the other journals, who ranged in quality down to the very bottom (where pretty much anyone can publish anything, and everyone knows it), and who are all ranked on this semi-absurd scale called Impact Factor that matters because everyone believes in it. What about peer review in those journals? What purpose does it serve, exactly?
It’s not like peer review as an institution is effectively serving as a gatekeeper into all of published science. You can get anything published somewhere. But once it is published, you’d hope that this research might be judged on its inherent quality, for the world to see - regardless of where it ends up. Upon publication, the peer review no longer really matters - or at least it shouldn’t so much, in a world with the internet.
All that goes to say, if you just one day got rid of pre-publication peer review entirely - just got rid of it, full stop - there’s no reason to believe that the overall quality of published research would go down, at all. You’re still incentivized to publish your best work; arguably more so because you no longer have the cover of “being peer reviewed” as legitimacy. There will still be good research, and bad research. And post-publication peer review will still be able to pass judgement on anything it wants. But there’s no reason to believe that research quality will actually change.
Malpractice detection
Another important benefit that peer review provided, in another era, is watching out for obviously fraudulent or dishonest research. In a world without the internet, you need this. But nowadays, I feel like we can probably do better. The math and physics community certainly gets by ok with the preprint system, where anything in preprint that gets viral attention is necessarily scrutinized and examined; probably by far more people who would otherwise have seen it, due to its open access nature.
Herding behaviour
Here’s another area where I’m going to diverge a bit from this paper. Herding behaviour is real in science, just as it is in tech or anywhere else - people want to run where they see everyone else running. Everything is really all about signalling, and if you ask me, peer review isn’t likely to change that basic fact.
That being said, I think that if you get rid of pre-production peer review, you’re going to really see some changes in how scientific herding behaviour works. The first is that herding behaviour will still exist, but it’ll probably happen at a much faster cycle, and that’s a good thing. One of the under appreciated aspects of how Silicon Valley works so well as that the FOMO-driven herd mentality operates at such a high cadence. This is a good thing, because it it gives more topics and ideas their 15 minutes of fame, and also it shortens the window of being topical, so you don’t get as many stale themes and institutions as you do elsewhere.
Slow-cycle herding behaviour is a lot more dangerous than fast-cycle herding behaviour, because slow-cycle herding gets huge numbers of people caught up in identical pursuits, under a slow feedback cycle of reacting to what’s interesting. Everyone benefits if you speed that up.
Long-run credit
Would getting rid of pre-publication peer review meaningfully change who gets credited with scientific work? The authors of this paper don’t think it’ll make much of a difference. But I disagree. I think it’ll make a huge difference, for the better.
It all comes back to the core issue of “paying tax”, where young academics have to pay tribute to established labs in order to get access to the good journals. If you think that moving peer review to post-publication wouldn’t change much about how attribution works, then you’re assuming that the same discoveries ultimately get made by the same labs; the only thing that would really change would be the dynamics and incentives around publishing it.
But I don’t think that’s what happens at all. If you get rid of the gatekeeping element of peer review and encourage preprint sharing, what that means is younger researchers and labs will be more incentivized to focus on what they actually want to do. And those early years are probably the years where you’re most likely to do actually original, creative work that contributes something completely new to the scientific field. Those are the discoveries and the papers that matter.
So if you get rid of peer review as a publishing gatekeeper, of course the dynamics of long-run credit will change - they’ll shift towards the younger labs and researchers who will actually make those great discoveries! This is good for science, and it’s what we want!
Quality guarantees for outsiders
If we get rid of preproduction peer review, can we still trust published research to the same degree? What if you’re an outsider to a particular field, and don’t know how to evaluate a piece of research critically - shouldn’t reviewers do that job for us? Well, here’s the thing: for high-end research at the forefront of importance, there’s no doubt that peer review will still take place. So anyone who wants that additional level of editorial guarantee should have no problem finding it. And for “low-end” research at the bottom of the stack, I have bad news for you if you thought that the current peer review system in place there is acting as any kind of quality control at all. It’s not.
So there’s little lost here, honestly, except perhaps in the middle. I’ll have to think about that. But this seems like an eminently fixable problem. Math and Physics have certainly worked it out. I’m not sure what’s so special about the life sciences that we can’t too.
The Matthew Effect
The last question they address, which is a real one, I’m honestly not sure about: the natural tendency of the world, in many ways including this one, is for the rich to get richer. The big scientists get bigger. The cited labs get cited even more. Shouldn’t we just expect this to happen even more? Maybe. But I kinda doubt it. Peer reviewers are probably the most susceptible to grading quality of research according to the quality of the name submitting it. There’s certainly no reason to believe they’re better than average, if we’re being honest. I’d rather get rid of the gatekeeper and see what happens, than preserve the gatekeeping in the name of some sort of measured promotion that probably just mimics the Matthew Effect anyway. So I think this is a wash at worst, and probably a net win.
At the end of the day, I think this is the right move, and it would be a net win for science if we could pull this off. I’m not sure if we’ll succeed, but I can’t picture a better opportunity to try.
Permalink to this post is here: Let’s just get rid of peer review | alexdanco.com
It’s been a hard week. So to leave on a light note, here’s this week’s Tweet of the Week, brought to you by Dave Itzkoff. In addition to his regular job at the NYT and his fabulous book on Robin Williams, Dave is probably best known on Twitter for always having the perfect gif or screenshot from The Simpsons for any - literally any - occasion. This week we weren’t disappointed:
Have a great week,
Alex