What is the etiquette for disclosing an anonymous review that you wrote? It’s not a trivial question because all of us have the natural sense that an anonymous review is supposed to stay anonymous. Even I, an advocate of non-anonymous open peer review, see the problem of going public with something that was written in private at the request of an editor, with the understanding from the author and the journal that the review is private and anonymous.
I tracked an excellent and extensive discussion of this on Twitter, I listened, I have been thinking hard about it for a few hours on the airplane. And the more I think about this, the clearer it is that it is deeply wrong to keep silent about a paper you reviewed and think is flawed. Yes, we have the responsibility to the author of the paper to be civil. But that author-reviewer contract stands in direct conflict with the responsibility to the scientific community.
Suppose I got the STAP paper to review, I saw through it and outlined the fundamental flaws. Then the Nature editor made the call to publish it anyway. Can I make my review public? By keeping silent, I am honoring a single author and disrespecting a world of researchers. By keeping silent, I am letting students and postdocs waste months or years chasing smoke. I have a responsibility to the countless scientists following up on the published work. I have the responsibility to science and society. I think that science, society and that world of researchers override my oath to the author.
Now we come to the question of the appropriate means of responding. Every single person whose work is questioned fires back with “This is unprecedented! Why didn’t you contact me first? Why didn’t you write to the editor? Where is your civility? You didn’t follow the etiquette!” Well, very often, any attempt to do it the “expected civil way” is blocked by the journal or the highly influential researcher you disagree with. It can take years to publish a rebuttal. Journals often reject letters of concern because it’s not fun to publish them. By arguing for the via-editor/journal way, we are placing an extraordinary burden on the scientist raising the question. We are telling that scientist to shut up and keep quiet.
Yes, we should be civil and focus on the science rather than personal motivations. We don’t need to accuse of misconduct, unless we have clear proof of it. We don’t need to be mean. But we have to be honest and we have to encourage open and critical post-publication discussion. In the Twitter thread on this, I see completely random lines drawn on the good and bad way to have post-publication discussion. Alerting the editor and trying to publish the rebuttal in the original journal is okay but elsewhere it’s not. Publishing elsewhere is okay if it’s a journal, but not okay if it isn’t peer-reviewed. Publishing a critique on a blog is okay, but disclosing your review on PubPeer is out of line.
Those lines are entirely random. My paper was ripped to shreds in a Naturepublication from Mike Snyder's group. No one reached out to me, even though I had personally discussed our results with Mike Snyder prior to submitting the work for publication. Mike Snyder could have easily contacted me as we had even been co-authors on another paper. I found the critique accidentally through my PubChaserecommendations. I responded on PubMed Commons and on PubPeer. At least my response went to the corresponding author via PubPeer. (Note, I have no problem with my paper being scrutinized and criticized. I don't think Mike Snyder's group needed to talk to me first. But see Note below on Mike Snyder's perspective regarding etiquette.)
There is no rhyme or reason to personal opinions of what is and isn’t ethical for post-publication disputes. By asking people not to criticize we undermine science. We hurt scientists. We hurt ourselves. We have to grow up. It’s not about us – it’s about the science. We have to learn to criticize each other in a firm but civil manner. That’s doable any way you like, Twitter, original journal, PubPeer, your blog, PubMed Commons. Some are going to be mean and uncivil no matter where. Most of us can be respectful and focus on the science. All of us need to embrace post-publication critique.
-------------------------------
Note: here are recent examples of post-publication critiques that were flagged as uncivil.
1. Yoav Gilad was criticized for disputing Mike Snyder’s work. And Mike Snyder said, " “If someone has concerns, the normal route is to contact the journal or the authors,. There was an avalanche of comments before we even knew about it.”
2. Lior Pachter was criticized for questioning the conclusions of Eric Lander and Manolis Kellis.
3. Vicki Vance was criticized for making public her review on PubPeer (more here).
4. The Berkeley students David Broockman and Joshua Kalla were criticized for attacking LaCour’s work and leading to a retraction of his paper. LaCour wrote, "I note that Broockman et al. (2015)’s decision to not present the lead author with the critique directly, by-pass the peer-review process, privately investigate data collection activities without knowledge or consent of the author, demand confidential identifying information from respondents in a study without grounds or standing to do so, publicize unsubstantiated allegations and hearsay prior to a formal investigation, is unprecedented, unethical, and anomalous in the relevant literature."
----------------------
[Comments to the post below]
----------------------
Anonymous
3 years ago
My personal experience on attempts to communicate with authors of flawed papers, as a reviewer or just as a reader, is horrendously bad.
No reply
No reply on a second email
No eye contact at conferences, no return greeting, turning away and walking off
Worse, there are by now two papers I found serious flaws in (and no, I wasn't wrong: if measurements are off by an order of magnitude, then I am well competent to check that, and the editors agreed with me) that later were published unaltered in other journals.
Lesson learned: I will now write two papers, each exposing the flaws of one paper I reviewed, and I will explicitly ask for the authors not to be allowed to review.
Admittedly, I had one very good experience. Just one. But although that colleague behaved perfectly professional, we now privately are no longer on speaking terms.
No reply
No reply on a second email
No eye contact at conferences, no return greeting, turning away and walking off
Worse, there are by now two papers I found serious flaws in (and no, I wasn't wrong: if measurements are off by an order of magnitude, then I am well competent to check that, and the editors agreed with me) that later were published unaltered in other journals.
Lesson learned: I will now write two papers, each exposing the flaws of one paper I reviewed, and I will explicitly ask for the authors not to be allowed to review.
Admittedly, I had one very good experience. Just one. But although that colleague behaved perfectly professional, we now privately are no longer on speaking terms.
Though academic journals provide an important service as a means to select articles, sort and categorize the scientific output, it is bad (and somewhat unethical) that the whole process of scientific debate depends exclusively on an intermediate (the editor) that has certainly (and quite understandably) less thought about possible problems of an article than its authors or the peers challenging it. There must be a scientific public place where scientists can meet and debate openly about their articles in a way which goes beyond (and deeper than) the binary question of accepting/rejecting it and where new versions could be uploaded according to peers' feedback. The scientific truth should not be decided through an unverifiable process but emerge from open consensus. This scientific public place should also have a native system of blogs, well connected to articles, so that any scientist may voice and elaborate on his particular views of his/her field, which may be contradictory with others'. There is a readily available platform for that matter: sjscience.org . (post by Michaël Bon)
-------------------
[Mike Tayor]
In 2006, a badly flawed paper on a subject that I know pretty well was published in PLoS Biology. I was part of a six-person team who wrote a rebuttal, which PLoS Biology rejected as insufficiently novel. Appeals were written and rejected. All six of us eventually found more rewarding things to do and drifted away from the project. So the original paper still stands unchallenged.
These days, I wouldn't waste my time on that process, which at best would have resulted in a response being published six months after the event when everyone had forgotten about it. I'd just go right ahead and blog about it. There's no argument to be had here: blogs are simply a much, much better medium for carrying on a dispute than peer-reviewed papers. They move faster, they're more interactive, and crucially they can't be blocked.
---------------------
[Nikolai Slavov]
Lenny,
I hear that some of these post publication discussions erupt on blogs because the journals refuse to consider and publish comments for the published papers.
If I write a review for a paper, I own the review. I refuse to give intellectual rights and ownership to the editors/journals. I better be able to stand up and take responsibility for my reviews and opinions. Since any review I write reflects my opinions, I see no reason not to be able to share my reviews/opinions with the world on say pubmed commons. Whether I was asked by an editor or not to read the paper seems insignificant to me. The important thing is that I read the paper, I have an opinion, it is well considered, and I am willing to take full responsibility for it!
-----------
[William Gunn]
I think I have a unique perspective on this because I see how reviews work in both the academic world and in the (academic) software world, where I've been for the past 8 years. It's exceedingly rare that someone gives a heads-up to the author when they're about to write a blog post about a piece of software. Generally, the time from the inspiration for the post to the time when you start writing is hours to days. The higher-profile the reviewer, the more likely they are to give a heads-up. For example, if you are interviewed and give some critical comments to a major media outlet, you'll may get a heads-up, but often only if you ask for it - by no means is this standard or common. If you're a senior VC, you might have your post read by a few other people before posting. Fred Wilson (at http://avc.com ) often does this.
In this sense, it's a technology problem. Tech companies all have systems for monitoring what gets said about us, with the assumption that not everyone will let us know. Often, people who are commenting are speaking to a very different audience and it might not even occur to them to invite you to comment. They may want to have a discussion among their group and feel like if you jumped in at the start, it would squelch the thread. I know that I always avoided jumping in too early in discussion threads about Mendeley that I found "in the wild" for exactly that reason.
So in that context, though it seems reasonable, it's unrealistic for any author to expect prior notice of coming criticism. It's also not scalable. If everyone asked everyone before releasing anything they said about anyone, nothing would ever get written. What you're essentially asking, when you ask for prior disclosure of criticism, is just to start the discussion earlier among a smaller group. This is often done to try to keep any possible mistakes you made under wraps. I think there are good arguments to be made for doing so, if you think the reviewer is misunderstood but has a way bigger reach than you do and thus the misunderstanding will become the consensus understanding, but I would suggest that in most cases that's not the dynamic and if your reviewer really is hostile and misunderstood, public reviews give you a perfect opportunity to show the world how misunderstood they really are. Where this dynamic is worst is when a senior scientist has some criticism of a person who's very much the junior.
With respect to peer review submitted in confidence, I would agree it may not be appropriate to publish your review verbatim ahead of the paper coming out, but once the paper is out, it's fair game and you should rewrite, if necessary, the substance of your review on Pubmed Commons or your blog or PubPeer or wherever you think is best. It's just common sense courtesy that you would invite the author to respond, but a heads-up is not practical in many cases.
Essentially this is a discussion about community norms, so I would suggest the following:
1. Inspiration and time is fleeting, so have a bias towards saying something rather than not.
2. Don't be an asshole.
3. Forgive other people who are assholes to you.
4. Review unto others as you would have them review unto you. (credit @ShitAcademicsSay)
5. Expect & forgive incompleteness - perfection is unattainable - do the best you can in the time you have.
Practical example: this comment itself is poorly organized, incomplete, and I have to fight the urge to reorganize it. If I invested that time, I would then have to fight the urge to just blog my (then) essay on the topic. However, I'll have to trust that anyone who reads this will give me the benefit of the doubt, because I now have other stuff I have to go do, and I'd never get around to polishing it up and posting it. The same thought process is behind pretty much any review anyone would write. Bias towards action!
--------------------
To be honest, I'm surprised there is so much debate around this. Overall, I agree with your considered thoughts.
It is pretty unpleasant if you hear on the grapevine that someone has criticised your work, but I don't think that means people shouldn't criticise it. I think, in general, it is a good idea to contact the author to let them know the critique is coming; they won't like it, but at least they won't hear through a third party. But there should be no expectation that the author has a right to veto publication of a critique. It's important to realise that, although it is good to be polite, politeness should never trump concern for accuracy. This is science, not a dinner party.
Once a paper is published, I wouldn't see any reason to hide my appraisal of it, just because the review was originally solicited by a journal editor. The example I gave of this is of a paper in PLOS One. I had reservations about the paper and expressed them in my review, but the paper was published anyhow. I don't think I would have taken further action, except that I saw that Jon Brock had published his reviewer comments on the PLOS One comment site - and he had made a number of points similar to those I made:
http://www.plosone.org/annotation/listThread.action?root=66527.
The authors did respond to Jon's comment, but in a way to imply they had addressed the problems. I didn't think they had, and felt I should add my voice to Brock's, so I also published my review as a comment in the same thread. I felt this was important because this was a paper by an influential author focussing on a serious developmental condition, and my impression was that there had been p-hacking and the results would not be reproducible.
My strong preference these days is to publish in the handful of journals that publish the peer review alongside the paper. PeerJ is exemplary in this regard. One reason I like this is because PeerJ is not well known and if people hear you have a paper there, they tend to assume it hasn't been reviewed properly. I like people to be able to see that the reviews are there, and, in my experience are typically as robust as anywhere else. I also use PubMed Commons increasingly to raise concerns about papers - though in general these are not papers I have reviewed. But I would not hesitate to write there if I felt an important reviewer comment had been overlooked.
Dorothy Bishop
