A reader who is working on some reviews for a high-impact Journal of the One-Word Name wants to know how to avoid being the kind of horror-story reviewer that writers of, and commenters on, blogs like to describe in scathing, gory detail.
Do you or your readers have any advice on what to do to not become one of those anecdotes, beyond the obvious stuff like don't steal ideas? What is reasonable, as far as requests for additional data, when reviewing for a journal with essentially unlimited space for supplemental materials and very high standards?
Also, this journal asked me to review multiple papers by different groups working on similar problems. They often publish multiple papers on the same topic in a single issue, with some accompanying commentary, to make it a theme issue. I have been explicitly asked to compare the papers to each other, to ensure that similar work is reviewed at a similar standard. This is not something that I've done before. Any thoughts? To me it seems straightforward, and maybe even fairer than most processes, because it's more likely that similar work will meet similar standards, but I've heard horror stories about weird things happening when these journals want a theme issue. Maybe there are some fairness issues that I'm overlooking.
I suppose that the simplest way to avoid becoming anybody's horror story is to recommend publication, because then the authors will have no reason to complain, but that approach has some rather obvious problems.
Indeed.. Let's assume that this person is semi-joking about the last comment. Clearly you have to give the best and most honest judgment you can, based on what is in the paper(s) under review.
And that's the key to the whole thing: Give your best judgment. Be critical, but polite and constructive. No matter what the journal.
As to the issue of proposing a lot of new research: Editors are ultimately the ones to blame for this, not (just) the reviewers. I could propose that the authors of a manuscript I am reviewing do 2 more years of intense data collection on the most expensive and inaccessible machines in the universe before the paper would meet my standards, but the editor doesn't have to take that seriously.
Editors can ask authors to explain why such requests are unrealistic/unnecessary, or can use their own judgment and say "I know that Reviewer 2 proposed that you do a series of expensive and time-consuming new experiments/analysis (or whatever), but you can ignore that comment."
Or editors can concur with these recommendations by reviewers, in which case, you can try to argue with them (politely and briefly) or you can just take your awesome paper to another journal.
But back to what a reviewer can do:
When I review a manuscript that does seem to have a gap that could/should be filled, I think very carefully about how strongly I word my recommendation about new work. Options are:
Unambiguous/strong statement: This work is unpublishable without the following ....
More ambiguous but still quite strong: This work would be greatly improved and the conclusions much more believable if you did the following...
Passive-aggressive in a mild way: Although it would have been useful/better if you had [done this and that], I think that your interpretation/conclusion is quite/mostly reasonable given the data/analysis presented.
Nicest: I am not suggesting at all that you do this because I think the manuscript is publishable with the existing dataset, but I wondered if in future research on this topic if you would be able to do [this other interesting thing that would help answer some additional important questions].
The issue of supplying supplemental material is also a major concern for authors. You need to provide sufficient documentation of your work, but at some point it becomes absurd if most of the content of the paper is in the supplement, other than some cryptic text (that can't be understood without the supplementary info) published in the main body of the article. Reviewers should only request essential supplementary material that is not already provided, following the norms of their field for archival material.
In the end, it's the editor's call on whether to use or ignore the reviewers' comments about adding more material to the paper and/or doing more research to include in the paper. All you can do as a reviewer, if you want any hope that your time and effort will be worthwhile, is to write a thorough, constructive, interesting review that helps improve the paper and helps the editor weigh the various review comments and make a good decision. [I have not reviewed a series of papers on a theme before, but perhaps others can chime in on that topic.] This is true whether you are reviewing for Journal of the One-Word Name or Journal of the Most Obscure Topic in the World.