Finished up another paper review yesterday and I was thinking about the process of actually doing the review. I’ve reviewed a bunch of papers over the years and I follow a general strategy that seems to work well for me. I’m sure there are lots of great ways of doing this- and I’m not trying to be comprehensive here, just giving some ideas.
The general process:
- I start by printing out the paper. I’ve reviewed a few manuscripts completely electronically, but I find that to be difficult. It really helps me to have a paper copy that I can jot notes on and underline sections.
- I do a first read through going pretty much straight through and not sweating it if I don’t get something right away since I know I’ll go back over it again.
- During this read through I mark sections that seem confusing, jot questions I have down in the margins, and underline misspelled or misused words.
- Generally at this point I’ll start writing up my review – which generally consists of a summary paragraph, a list of major comments and a list of minor comments- but check the journal guidelines for specifics. This allows me to start the process and get something down on paper. I generally start by listing out the minor comments, and slowly add in the major comments.
- I re-read the paper being guided by my questions I’ve noted. This allows me to delve in to sections that are confusing to see if the section is actually confusing or if I’m just missing something. That’s sometimes the hardest call to make as a reviewer. As I go back through the paper I try to develop and refine my major comments.
Here are some things to remember as you’re reviewing papers:
- You have an obligation and duty as a reviewer to be thorough and make sure that you’ve really tried to understand what the authors are saying. For me this means not ignoring those nagging feelings that I sometimes get, “well it seems OK, but this one section is a little fuzzy. It seems odd”. It’s easy to brush that feeling aside and believe that the authors know what they’re talking about. But don’t do that. Really look at the argument they’re making and try to understand it. Many times I’ll be able to discriminate if they’ve got it right or not by putting a little effort into it. If you can’t understand it after having tried then you’re in the shoes of a future reader- and it’s perfectly all right to comment that you didn’t understand it and it needs to be made more clear.
- You also have an obligation to be as clear as possible in your communication. That is, try to be specific about your comments. List the page and line numbers that you’re referring to. Specify exactly what your problem with the text is- even if it’s that you don’t understand. If you can, suggest the kind of solution that you’d like to see in a revised manuscript.
- Before rejecting a paper think it over carefully. Would a revision reasonably be able to fix the problems? Did the paper annoy you for some reason? If so was that annoyance a significant flaw in the paper, or was it a pet peeve that doesn’t merit a harsh penalty? Rejection is part of the business and it’s really not unusual for papers to get rejected, just make sure you’re rejecting for the right reasons.
- Before accepting a paper think it over carefully. This paper will enter the scientific record as “peer reviewed”- which should mean something. The review will reflect on you personally, whether or not it is anonymous. If it’s not anonymous (some journals post the names of the reviewers and in some cases the reviews themselves) then everyone will be able to make their own judgement about whether you screwed up by accepting- are you good with that? If the review is anonymous the editor (who can frequently be someone well-known and/or influential) still knows who you are. Also, many sub-sub-sub-discplines are small enough that the authors may be able to glean who you are from your review, they may even have suggested you as a reviewer. This is especially true if your review includes a comment like, “the authors neglected to mention the seminal work of McDermott, et al. (McDermott, et al. 2009, McDermott, et al. 2010).”
- Remember that it’s OK to say that you don’t know or that you aren’t an expert in a particular area. You can either communicate directly with the editor prior to completion of the review (if you feel that you really aren’t suited to provide a review at all) and request that you be removed from the review process or state where you might be a bit shaky in the comments to the editor (this is generally a separate text box on your review page that lets you communicate with the editor, but the authors don’t see it).
- Added (h/t Jessie Tenenbaum): It’s ok to decline a review because you are in a particularly busy period and just can’t devote the time it will require (e.g. when traveling, or a grant is due) but do remember we’re all busy, but we all rely on others agreeing to review OUR papers.
- Added (h/t Jessie Tenenbaum): If you must decline, recommendations of another qualified reviewer are GREATLY appreciated, especially for those sub-sub-sub-specialty areas
- Added (h/t Jessie Tenenbaum): The reviewer’s role is ideally more of coach than critic. It’s helpful to approach the review with the goal of helping the authors to make it a better paper for publication some day- either this submission or elsewhere.
Some general things to look out for
Sure, papers from different disciplines and sub-disciplines and sub-sub-disciplines require different kinds of review and have different red flags but here are some things I think are fairly general to look out for in papers (see also Eight Red Flags in Bioinformatic Analysis).
- Are the main arguments made in the paper understandable? Is the data presented sufficient to be able to evaluate the claims of the paper? Is this data accessible in a sufficiently raw form, say as a supplement? For bioinformatics-type papers is the code available?
- Have the appropriate controls been done? For bioinformatics-type papers this usually amounts to answering the question: “Would similar results have been seen if we’d looked in an appropriately randomized dataset?”- possibly my most frequent criticism of these kinds of papers.
- Is language usage appropriate and clear? This can be in terms of language usage itself (say by non-native English speaking authors), consistency (same word used for same concept all the way through), and just general clarity. Your job as reviewer is not to proofread the paper- you should probably not take your time to correct every instance of misused language in the document. Generally it’s sufficient to state in the review that “language usage throughout the manuscript was unclear/inappropriate and needs to be carefully reviewed” but if you see repeated offenses you could mention them in the minor comments.
- Are conclusions appropriate for the results presented? I see many times (and get back as comments on my own papers) that the conclusions drawn from the results are too strong, that the results presented don’t support such strong conclusions, or sometimes that conclusions are drawn that don’t seem to match the data presented at all (or not well).
- What does the study tell you about the underlying biology? Does it shed significant light on an important question? Can you identify from the manuscript what that question is (this is a frequent problem I see- the gap being addressed is not clearly stated, or stated at all)? Evaluation of this question should vary depending on the focus of the journal- some journals do not (and should not) require groundbreaking biological advances.
- Are there replicates? That is, did they do the experiments more than once (more than twice, actually, should be at least three times)? How were the replicates done? Are these technical replicates – essentially where some of the sample is split during some point in the processing and analyzed- or biological replicates – where individual and independent biological samples were taken (say from different patients, cultures, or animals) and processed and analyzed independently?
- Are the statistical approaches used to draw the conclusions appropriate and convincing? This is a place where knowing about the limitations on the p-value come in handy: for example that a comparison can have a highly significant p-value but a largely meaningless effect size. It is also OK to state that you don’t have a sufficient understanding of the statistical methods used in the paper to provide an evaluation. You’re then kicking it back to the editor to make a decision or to get a more statistically-saavy reviewer to evaluate the manuscript.
It’s important to take your role seriously. You and the one or two other reviewers plus the editor for the paper are the keepers of the scientific peer review flame. You will help make the decision on whether or not the work that is presented, that probably took a lot of time and effort to produce, is worthy of being published and distributed to the scientific community. If you’ve been on the receiving end of a review (and who hasn’t) think about how you felt- did you complain about the reviewer not spending time on your paper, about them “not getting” it, about them doing a poor job that set you back months? Then don’t be that person. Finally, try to be on time with your reviews. The average time in review is long (I have an estimate based on my papers here) but it doesn’t need to be so long. The process of peer review can be very helpful for you the reviewer. I find that it helps my writing a lot to see good and bad examples of scientific manuscripts, to see how different people present their work, and to think critically about the science.