Making a super villain

I’ve written about Reviewer 3 before (here, here, here, and here). Somehow the third reviewer has come to embody the capriciousness (and sometimes meanness) of the anonymous peer review process. Note that I believe in the peer review process, but am a realist about what it means and what it accomplishes. It doesn’t mean that every paper passing peer review is perfect and it doesn’t mean that every peer reviewer is doing a great job of reviewing.

When I’m a reviewer I see the peer review process through the lens of the line from Spiderman (Stan Lee), “with great power comes great responsibility”. I strive to put as much effort in to each paper I review as I would expect and want from the reviewers who review my papers. Sometimes that means that I don’t get my reviews back exactly on time- but better that than a crappy, half-thought-through review. I’m not sure that I always succeed. Sometimes I think that I may have missed points made by the authors, or I may have the wrong idea about an approach or result. However, if I’ve done a good job of trying to get it right the peer review process is working.



I’ve posted before about some of my organizational approaches (and attempts at it) but it can sometimes be impossible not to get overwhelmed and busy. Being busy on multiple tasks, with multiple deadlines can be a killer, but sometimes it crystallizes a resolve to move some of those items off your todo list and you increase your overall effectiveness (you know, less Twitter and blogging and comic-making and stuff). I’ve also seen people who seem to make it a part of their academic persona to be perpetually too busy. This seems to be considered a status symbol (often times mostly by the person being so busy). The key to busy-ness and keeping your head above water (and the seals at bay) is balance. Make sure to keep perspective about what you’re doing and know that often (maybe always) banging your head against the same task for hours on end is counterproductive.

Anyway, here’s a handy tool to help you assess your level of busy-ness, fresh from the RedPen/BlackPen labs.


Proposal gambit – Betting the ranch

Last spring I posted about a proposal I’d put in where I’d published the key piece of preliminary data in F1000 Research, a journal that offers post-publication peer review.

The idea was that I could get my paper published (it’s available here) and accessible to reviewers prior to submission of my grant. It could then be peer-reviewed and I could address the revisions after that. This strategy was driven by the lag time between proposal submission and review for NIH, which is about 4 months. Also, it used to be possible to include papers that hadn’t been formally accepted by a journal as an appendix to NIH grants. This hasn’t been possible for some time now. But I figured this might be a pretty good way to get preliminary data out to the grant reviewers in a published form with quick turnaround. Or at least that you could utilize that lag time to also function as review time for your paper.

I was able to get my paper submitted to F100 Research and obtained a DOI and URL that I could include as a citation in my grant. Details here.

The review for the grant was completed in early June of this year and the results were not what I had hoped- the grant wasn’t even scored, despite being totally awesome (of course, right?). But for this post I’ll focus on the parts that are pertinent to the “gambit”- the use of post-publication peer review as preliminary data.

The results here were mostly unencouraging RE post-publication peer review being used this way, which was disappointing. But let me briefly describe the timeline, which is important to understand a large caveat about the results.

I received first-round reviews from two reviewers in a blindingly fast 10 and 16 days after initial submission. Both were encouraging, but had some substantial (and substantially helpful) requests. You can read them here and here. It took me longer than it could have to address these completely – though I did some new analysis and added additional explanation to several important points. I then resubmitted on around May 12th or so. However, due to some kind of issue the revised version wasn’t made available by F1000 Research until May 29th. Given that the NIH review panel met in the first week of June it is likely that the reviewers didn’t see the revised (and much improved version). The reviewers then got back final comments in early June (again- blindingly fast). You can read those here and here. The paper was accepted/approved/indexed in mid-June.

The grant had comments from three reviewers and each had something to say about the paper as preliminary data.

The first reviewer had the most negative comments.

It is not appropriate to point reviewers to a paper in order to save space in the proposal.

Alone this comment is pretty odd and makes me think that the reviewer was annoyed by the approach. So I can’t refer to a paper as preliminary data? On the face of it this is absolutely ridiculous. Science, and the accumulation of scientific knowledge just doesn’t work in a way that allows you to include all your preliminary data completely (as well as your research approach and everything else) in the space of 12 page grant. However, their further comments (which directly follow this one) shed some light on their thinking.

The PILGram approach should have been described in sufficient detail in the proposal to allow us to adequately assess it. The space currently used to lecture us on generative models could have been better used to actually provide details about the methods being developed.

So reading between the (somewhat grumpy) lines I think they mean to say that I should have done a better job of presenting some important details in the text itself. But my guess is that the first reviewer was not thrilled by the prospect of using a post-publication peer reviewed paper as preliminary data for the grant. Not thrilled.

  • Reviewer 1: Thumbs down.

Second reviewer’s comment.

The investigators revised the proposal according to prior reviews and included further details about the method in the form of a recently ‘published’ paper (the quotes are due to the fact that the paper was submitted to a journal that accepts and posts submissions even after peer review – F1000 Research). The public reviewers’ comments on the paper itself raise several concerns with the method proposed and whether it actually works sufficiently well.

This comment, unfortunately, is likely due to the timeline I presented above. I think they saw the first version of the paper, read the paper comments, and figured that there were holes in the whole approach. If my revisions had been available it seems like there still would have been issues, unless I had already gotten the final approval for the paper.

  • Reviewer 2: Thumbs down- although maybe not with the annoyed thrusting motions that the first reviewer was presumably making.

Finally, the third reviewer (contrary to scientific lore) was the most gentle.

A recent publication is suggested by the PI as a source of details, but there aren‟t many in that manuscript either.

I’m a little puzzled about this since the paper is pretty comprehensive. But maybe this is an effect of reading the first version, not the final version. So I would call this neutral on the approach.

  • Reviewer 3: No decision.


The takeaway from this gambit is mixed.

I think if it had been executed better (by me) I could have gotten the final approval through by the time the grant reviewers were looking at it and then a lot of the hesitation and negative feelings would have gone away. Of course, this would be dependent on having paper reviewers that were as quick as those that I got- which certainly isn’t a sure thing.

I think that the views of biologists on preprints, post-publication review, and other ‘alternative’ publishing options are changing. Hopefully more biologist will start using these methods- because, frankly, in a lot of cases they make a lot more sense than the traditional closed-access, non-transparent peer review processes.

However, the field can be slow to change. I will probably try this, or something like this, again. Honestly, what do I have to lose exactly? Overall, this was a positive experience and one where I believe I was able to make a contribution to science. I just hope my next grant is a better substrate for this kind of experiment.

Other posts on this process:



Writing Yourself Into A Corner

I’ve been fascinated with the idea of investment, and how it can color your thoughts, feelings, and opinions about something. Not the monetary sense of the word (though probably that too) but the emotional and intellectual sense of the word. If you’ve ever been in a bad relationship you might have fallen prey to this reasoning- “I’m in this relationship and I’m not getting out because reasons so admitting that’s it’s absolutely terrible for me is unthinkable so I’m going to pretend like it’s not and I’m going to believe that it’s not and I’m going to tell everyone that I’m doing great”. I really believe this can be a motivating factor for a big chunk of human behavior.

And it’s certainly a problem in science. When you become too invested in an idea or an approach or a tool- that is, you’ve spent a considerable amount of time researching or promoting it- it can be very difficult to distance yourself from that thing and admit that you might have it wrong. That would be unthinkable.

Sometimes this investment pitfall is contagious. If you’re on a project working together with others for common goals the problem of investment can become more complicated. That is, if I’ve said something, and some amount of group effort has been put into this idea, but it turns out I was wrong about it, it can be difficult to raise that to the rest of the group. Though, I note, that it is really imperative that it is raised. This can become more difficult if the ideas or preliminary results you’ve put forward become part of the project- through presentations made by others or through further investment of project resources to follow up on these leads.

I think this sometimes happens when you’re writing an early draft of a document- though the effect can be more subtle here. If you write words down and put out ideas that are generally sound and on-point it can be hard for you, or others who may edit the paper after you, to erase these. More importantly a first draft, no matter how preliminary or draft-y, can establish an organization that can be hard to break. Clearly if there are parts that really don’t work, or don’t fit, or aren’t true, they can be removed fairly easily. The bigger problems lie in those parts that are *pretty good*. I’ve looked back at my own preliminary drafts and realized (after a whole lot of work trying to get things to fit) that the initial overall organization was somehow wrong- and that I really need to rip it all apart and start over, at least in terms of the organization. I’ve also seen this in other people’s work, where something just doesn’t seem right about a paper, but I really can’t place my finger on what- at least not without a bunch of effort.

Does this mean that you should very carefully plan out your preliminary drafts? Not at all. That’s essentially the route to complete gridlock and non-productivity. Rather, you should be aware of this problem and be willing to be flexible. Realize that what you put down on the paper for the first draft (or early versions of analysis) is subject to change- and make others you are working with aware of this explicitly (simply labeling something as “preliminary analysis” or “rough draft” isn’t explicit enough). And don’t be afraid to back away from it if it’s not working out. It’s much better if that happens earlier in the process than later- that is, it’s better to completely tear down a final draft of a paper than to have reviewers completely miss the point of what you’re trying to say after you’ve submitted it.


Your Manuscript On Peer Review

I’m a big fan of peer review. Most of the revisions that reviewers suggest are very reasonable and sometimes really improve the manuscript. Other times it doesn’t seem to work that way. I’ve noticed this is especially true when the manuscript goes through multiple rounds of peer review at different journals. It can become a franken-paper, unloved by the very reviewers who made it.

Proposal gambit – Update 1

Last week I posted about my strategy for a proposal I’m just submitting. Pretty simple really, just using a publication in a post-publication peer review journal (F1000 Research) as the crucial piece of my preliminary data in my grant. Here’s an update on the process.

So, if you’re going to predicate an R01 submission on having a citation to a paper with a crucial set of preliminary data in it… don’t leave it until the last minute. I submitted my paper to F1000 Research on Thursday (one week prior to the submission date for my grant). They responded very quickly – next day, with requests for some minor changes and to send the figures separately (I had included them in the document). No problems, but then the weekend came up and I ended up getting everything back to them on Sunday evening. Fine. Monday came and went and I didn’t have a link. Also on Monday I was surprised because I was erroneously told that I had to have the absolute final version of my grant to our grants and contracts office that day. With no citation. I scrambled to make myself an arXiv account so that I could get it out that way (a good thing in any case). But turns out it was incorrect and I could still make minor modifications after that.

So yesterday (Tuesday) I pinged F1000 Research, politely and with acknowledgment that this was a short turnaround time, and mentioned that I wanted to put the citation in the grant. They replied on Wednesday morning apologizing for the delay (nice, but there was no delay- I was really trying to push things fast) and saying that the formatted version should be ready in a couple of days and GIVING ME A DOI for the paper! Perfect. That’s what I really needed to include in the grant.

So today the updated grant was actually submitted- a whole day early, probably a first. Now it’s just a matter of settling in until June when it will be reviewed. Of course, I still need to get my paper reviewed, but I think that won’t be a huge problem.

Overall this process is going swimmingly. And I’ve been really pleased with my interactions with F1000 Research so far.


Well, there probably ARE some exceptions here.

Well, there probably ARE some exceptions here.

So I first thought of this as a funny way of expressing relief over a paper being accepted that was a real pain to get finished. But after I thought about the general idea awhile I actually think it’s got some merit in science. Academic publication is not about publishing airtight studies with every possibility examined and every loose end or unconstrained variable nailed down. It can’t be. That would limit scientific productivity to zero because it’s not possible. Science is an evolving dialogue, some of it involving elements of the truth.

The dirty little secret (or elegant grand framework, depending on your perspective) of research is that science is not about finding the truth. It’s about moving our understanding closer to the truth. Often times that involves false positive observations- not because of the misconduct of science but because of it’s proper conduct. You should never publish junk or anything that’s deliberately misleading. But you can’t help publishing things that sometimes move us further away from the truth. The idea in science is that these erroneous findings will be corrected by further iterations and may even provide an impetus for driving studies that advance science. So publish away!

The $1000 paper

[Updated 11/2/2014 with green open access and NIH PubMed central caveats]

Anyone familiar with the debate around open access scientific journals knows that it can be expensive to publish your work there (see this list of some publication charges). In one model of open access publication the cost is shifted to the authors, who are usually funding publications from their grant money, and those charges can be in the thousands of US dollars per paper. The Public Library of Science (PLoS) journals charge between $1300 and $2900 per article, though they have a program for partial to full coverage of these charges. The result is that anyone can access, download, and read the paper free of charge opening up the research to a much wider audience.

During the Twitter discussion of alternative scientific metrics spawned by the so-called “Kardashian index” paper (see my post here) some metrics regarding publishing were suggested. One that was suggested to me (though unfortunately who suggested it is now lost in my Twitter feed- sorry) was to create a metric that calculated how expensive a paper would be to read, if you didn’t have institutional or other subscriptions to the publishers.

Here are the assumptions used:

  1. No access to any subscriptions
  2. You would purchase every paper/chapter cited in the paper
  3. You would pay non-student prices (where applicable)
  4. You’d buy the book if you couldn’t purchase individual chapters
  5. Updated! Pointed out by  that I forgot one very important caveat. Many of these for-pay papers may be available as “green open access” (self archiving their own publications) or by requirements such as those imposed by the NIH that require deposition of papers in the PubMed Central repository.

This is actually an interesting idea – and it’s only taken me about 5 months to get to it but I calculated numbers for three papers (see Table and full spreadsheet here).

The bottom line is that it would be EXPENSIVE to read a single paper this way, over $1000 for each paper (with the caveat that I’ve only looked at 3 papers total).

 Table showing cost of citations for three papers
Summary Journal Total number OA Average cost Total cost
Paper 1 PLoS Computational Biology 37 5 $38.43 $1,422.05
Paper 2 Nature 27 3 $27.87 $752.50
Paper 3 Journal of Bacteriology 41 3 $38.71 $1,587.22

This has a linear relationship with the number of citations in the paper as demonstrated in this graph (again, small sample size).


Of course, this is mostly an academic exercise (like most things I do- I’m an academic) since nobody reads every citation and most people who want to read specific citations would have access to institutional subscriptions. However, it points out a hidden cost to research publication that (I don’t think) is thought about by most researchers.

It would be fairly simple to code up a calculator for this metric given that many journals are published by the same publishers who have pretty consistent pricing. But I’ve got to get back to work now and publish more papers.



Literature Search Party

Continuing on my adventure metaphor theme: has this ever happened to you? You have a great idea, it’s brilliant, it’s revolutionary, it’s a thing that will change the way that people think about other things. You work on it, sometimes feverishly. And get… great results! Then you think, “hey – wait a minute. If this is such a great idea and so simple, why hasn’t anyone ever thought of it before?” Pause about 10 minutes. “Ohhhhhh… no. They probably have.” A quick PubMed search turns up that seminal paper from 1995 demonstrating what you’ve just ‘discovered’. My diagram on how to do science highlights this point.

Anyway, why does this problem happen and how can you avoid it. I don’t have the answers but here are some general ideas.

For me this often happens because, in coming up with a brilliant new idea you’re pushing your knowledge and experience past it’s limits. In the early stages this means that your ideas are not very well formed; you don’t have a clear idea of what you’re thinking about and how it might relate to other things. And you don’t know the area you’re moving in to. So even doing a literature search at this point can be useless. I’ve had the situation where what I was searching for actually had been done before, but I didn’t know what to call it- so PubMed was useless.

After you’ve started to get some legs to the project, maybe doing a few tests to see if it would even work and getting positive results, excitement can take over. Then you just want to get through it and get the good results. Even then you may not be able to see your idea in a greater context to be able to know what to look for.

Finally, in the later stages of the project you can suffer from “investment blindness”. You may ignore the issue of searching the literature because what if you found that you weren’t doing something new? You’d put SO much work in to it, it would be unthinkable to have to abandon it all! And you’re on a roll- the good results are coming in, the implications are starting to fall into place, and the shape of the thing, the idea you’ve had, is starting to make itself clear. It’s generally at this point that that creeping, nagging, suspicious feeling comes up. Yep, you’re pretty sure somebody MUST have done this before.

Sometimes you’re wrong. Other times you’re right, but the spin you’ve put on things and the results you’ve gotten are actually novel and you can still get a story out (this is the most common actually). Then there are the times when there’s just nothing you can do. Your exact idea has been done somewhere else and published in Nature or Science or Cell, Nature, and Science.

I guess the idea is that you know your field so well that you can see the gaps and know when you are trying something new. That’s true of a number of different projects I’ve initiated. Generally, these are not the most interesting or groundbreaking. Sometimes they’re downright boring, small steps forward.

How can you prevent this? I guess by being aware of that three-step progression I outlined above, and trying at each step to do your literature searches with that in mind. Also, be pessimistic: always start from the point of view that someone has done it before. You’re then not surprised if they have done it, and you can start to evaluate how different and novel your approach is from theirs. Approaching your literature search from the point of view that you’re looking for something will make it more likely that you will find something.

Also, consult friends and colleagues who are working in similar areas. Sometimes they may know what you’re talking about – that is, that someone has already done and they know the name of what it is you’re doing. Sometimes they might just be able to provide you with a sounding board for your idea that will allow you to clarify your thoughts.

Above all, be flexible. If it turns out that someone has done it before read their paper carefully and any follow-on papers you can find. Look for the gaps and ask how what you’ve done can answer a critical question they’ve left open.

Dude. You want a beer or something? It's hot work making it all the way up here.

Dude. You want a beer or something? It’s hot work making it all the way up here.