This is so true it hurts.
So you toil for 4+ years in graduate school, 4+ years as a post-doc, land your first academic gig. Now you get to do all this awesome science, right? Well, sorta…
Last spring I posted about a proposal I’d put in where I’d published the key piece of preliminary data in F1000 Research, a journal that offers post-publication peer review.
The idea was that I could get my paper published (it’s available here) and accessible to reviewers prior to submission of my grant. It could then be peer-reviewed and I could address the revisions after that. This strategy was driven by the lag time between proposal submission and review for NIH, which is about 4 months. Also, it used to be possible to include papers that hadn’t been formally accepted by a journal as an appendix to NIH grants. This hasn’t been possible for some time now. But I figured this might be a pretty good way to get preliminary data out to the grant reviewers in a published form with quick turnaround. Or at least that you could utilize that lag time to also function as review time for your paper.
I was able to get my paper submitted to F100 Research and obtained a DOI and URL that I could include as a citation in my grant. Details here.
The review for the grant was completed in early June of this year and the results were not what I had hoped- the grant wasn’t even scored, despite being totally awesome (of course, right?). But for this post I’ll focus on the parts that are pertinent to the “gambit”- the use of post-publication peer review as preliminary data.
The results here were mostly unencouraging RE post-publication peer review being used this way, which was disappointing. But let me briefly describe the timeline, which is important to understand a large caveat about the results.
I received first-round reviews from two reviewers in a blindingly fast 10 and 16 days after initial submission. Both were encouraging, but had some substantial (and substantially helpful) requests. You can read them here and here. It took me longer than it could have to address these completely – though I did some new analysis and added additional explanation to several important points. I then resubmitted on around May 12th or so. However, due to some kind of issue the revised version wasn’t made available by F1000 Research until May 29th. Given that the NIH review panel met in the first week of June it is likely that the reviewers didn’t see the revised (and much improved version). The reviewers then got back final comments in early June (again- blindingly fast). You can read those here and here. The paper was accepted/approved/indexed in mid-June.
The grant had comments from three reviewers and each had something to say about the paper as preliminary data.
The first reviewer had the most negative comments.
It is not appropriate to point reviewers to a paper in order to save space in the proposal.
Alone this comment is pretty odd and makes me think that the reviewer was annoyed by the approach. So I can’t refer to a paper as preliminary data? On the face of it this is absolutely ridiculous. Science, and the accumulation of scientific knowledge just doesn’t work in a way that allows you to include all your preliminary data completely (as well as your research approach and everything else) in the space of 12 page grant. However, their further comments (which directly follow this one) shed some light on their thinking.
The PILGram approach should have been described in sufficient detail in the proposal to allow us to adequately assess it. The space currently used to lecture us on generative models could have been better used to actually provide details about the methods being developed.
So reading between the (somewhat grumpy) lines I think they mean to say that I should have done a better job of presenting some important details in the text itself. But my guess is that the first reviewer was not thrilled by the prospect of using a post-publication peer reviewed paper as preliminary data for the grant. Not thrilled.
- Reviewer 1: Thumbs down.
Second reviewer’s comment.
The investigators revised the proposal according to prior reviews and included further details about the method in the form of a recently ‘published’ paper (the quotes are due to the fact that the paper was submitted to a journal that accepts and posts submissions even after peer review – F1000 Research). The public reviewers’ comments on the paper itself raise several concerns with the method proposed and whether it actually works sufficiently well.
This comment, unfortunately, is likely due to the timeline I presented above. I think they saw the first version of the paper, read the paper comments, and figured that there were holes in the whole approach. If my revisions had been available it seems like there still would have been issues, unless I had already gotten the final approval for the paper.
- Reviewer 2: Thumbs down- although maybe not with the annoyed thrusting motions that the first reviewer was presumably making.
Finally, the third reviewer (contrary to scientific lore) was the most gentle.
A recent publication is suggested by the PI as a source of details, but there aren‟t many in that manuscript either.
I’m a little puzzled about this since the paper is pretty comprehensive. But maybe this is an effect of reading the first version, not the final version. So I would call this neutral on the approach.
- Reviewer 3: No decision.
The takeaway from this gambit is mixed.
I think if it had been executed better (by me) I could have gotten the final approval through by the time the grant reviewers were looking at it and then a lot of the hesitation and negative feelings would have gone away. Of course, this would be dependent on having paper reviewers that were as quick as those that I got- which certainly isn’t a sure thing.
I think that the views of biologists on preprints, post-publication review, and other ‘alternative’ publishing options are changing. Hopefully more biologist will start using these methods- because, frankly, in a lot of cases they make a lot more sense than the traditional closed-access, non-transparent peer review processes.
However, the field can be slow to change. I will probably try this, or something like this, again. Honestly, what do I have to lose exactly? Overall, this was a positive experience and one where I believe I was able to make a contribution to science. I just hope my next grant is a better substrate for this kind of experiment.
Other posts on this process:
- My original Proposal gambit post
- Proposal gambit – update 1
- A press release from F1000 Research about this paper
- The published paper “Prediction of multi-drug resistance transporters using a novel sequence analysis method”
- The GitHub repository with code
- My post giving an overview of the paper, along with a fun infographic
Last week I posted about my strategy for a proposal I’m just submitting. Pretty simple really, just using a publication in a post-publication peer review journal (F1000 Research) as the crucial piece of my preliminary data in my grant. Here’s an update on the process.
So, if you’re going to predicate an R01 submission on having a citation to a paper with a crucial set of preliminary data in it… don’t leave it until the last minute. I submitted my paper to F1000 Research on Thursday (one week prior to the submission date for my grant). They responded very quickly – next day, with requests for some minor changes and to send the figures separately (I had included them in the document). No problems, but then the weekend came up and I ended up getting everything back to them on Sunday evening. Fine. Monday came and went and I didn’t have a link. Also on Monday I was surprised because I was erroneously told that I had to have the absolute final version of my grant to our grants and contracts office that day. With no citation. I scrambled to make myself an arXiv account so that I could get it out that way (a good thing in any case). But turns out it was incorrect and I could still make minor modifications after that.
So yesterday (Tuesday) I pinged F1000 Research, politely and with acknowledgment that this was a short turnaround time, and mentioned that I wanted to put the citation in the grant. They replied on Wednesday morning apologizing for the delay (nice, but there was no delay- I was really trying to push things fast) and saying that the formatted version should be ready in a couple of days and GIVING ME A DOI for the paper! Perfect. That’s what I really needed to include in the grant.
So today the updated grant was actually submitted- a whole day early, probably a first. Now it’s just a matter of settling in until June when it will be reviewed. Of course, I still need to get my paper reviewed, but I think that won’t be a huge problem.
Overall this process is going swimmingly. And I’ve been really pleased with my interactions with F1000 Research so far.
I am currently (this minute… well, not THIS minute, but just a minute ago, and in a minute) in the throes of revising a resubmission of a previously submitted R01 proposal to NIH. This proposal generally covers novel methods to build protein-sequence-based classifiers for problematic functional classes- that is, groups of proteins that have a shared function but either are very divergent in their sequence (meaning that they can’t be associated by traditional sequence similarity approaches) or have a lot of similar sequences with divergent functions (and the function that’s interesting can’t be easily disambiguated).
I got good feedback from reviewers on the previous version (though I did not get discussed- for those who aren’t familiar with the process, to get a score- and thus a chance at funding- your grant has to be in the top 50% of the grants that the review panel reads, then it moves on to actual discussion in the panel and scoring). Their main complaint was that I had not described the novel method I was proposing in sufficient detail, and so they were intrigued but couldn’t assess if this would really work or not. The format of NIH R01-level grants (12 pages for the research part) means that to provide details of methods you really need to have published your preliminary results. Also- if it’s published it really lends weight to the fact that you can do it and get it through peer review (or pay your way into a publication in an fly-by-night journal).
So anyway. I’ve put this resubmission off since last year and I’m not getting any younger and I don’t have a publication to reference on the method in the proposal yet. So here’s my gambit. I’ve been working on the paper that will provide preliminary data and it was really nearly finished it just needed a good push to get it finalized, which came in the form of this grant. My plan is to finish up the last couple of details on the paper and submit it to F1000 Research because it offers online publication immediately with subsequent peer review. I’ve been intrigued by this emerging model recently and wanted to try it anyway. But this allows me to reference the online version very soon after I upload it (maybe tomorrow) and include it as a bona fide citation for my grant. The idea is that by the time it’s reviewed (3 months hence) it will have passed peer review and will be an actual citation.
But it’s a gambit. It’s possible that the paper will still be under review or will have received harsh reviews by the time the reviewers look at it. It’s also possible that since I won’t have a traditional journal citation in text for the proposal- I’ll need to supply a URL to my online version- that the reviewers will just frown on this whole idea and it might even piss them off making them think I’m trying to get away with something (which I totally am, though it’s not unethical or against the rules in any way that I can see). However, I’m pretty sure that this is a lot more common on the CS side (preprint servers, and the like) so I’m betting on that flying.
Anyway, I’ll have an update in 3+ months on how this worked out for me. I actually have high hopes for this proposal- which does scare me a little. But I’m totally used to dealing with rejection, as I’ve mentioned before on numerous occasions. Wish me luck!
I had a dream the other night that inspired this comic. My dream was about waiting for a connecting flight. I decided to take it easy and do something fun, then realized that my flight was leaving soon and I was nowhere near the gate. Then I got on a train and realized I was going the wrong direction. Anyway, I woke up to the realization that I’d relaxed and done fun stuff most of the weekend (I did work some in the evenings) and that I had an unfinished grant that was still due this week. As it turned out I finished up my grant quite nicely despite the slacking off- or maybe even because of the slacking off. But it gave me the inspiration for this comic.
You see, writing and submitting a grant proposal is a lot like planning for a vacation that you’ll probably never get to take. The work you’re proposing should be fun and interesting (otherwise, why are you trying to get money to do it, right?) but your chances are pretty slim that you’ll ever get to do it- at least in the form that you propose it. I’ve started to think of the grant process as a long game (see this post from one DrugMonkey)- one where the act of writing a single grant is mainly just positioning for the next grant you’ll write down the line. Writing grants give you opportunity to come up with ideas, to consolidate your thoughts, and think through the science that you want to do and how you want to do it. The process can push you to publish your work so that you can cite it as preliminary data. And it can forge long-lasting collaborations that go beyond failed proposals (though funded proposals certainly help to cement these relationships in a much more sure way).
I think “A Fine Trip Spoiled” may be the title of my autobiography when I get rich and famous.
One of the great things about being a purely computational researcher is that, nowadays, my office is pretty much wherever I want it to be. I’ve got my laptop, WiFi is omnipresent, and I have noise-canceling headphones for the serious business. There are lots of reasons that I have to be at my office – meetings and increased ability to focus being primary. However, it’s not the case that you have to be purely computational to get a lot out of working in non-traditional locales. Writing is the place where we all (as researchers) can do this. Writing manuscripts and grants being the biggest time sucks. Some of you will have the ability to be flexible in your actual work time, others this might pertain mostly to the ‘extra’ work you do writing grants and papers.
So here is my random collection of thoughts on this topic.
Why take your work outside the standard work environment?
- Flexibility and efficient use of time. If you have your laptop with you you can fit in writing wherever you are (see list below). This allows you to use your time well instead of standing around checking Facebook on your phone. Not all writing work is suited for the short bits of time (probably no less than about 20-30 minutes at a time) but if you plan what to work on you can get a lot done this way. If you don’t have your laptop a surprising amount of work can get done with just a pen and paper.
- Freedom from distraction. OK, a coffee shop can be a pretty distracting place, that’s a given. But sometimes being in your office can be pretty distracting too. People stop by to chat for a minute, phones ring, drawers need organizing, etc. If you can ignore the distractions outside your office (wherever you’re choosing to work) then this can be a productive way to go. Also, try working somewhere WITHOUT WiFi (it can be done)- and cut out the social media chatter.
- Creative stimulation. Changing your work environment drastically can give you a shot of creative energy. It can be refreshing wot work outside at a park, or while enjoying a glass of your favorite beverage at a cafe or bar.
What to work on?
- Reviewing papers/grants
- Catching up on answering emails
- Reading papers- no laptop required
- Planning and outlining- also no laptop required, use a pen and notebook
Where can you do this?
- Coffee shop. Everyone pretty much knows about this one. Can be distracting, but find a quiet corner and bring headphones. Also, try not to drink 15 double espressos while you’re there (not that I would have ANY experience with that)
Bar/pub. These can be awesome places to work- probably not on a Friday or Saturday night, but other times. Many have WiFi and they have BEER! Also, try not to drink 8 beers while you’re there. Alcohol is actually a consideration since it can affect your motivation pretty severely. Ordering ONE beer and some food works OK for me, but certainly use your best judgement- and they will always have alternate non-alcoholic beverage options.
- Public library. This is really just a no-brainer. No cost (though many libraries have coffee shops attached and allow you to bring covered cups in), free WiFi, lots of sitting areas, quiet atmosphere, surrounded by the smell of knowledge.
- Park. Working outside is sometimes really nice in nice weather. If you’re lucky enough to have workable weather (not too hot, not too cold, not too windy or rainy) then find a table in the shade and settle in. I’ve never found this particularly effective myself, though the idea is wonderful, but I’m sure it could work for others.
- Doctor/dentist office, DMV, etc. This option is one I use quite a bit, but it only works for things that you can do a little bit on before being interrupted. I find that making todo lists and outlines work well here. Also reading background material can also work well.
- Car. Not while you’re driving! I mean if you’re sitting and waiting for something or someone this can be a good time too.
- Public transportation. When I was in Seattle I rode the commuter train in from Everett to work several times a week. A great place to work. An hour of uninterrupted time while beautiful countryside rolls by. Buses can work too, though not always for actual writing since often they bump and move too much for a laptop. Subways/metros also work well. Of course, this is pretty dependent on the density of people. It’s really hard to do anything productive when you have an elbow in your face and about 6 inches of standing room.
Airplane/airport. So much wasted time in airports- which are great places to work if you find the right spots. Airplanes can be a bit problematic in terms of an actual laptop (I find I can do it if I type like a T-rex) but I bring papers to read and a notebook to do planning and write ideas. In airports try to find places where there aren’t many people- away from your departing gate if you have time. More chance of getting a power outlet and fewer distractions. If you’re really in need of an outlet try looking in places where other people aren’t going to be sitting (hallways and walkways) and sit on the floor- it can be done.
- Hotel. Also in the traveling realm. Hotels can be excellent places to write. Free from a lot of the distractions and obligations of home and office. If you have extra time after a day at a conference or between sessions or before you catch your plane- use it. Many hotels are set up with desks, comfy chairs, outlets, coffee makers, and WiFi. When I travel to the east coast and my return flight is early I will frequently work through the night. Not for everyone, but I’m a night owl and I find it easier to do this (sometimes) than to sleep for a few hours then drag myself out of bed at 5 AM (3 AM my time) to get to the airport. Also, no danger of oversleeping – unless of course you accidentally crash. So if you do this make sure to arrange a wake up call and set an alarm for backup.
- Other locations. Be on the lookout for other opportunities. I have worked on a grant while pouring wine for a wine tasting at a friend’s house (not a wine-tasting party, mind you- this was a professional activity, so quite a bit of down time). That was pretty epic really but it still didn’t get my grant funded.
Just got word of this from the Twitters- the NIH is announcing rollout of a new biosketch format for grant applications. I thought I’d summarize the information about it here to make things easy.
- Starting for grants that would be funded in 2016 (so anything you apply for in 2015 will have this, essentially)
- Will be a five page limit (as opposed to 2 page for current format)
- Will NOT include a ‘bare’ list of 15 publications
- Now includes a new list of up to 5 of your “contributions to science”, which can include up to 4 citations each (your own presumably). So that’s a total of 20 citations you can have.
- You will be able to include a link to your full citation list in an online database (NIH resources sciENcv or My Bibliography are mentioned but could be others I guess)
- Here’s a template that includes an example new biosketch
- More information can be found in the announcement here from the NIH on Salley Rockey’s blog
- As it’s being rolled out the requirement for the new format will be stated in the RFA – SO LOOK FOR THIS IN ALL RFAs from here out. Two pilots already have it RFA-CA-13-501 and RFA-CA-13-502
The upsides: In my opinion the addition of the contributions to science will be the biggest one and should allow you to really highlight your publications (or lack thereof) in appropriate context. Plus space for moar buzzwords!
The downsides: It’s gonna be a pain to write (like a mini grant in there) and reviewers won’t read this one either.
The famous Spinal Tap quote (see the video here) is great because Nigel is explaining how his amp is better than other rockers since you can turn it up to 11. “Why don’t you just make ten louder and make ten be the top number and make that a little louder?” asks the mockumentarian Rob Reiner. Good question.
The humor in this scene reminds me strongly of this recent paper on the introduction of an artificial nucleotide base pair into a bacteria. Essentially they got a bacteria to incorporate an artificial nucleotide pair into its DNA, it replicates stably (that is, the new pair stays in the bacterial DNA for generations), and its not removed by DNA repair mechanisms that look for problems in the DNA. Novel nucleotides are not new- researchers have created a large number of these and incorporation into DNA has been done in limited ways in test tube (in vitro) systems, not in a living organism. This is really a pretty cool technical achievement – the researchers had to solve a number of complicated problems to get this to work and, more importantly, it’s likely that they got very lucky with their choices (where ‘luck’ here is a combination of knowledge, trial and error, and actual bona fide luck).
The paper itself doesn’t really overstate the implications of this paper. The only implications statement in the paper comes at the end:
In the future, this organism, or a variant with the UBP incorporated at other episomal or chromosomal loci, should provide a synthetic biology platform to orthogonally re-engineer cells, with applications ranging from site-specific labelling of nucleic acids in living cells to the construction of orthogonal transcription networks and eventually the production and evolution of proteins with multiple, different unnatural amino acids.
And all of this seems very reasonable and potentially achievable.
However, as happens with many high profile papers, the press coverage I’ve seen on this is terrible. From Gizmodo touting that “scientists have created alien DNA” (only for a very limited definition of ‘alien’) to New Scientist stating that researchers have expanded the ‘genetic code’ of a bacterium (not really, a code needs to have meaning- that is, to be translatable into something that has meaning, this advance doesn’t yet). However, perhaps the most troubling is coverage from NPR, largely based on an interview with the senior author of the paper. In this piece Floyd Romesburg introduces a simple, and largely apt analogy, for what his work has done:
Maybe you get three consonants and one vowel. Maybe there are some words you can write and you can string them together to make, sort of, primitive stories. But if you could have a couple extra letters, there’s more that you could write. Having the ability to store increased information would allow you to write more interesting words, bigger words, more complicated words, more nuanced words, better stories. – from NPR interview
He goes on to say:
It’s not so much that I think life needs more genetic information but I think that there are things that we could really learn and drugs that could be developed by getting cells to be able to do more
So it’s not a bad analogy, but one where he’s essentially said: “This one goes to 11.” . And it points out exactly why this work is so limited in its implications (exactly the opposite of what he’s trying to point out by using it BTW). They have added, in a very constrained and limited way, added two letters to the standard ATCG alphabet used by nearly all life. Will this introduce the ability to build more complex or useful biological systems? Not in the slightest. Imagine that we added a couple of letters to the English alphabet. Now we give those extra letters to someone like William Shakespeare. Does anyone think that he would be able to do more with more letters? Write better, more complex, more interesting, more profound plays or sonnets? No, of course not. Even if you gave him a whole bunch of new words that contained the new letters (which they haven’t done at all in this paper- they haven’t actually introduced this new addition into the code itself, only into the alphabet), he would likely have produced very similar works. Maybe those works would be slightly shorter, but they would NOT contain more information. Adding a letter to the alphabet doesn’t increase the information or the complexity of the code. It just doesn’t. The computer that I’m writing this on is based on a binary alphabet (1 and 0 are the only letters it uses) and yet I’m able to put these together (with the help of the underlying OS and software) into complicated and information-rich constructs. Having a computer based on 0 , 1 AND 2 wouldn’t help me write this post any more better [sic].
The idea that this would lead to development of new drugs, new forms of life, new biology is a far, far, far distant stretch that causes confusion and even fear. The problem here is not purely driven by misunderstanding and misrepresentation of the work by scientific journalists (though it looks like there’s some of that) but from the actions and statements of the senior author himself. As I mentioned, this is a sound paper and is pretty interesting- a technical achievement. It may indeed lead to some interesting new discoveries and methodologies that may be broadly applicable. But it’s not alien DNA and it’s not going to help us cure cancer with new drugs, and it’s not going to provide the ability to make the biology more complex, but it might make our rocker friends green with jealousy when we reveal that we have six nucleotide bases compared to their paltry four bases because “these go to eleven” (Nigel Tufnel)
Recent experience talking here (interested, non-Science types, there’s a short explanation to give context at the end).
- Boundless, unfounded optimism: Yes! You’ve just had the greatest idea EVER! Sure it’s lacking a few details, but the reviewers are gonna love it! The whole idea is there, it just needs a little writing and you’ll be done. This time, for sure you’ll be able to knock it out quick and without much effort since it’s such a great idea. Just great.
- Period of procrastination: Since it’ll be so easy you don’t have to worry about it right now. Sit back and relax since you’ve still got loooooooooads of time before this baby is due. Maybe talk with some people about how you’ll need to start working on it sometime, and about how great it is, but that you’re not worried about it now because it’s not due anytime soon.
- Panic: HOLY SH*T! What day is it? How did that happen?! Can it be that the due date is actually that close? It’s impossible- better check the calendar and the call a few more times. Gulp.
- Outline stage: What were those great ideas you had early on? Oh yeah- put those down on paper. Wow. Somehow they seemed a lot grander in your head. They look a little thin on paper actually. Are you sure that’s all? Maybe you forgot that one cool part.
- Cut-and-paste: Take text from wherever you can, manuscripts, other proposals, your grocery lists, and put it down in the sections that kinda make sense. It makes you feel a lot better to have something on paper and you can fill it in later, right?
- Sculpting: Now take all that great raw material you have and make it look like Michaelangelo’s David. Hmmmm…. that’s not quite right- it looks more like Frankenstein’s monster’s ugly cousin. Well, keep sculpting- the more sculpting the better. Put hours into that transition between Aims 2 and 3. It’s super important that you phrase it perfectly so you can rip it all apart in step 8.
- Imposter’s syndrome: “What am I even doing here?” You’ve got the simplest idea coupled with the most contrived, Rube Golbergian implementation. Nothing really fits together into a coherent picture. And it has no point. You wouldn’t even believe this crap if you were the reviewer.
- Demolition derby: A flash of inspiration and you see how all the pieces fit together. Unfortunately it requires ripping everything apart and rearranging your aims, renumbering your figures, and stitching it all back together to make it look like you meant it to go that way the whole time. Oh, and you’ve got a little under 24 hours to accomplish this.
- Amazing period of focus and productivity: Where did that come from and why has it been hiding for so long?! You just knocked out 5 pages of text in 3 hours and it looks great. It would have been REALLY nice to be at this point a wee bit earlier in the process.
- Rose colored glasses: Yes. It’s finished and it’s perfectly wonderful! There could be some blemishes, or some outright holes. It doesn’t matter now ’cause you can’t do a damn thing about it. Too late. So it’s all wonderful and the best idea ever!
- Last minute scramble: Generally it involves either figure or citation formatting- formatting of some sort. And it’s absolutely incomprehensible why it’s screwing up now. Why won’t EndNote work the way it was designed to? You’ve got 2 hours before your grant administrator has a friggin’ stroke and you can’t get X to work correctly (where ‘X’ is some formerly inconsequential software feature that just wants attention- but you can’t ignore it at this point)
- Now where did that focus and productivity go? It’s off. Submitted safely at the funding agency. A warm sense of accomplishment in your belly. The drive to do more still rattling around in your bones. OK- now that you found that amazing kernel of productivity that you didn’t remember existed it should be easy to get all that stuff done that you put off during your proposal process. If only you weren’t so tired. And apathetic. *Sigh*
Here’s a little background for those of you not so “fortunate” to be tied to the whims and whimsey of academic funding (and, importantly, are still reading this). Academic researchers (and those in national labs, like me) have to, for the most part, fund the research they do by applying for grants from funding agencies. These are governmental agencies like the National Institutes of Health (NIH), National Science Foundation (NSF), the Department of Energy (DOE), and others, or non-governmental funding agencies (private philanthropic funds, the Gates Foundation, etc.) and they issue ‘calls’ for proposals that specify the kinds of research they’re looking for. There are deadlines associated with these calls and some recur on a regular basis but others are single-shot. After you submit your proposal the agency will review it for merit using criteria generally outlined in the call. NIH does this with an established peer review process. If you are so lucky to be funded then you get to do what you propose- and live for a little longer in the academic research game. Yay!