Proposal gambit – Betting the ranch

Last spring I posted about a proposal I’d put in where I’d published the key piece of preliminary data in F1000 Research, a journal that offers post-publication peer review.

The idea was that I could get my paper published (it’s available here) and accessible to reviewers prior to submission of my grant. It could then be peer-reviewed and I could address the revisions after that. This strategy was driven by the lag time between proposal submission and review for NIH, which is about 4 months. Also, it used to be possible to include papers that hadn’t been formally accepted by a journal as an appendix to NIH grants. This hasn’t been possible for some time now. But I figured this might be a pretty good way to get preliminary data out to the grant reviewers in a published form with quick turnaround. Or at least that you could utilize that lag time to also function as review time for your paper.

I was able to get my paper submitted to F100 Research and obtained a DOI and URL that I could include as a citation in my grant. Details here.

The review for the grant was completed in early June of this year and the results were not what I had hoped- the grant wasn’t even scored, despite being totally awesome (of course, right?). But for this post I’ll focus on the parts that are pertinent to the “gambit”- the use of post-publication peer review as preliminary data.

The results here were mostly unencouraging RE post-publication peer review being used this way, which was disappointing. But let me briefly describe the timeline, which is important to understand a large caveat about the results.

I received first-round reviews from two reviewers in a blindingly fast 10 and 16 days after initial submission. Both were encouraging, but had some substantial (and substantially helpful) requests. You can read them here and here. It took me longer than it could have to address these completely – though I did some new analysis and added additional explanation to several important points. I then resubmitted on around May 12th or so. However, due to some kind of issue the revised version wasn’t made available by F1000 Research until May 29th. Given that the NIH review panel met in the first week of June it is likely that the reviewers didn’t see the revised (and much improved version). The reviewers then got back final comments in early June (again- blindingly fast). You can read those here and here. The paper was accepted/approved/indexed in mid-June.

The grant had comments from three reviewers and each had something to say about the paper as preliminary data.

The first reviewer had the most negative comments.

It is not appropriate to point reviewers to a paper in order to save space in the proposal.

Alone this comment is pretty odd and makes me think that the reviewer was annoyed by the approach. So I can’t refer to a paper as preliminary data? On the face of it this is absolutely ridiculous. Science, and the accumulation of scientific knowledge just doesn’t work in a way that allows you to include all your preliminary data completely (as well as your research approach and everything else) in the space of 12 page grant. However, their further comments (which directly follow this one) shed some light on their thinking.

The PILGram approach should have been described in sufficient detail in the proposal to allow us to adequately assess it. The space currently used to lecture us on generative models could have been better used to actually provide details about the methods being developed.

So reading between the (somewhat grumpy) lines I think they mean to say that I should have done a better job of presenting some important details in the text itself. But my guess is that the first reviewer was not thrilled by the prospect of using a post-publication peer reviewed paper as preliminary data for the grant. Not thrilled.

  • Reviewer 1: Thumbs down.

Second reviewer’s comment.

The investigators revised the proposal according to prior reviews and included further details about the method in the form of a recently ‘published’ paper (the quotes are due to the fact that the paper was submitted to a journal that accepts and posts submissions even after peer review – F1000 Research). The public reviewers’ comments on the paper itself raise several concerns with the method proposed and whether it actually works sufficiently well.

This comment, unfortunately, is likely due to the timeline I presented above. I think they saw the first version of the paper, read the paper comments, and figured that there were holes in the whole approach. If my revisions had been available it seems like there still would have been issues, unless I had already gotten the final approval for the paper.

  • Reviewer 2: Thumbs down- although maybe not with the annoyed thrusting motions that the first reviewer was presumably making.

Finally, the third reviewer (contrary to scientific lore) was the most gentle.

A recent publication is suggested by the PI as a source of details, but there aren‟t many in that manuscript either.

I’m a little puzzled about this since the paper is pretty comprehensive. But maybe this is an effect of reading the first version, not the final version. So I would call this neutral on the approach.

  • Reviewer 3: No decision.

Summary

The takeaway from this gambit is mixed.

I think if it had been executed better (by me) I could have gotten the final approval through by the time the grant reviewers were looking at it and then a lot of the hesitation and negative feelings would have gone away. Of course, this would be dependent on having paper reviewers that were as quick as those that I got- which certainly isn’t a sure thing.

I think that the views of biologists on preprints, post-publication review, and other ‘alternative’ publishing options are changing. Hopefully more biologist will start using these methods- because, frankly, in a lot of cases they make a lot more sense than the traditional closed-access, non-transparent peer review processes.

However, the field can be slow to change. I will probably try this, or something like this, again. Honestly, what do I have to lose exactly? Overall, this was a positive experience and one where I believe I was able to make a contribution to science. I just hope my next grant is a better substrate for this kind of experiment.

Other posts on this process:

 

 

Asked and answered: Computational Biology Contribution?

So someone asked me this question today: “as a computational biologist,how can you be useful to the world?”. OK so they didn’t ask me, per se, they got to my blog by typing the question into a search engine and I saw this on my WordPress stats page (see bottom of this post). Which made me think- “I don’t know what page they were directed to- but I know I haven’t addressed that specific question before on my blog”. So here’s a quick answer, especially relevant since I’ve been talking with CS people about this at the ACM-BCB meeting the last few days.

As a computational biologist how can you be useful to the world?

  1. Choose your questions carefully. Make sure that the algorithm you’re developing, the software that you’re designing, the fundamental hypothesis that you’re researching is actually one that people (see collaborators, below) are interested in and see the value in. Identify the gaps in the biology that you can address. Don’t build new software for the sake of building new software- generally people (see collaborators) don’t care about a different way to do the same thing, even if it’s moderately better than the old way.
  2. Collaborate with biologists, clinicians, public health experts, etc. Go to the people who have the problems. What they can offer you is focus on important problems that will improve the impact of your research (you want NIH funding? You HAVE to have impact and probably collaborators). What you can give them is a solution to a problem that they are actually facing. Approach the relationship with care though since this is where the language barrier between fields can be very difficult (a forthcoming post from me in the near future on this). Make sure that you interact with these collaborators during the process- that way you don’t go off and do something completely different than what they had in their heads.
  3. In research be rigorous. The last thing that anyone in any discipline needs is a study that has not considered validation, generalizability, statistical significance, or having a gold-standard or reasonable facsimile thereof to compare to. Consider collaborating with a statistician to at least run your ideas by- they can be very helpful, or a senior computational biologist mentor.
  4. In software development be thoughtful. Consider robustness of your code- have you tested it extensively? How will average users (see collaborators, above) be able to get their data into it? How will average users be able to interpret the results of your methods? Put effort into working with those collaborators to define the user interface and user experience. They don’t (to a point) care about execution times as long as it finishes in a reasonable amount of time (have your software estimate time to completion and display it) and it gives good results. They do care if they can’t use it (or rather they completely don’t care and will stop working with you on the spot).
  5. Sometimes people don’t know what they need until they see it. This is a tip for at least 10th level computational biologists (to make a D&D analogy). This was a tenet of Steve Jobs of Apple and I believe it to be true. Sometimes, someone with passion and skill has to break new ground and do something that no one is asking them to do but that they will LOVE and won’t know how they lived without it. IT IS HIGHLY LIKELY THAT THIS IS NOT YOU. This is a pretty sure route to madness, wearing a tin hat, and spouting “you fools! you’ll never understand my GENIUS”- keep that in mind.
  6. For a computational biologist with some experience make sure that you pass it along. Attend conferences where there are likely to be younger faculty/staff members, students, and post-docs. Comment on their posters and engage. When possible suggest or make connections with collaborators (see above) for them. Question them closely on the four points above- just asking the questions may be an effective way of conveying importance. Organize sessions at these conferences. In your own institution be an accessible and engaged mentor. This has the most potential to increase your impact on the world. It’s true.

Next week: “pathogens found in confectionary” (OK- probably not going to get to that one, but interesting anyway)

People be searchin'

People be searchin’

Career Strategy

Including this on my actual CV could be a problem though...

Including this on my actual CV could be a problem though…

I Tweeted this last week as a brilliant idea for a career move.

The largest barrier I see to this actually working is that it will be difficult to include it on my  printed CV. And have the same effect at least. I’m working on it. Also it has a similar bad taste as the “research” group that bought their own journal to publish their ridiculous paper on Sasquatch DNA.

Turns out, of course, that I’d been beaten to the idea by actual publishers:  

Screen Shot 2014-09-02 at 9.30.18 AM home_cover Screen Shot 2014-09-02 at 9.32.46 AM

As pointed out:

I think I’ve still got something novel with the whole “Other High Impact” journal idea. *BRILLIANT!!!*

Important Tweeting

This week’s comic made me think about meeting attendance and engagement. I’ve seen lots of people who attend meetings but who aren’t really there. They’re checking their phones or working on their computers. And I’ve been one of those people on a fair number of times. However, I’ve started to make changes around this. The idea is that I’m at the meeting for a reason, or sometimes several reasons. I need to present myself well and to get the most out of the meeting as I can. Otherwise I probably shouldn’t be showing up. There are required meetings, meetings you initiate, and meetings where you have no idea why you’re there- so there’s certainly a continuum, and my thoughts here probably don’t apply in every situation.

It’s important to be engaged and present yourself that way. I depend a lot on other people to accomplish my work, feed me data, and provide funding for me. People who may want to work with you, give you opportunities, fund you, etc. are tuned in to your level of engagement. They’re also not impressed with how you’re so busy that you have to be working on your computer during the entire meeting.

Part of engagement is about appearances- you should be presenting yourself in a way that doesn’t make people think you are ignoring them or tuning them out. Even making notes in a notebook (like, the paper kind. You know, with a pen.) is better than being behind your laptop. Cellphones are marginally better since they are less intrusive and obvious than a laptop, but still send the message that you’re not paying attention. And you probably aren’t. But this can be very important.

What you get out of a meeting is also important. Clearly, if you’re not paying attention you won’t get much out of the meeting. And some meetings just aren’t set up for you to get anything out of. But often times if you’re paying attention you can get a lot out of meetings. Not just in terms of what’s actually being presented or discussed, but also how people interact with each other, how they present their work, and how engaged they are.

So next time you’re sitting down in a meeting think about WHY you’re there and how you want to present yourself.

Oops, gotta go #introuble

Oops, gotta go #introuble

The Grad Circus

This comic is an homage to my time in graduate school- stimulating and frustrating, a time for development, exploration, and maturation. Pretty much 5 years for me. It’s not strictly autobiographical, though it certainly contains elements from my experience (and my foibles). There were plenty of ways to waste time. On the other hand, if you consider your time as ‘productive’ only when you have your nose to the grindstone you will not end up learning much. Here are some of the things that I didn’t include:

  • I can remember my advisor unlocking the door to HIS office, a bit early in the morning, and finding me playing a first-person shooter on his computer. Ummmm….. oops?
  • Going ‘bowling’ with excess liquid nitrogen. We would take a dewar flask full of the stuff and roll it down the hall. Not much point, but very cool to see.
  • I once scored an entire half of a gigantic sheet cake as ‘free food’. It was left over from a retirement party somewhere in the med school. The admin from our department tipped me off and I retrieved it. Much feasting was had on that day!
  • Many a late night was spent in the lab playing video games with my best friend across the hall. We’d be yelling back and forth at 3 AM then one of us would have to stop the game to collect a sample for a time point.
  • I once found a broken VCR in a dumpster, fixed it, and then sold it for $50- which is like $500 to a grad student. I then proceeded to collect every piece of broken equipment and electronics I could get my hands on- storing it in the lab. I built a rocking platform (retailed at about $600) and repaired several pieces of lab equipment with my haul. However, it really ended up taking up a lot of space.

Graduate school was a great time and one with a lot of freedom that came with the stress. I was lucky to have a great advisor who supported me even when he thought I was goofing off (he did not support me IN the goofing off- he supported me in my career despite that I was goofing off) and a bunch of great friends who I still keep in touch with. There was so much potential in that time- the potential is still there in my career, I just need to take more time to recognize it. And maybe score some more free food too.

 

GradCircus_RedPenBlackPen

On a different note I’m pretty proud of this comic. Some of my previous ideas and concepts have been pretty complicated (see here and here). I’ve thought that I should do an outline sketch first then do a final version with more care. Both of these previous cases I started sketching and just decided to keep it as is. They’re pretty good considering. With this one I actually did sketch out an idea:

 

Initial pen sketch of ideas for the grad circus

Initial pen sketch of ideas for the grad circus

then did a pencil outline of the final

Pencil outlines for the final grad circus. Didn't include details or text.

Pencil outlines for the final grad circus. Didn’t include details or text.

then traced with pen and filled in text and details.

My wife was impressed but said, “maybe for your next one you should do something about a guy that has a full time job and three kids.” Funny. This process did end up taking me about 5 hours (and a few cramped hands) of late evening and weekend time to complete giving me new respect for cartoonists, especially Bil Keane (and now Jeff Keane), whose simple comics look so ‘easy’. These maps are hard to draw!

 

 

 

 

Academic Rejection Training

Following on my previous post about methods to deal with the inevitable, frequent, and necessary instances of academic rejection you’ll face in your career I drew this comic to provide some helpful advice on ways to train for proposal writing. Since the review process generally takes months (well, the delay from the time of submission to the time that you find out is months- not the actual review itself) it’s good to work yourself up to this level slowly. You don’t want to sprain anything in the long haul getting to the proposal rejection stage.

ThreeQuickWays

Dealing with Academic Rejection

Funny, it feels like I’ve written about exactly this topic before…

I got rejected today, academically speaking*. Again. I was actually pretty surprised at how

"Not Discussed", again

“Not Discussed”, again

nonplussed I was about the whole thing. I’ve gotten mostly immune to the being rejected- at least for grant proposals and paper submissions. It certainly could be a function of my current mid-career, fairly stable status as a scientist. That tends to lend you a lot of buffer to deal with the frequent, inevitable, and variably-sized rejections that come as part of the job. However, I’ve also got a few ideas about advice to deal with rejection (some of which I’ve shared previously).

Upon rejection:

  1. Take a deep, full breath: No, it won’t help materially- but it’ll help you feel better about things. Also look at beautiful flowers, treat yourself to a donut, listen to a favorite song, give yourself something positive. Take a break and give yourself a little distance.
  2. Put things in perspective: Run down Maslow’s hierarchy of needs. How you doing on there? I’ll bet you’ve got the bottom layers of the pyramid totally covered. You’re all over that. And it’s unlikely that this one rejection will cause you to slip on this pyramid thing.
  3. Recognize your privilege: In a global (and likely local) perspective you are extremely privileged just to be at this level of the game. You are a researcher/academic/student and get to do interesting, fun, rewarding, and challenging stuff every day. And somebody pays you to do that.
  4. Remember: science is ALL about failure. If you’re not failing, you’re not doing it right. Learn from your failures and rejections. Yes, reviewers didn’t get you. But that means that you need to do a better job of grabbing their attention and convincing them the next time.
  5. Recognize the reality: You are dealing with peer review, which is arbitrary and capricious. Given the abysmal levels of research funding and the numbers of papers being submitted to journals it is the case that many good proposals get rejected. The system works, but only poorly and only sometimes. And when everyone is scraping for money it gets worse.
  6. Evaluate: How do YOU feel about the proposal/submission: forget what the reviewers said, forget the rejection and try to put yourself in the role of reviewer.
    This is YOU on the steps of the NIH in 6 months! Winning!

    This is YOU on the steps of the NIH in 6 months! Winning!

    Would YOU be impressed? Would YOU fund you? If the answer is ‘no’ or ‘maybe’ then you need to reevaluate and figure out how to make it into something that you WOULD or decide if it’s something you should let go.

  7. Make plans: Take what you know and plan the next step. What needs to be done and what’s a reasonable timeline to accomplish this. This step can be really helpful in terms of helping you feel better about the rejection. Instead of wallowing in the rejection you’re taking ACTION. And that can’t be a bad thing. It may be helpful to have a writing/training montage to go along with this since that makes things more fun and go much faster. Let me suggest as the theme to Rocky as a start.

I’m not saying you (or I) can do all of these in a short time. This process can take time- and sometimes distance. And, yes, I do realize that some of this advice is a little in the vein of the famous Stuart Smalley. But, gosh darn it, you ARE smart enough.

stuart_smalley

*For those interested, I submitted an R01 proposal to the NIH last February. It was reviewed at the NIH study section on Monday and Tuesday. The results of this review were updated in the NIH submission/tracking system, eRA commons, just this morning. I won’t know why the proposal was ‘not discussed’ for probably a week or so, when they post the summary of reviewers’ written comments. But for now I know that it was not discussed at the section and thus will not be funded.

At this point I’ve submitted something like 8 R01-level proposals as a PI or co-PI. I’ve been ‘Not Discussed’ on 7 of those. On the eight I got a score, but it was pretty much the lowest score you can get. Given that NIH pay lines are something around 10% I figure that one of the next 2 proposals I submit will be funded. Right? But I’ve been successful with internal funding, collaborations, and working on large center projects that have come to the lab- so I really can’t complain.

New biosketch format for NIH

Just got word of this from the Twitters- the NIH is announcing rollout of a new biosketch format for grant applications. I thought I’d summarize the information about it here to make things easy.

  1. Starting for grants that would be funded in 2016 (so anything you apply for in 2015 will have this, essentially)
  2. Will be a five page limit (as opposed to 2 page for current format)
  3. Will NOT include a ‘bare’ list of 15 publications
  4. Now includes a new list of up to 5 of your “contributions to science”, which can include up to 4 citations each (your own presumably). So that’s a total of 20 citations you can have.
  5. You will be able to include a link to your full citation list in an online database (NIH resources sciENcv or My Bibliography are mentioned but could be others I guess)
  6. Here’s a template that includes an example new biosketch
  7. More information can be found in the announcement here from the NIH on Salley Rockey’s blog
  8. As it’s being rolled out the requirement for the new format will be stated in the RFA – SO LOOK FOR THIS IN ALL RFAs from here out. Two pilots already have it RFA-CA-13-501 and  RFA-CA-13-502

The upsides: In my opinion the addition of the contributions to science will be the biggest one and should allow you to really highlight your publications (or lack thereof) in appropriate context. Plus space for moar buzzwords!

The downsides: It’s gonna be a pain to write (like a mini grant in there) and reviewers won’t read this one either.

Not being part of the rumor mill

I had something happen today that made me stop and think. I repeated a bit of ‘knowledge’ – something science-y that had to do with a celebrity. This was a factoid that I have repeated many other times. Each time I do I state this factoid with a good deal of authority in my voice and with the security that this is “fact”. Someone who was in the room said, “really?” Of course, as a quick Google check to several sites (including snopes.com) showed- this was, at best, an unsubstantiated rumor, and probably just plain untrue. But the memory voice in my head had spoken with such authority! How could it be WRONG? I’m generally pretty good at picking out bits of misinformation that other people present and checking it, but I realized that I’m not always so good about detecting it when I do it myself.

Of course, this is how rumors get spread and disinformation gets disseminated. As scientists we are not immune to it- even if we’d like to think we are. And we actually could be big players in it. You see, people believe us. We speak with the authority of many years of schooling and many big science-y wordings. And the real danger is repeating or producing factoids that fall in “science” but outside what we’re really experts in (where we should know better). Because many non-scientists see us as experts IN SCIENCE. People hear us spout some random science-ish factoid and they LISTEN to us. And then they, in turn, repeat what we’ve said, except that this time they say it with authority because it was stated, with authority, by a reputable source. US. And I realized that this was the exact same reason that it seemed like fact to me. Because it had been presented to me AS FACT by someone who I looked up to and trusted.

So this is just a note of caution about being your own worst critic – even in normal conversation. Especially when it comes to those slightly too plausible factoids. Though it may not seem like it sometimes people do listen to us.

The first day of the rest of your career

I remember well what that day felt like. Actually there were two days. The first, most exhilarating, was when you went into mortal, hand-to-hand combat with a bunch of seasoned veterans (your committee) and emerged victorious! Well, bruised, battered, downtrodden, possibly demoralized, but otherwise victorious! After years of toil, and months of writing, and weeks of preparation, and days of worrying you’d survived, and maybe even said something semi, halfway smartish.

Then there’s the graduation ceremony. The mysterious hooding. It has to be special because it’s a HOOD for god’s sake- and nobody knows WHY! (OK- I’m sure there are lots of people who do know why if I’d bother to Google it, which I won’t. Leave the mystery). Your family and/or loved ones afterward to welcome you and elbow you saying (slyly) “Doctor?” Feels awesome.

When the smoke and mirrors clear and the buzz has died down you might ask yourself: what next? Most will already have a post-doc lined up but not everybody. But even if you do your newly minted PhD isn’t the entire world. Let me tell you what that amazing, splendiferous, wholly unmatched in the history of science-dom PhD looks like to a future employer:

Square One.

This is the first day of the rest of your career. Revel in it. Be proud of it. But know what it means: a foot (barely) in the door. No doubt a very important foot in the door- but it’s just so you can compete in the next round(s).

PhD_comic2