Busy-o-Meter

I’ve posted before about some of my organizational approaches (and attempts at it) but it can sometimes be impossible not to get overwhelmed and busy. Being busy on multiple tasks, with multiple deadlines can be a killer, but sometimes it crystallizes a resolve to move some of those items off your todo list and you increase your overall effectiveness (you know, less Twitter and blogging and comic-making and stuff). I’ve also seen people who seem to make it a part of their academic persona to be perpetually too busy. This seems to be considered a status symbol (often times mostly by the person being so busy). The key to busy-ness and keeping your head above water (and the seals at bay) is balance. Make sure to keep perspective about what you’re doing and know that often (maybe always) banging your head against the same task for hours on end is counterproductive.

Anyway, here’s a handy tool to help you assess your level of busy-ness, fresh from the RedPen/BlackPen labs.

comic_scan

Phenomenal science powers!

So you toil for 4+ years in graduate school, 4+ years as a post-doc, land your first academic gig. Now you get to do all this awesome science, right? Well, sorta…

Oy! 10,000 years in the cave of graduate school will give you SUCH a crook in the neck!

Oy! 10,000 years in the cave of graduate school will give you SUCH a crook in the neck!

Proposal gambit – Betting the ranch

Last spring I posted about a proposal I’d put in where I’d published the key piece of preliminary data in F1000 Research, a journal that offers post-publication peer review.

The idea was that I could get my paper published (it’s available here) and accessible to reviewers prior to submission of my grant. It could then be peer-reviewed and I could address the revisions after that. This strategy was driven by the lag time between proposal submission and review for NIH, which is about 4 months. Also, it used to be possible to include papers that hadn’t been formally accepted by a journal as an appendix to NIH grants. This hasn’t been possible for some time now. But I figured this might be a pretty good way to get preliminary data out to the grant reviewers in a published form with quick turnaround. Or at least that you could utilize that lag time to also function as review time for your paper.

I was able to get my paper submitted to F100 Research and obtained a DOI and URL that I could include as a citation in my grant. Details here.

The review for the grant was completed in early June of this year and the results were not what I had hoped- the grant wasn’t even scored, despite being totally awesome (of course, right?). But for this post I’ll focus on the parts that are pertinent to the “gambit”- the use of post-publication peer review as preliminary data.

The results here were mostly unencouraging RE post-publication peer review being used this way, which was disappointing. But let me briefly describe the timeline, which is important to understand a large caveat about the results.

I received first-round reviews from two reviewers in a blindingly fast 10 and 16 days after initial submission. Both were encouraging, but had some substantial (and substantially helpful) requests. You can read them here and here. It took me longer than it could have to address these completely – though I did some new analysis and added additional explanation to several important points. I then resubmitted on around May 12th or so. However, due to some kind of issue the revised version wasn’t made available by F1000 Research until May 29th. Given that the NIH review panel met in the first week of June it is likely that the reviewers didn’t see the revised (and much improved version). The reviewers then got back final comments in early June (again- blindingly fast). You can read those here and here. The paper was accepted/approved/indexed in mid-June.

The grant had comments from three reviewers and each had something to say about the paper as preliminary data.

The first reviewer had the most negative comments.

It is not appropriate to point reviewers to a paper in order to save space in the proposal.

Alone this comment is pretty odd and makes me think that the reviewer was annoyed by the approach. So I can’t refer to a paper as preliminary data? On the face of it this is absolutely ridiculous. Science, and the accumulation of scientific knowledge just doesn’t work in a way that allows you to include all your preliminary data completely (as well as your research approach and everything else) in the space of 12 page grant. However, their further comments (which directly follow this one) shed some light on their thinking.

The PILGram approach should have been described in sufficient detail in the proposal to allow us to adequately assess it. The space currently used to lecture us on generative models could have been better used to actually provide details about the methods being developed.

So reading between the (somewhat grumpy) lines I think they mean to say that I should have done a better job of presenting some important details in the text itself. But my guess is that the first reviewer was not thrilled by the prospect of using a post-publication peer reviewed paper as preliminary data for the grant. Not thrilled.

  • Reviewer 1: Thumbs down.

Second reviewer’s comment.

The investigators revised the proposal according to prior reviews and included further details about the method in the form of a recently ‘published’ paper (the quotes are due to the fact that the paper was submitted to a journal that accepts and posts submissions even after peer review – F1000 Research). The public reviewers’ comments on the paper itself raise several concerns with the method proposed and whether it actually works sufficiently well.

This comment, unfortunately, is likely due to the timeline I presented above. I think they saw the first version of the paper, read the paper comments, and figured that there were holes in the whole approach. If my revisions had been available it seems like there still would have been issues, unless I had already gotten the final approval for the paper.

  • Reviewer 2: Thumbs down- although maybe not with the annoyed thrusting motions that the first reviewer was presumably making.

Finally, the third reviewer (contrary to scientific lore) was the most gentle.

A recent publication is suggested by the PI as a source of details, but there aren‟t many in that manuscript either.

I’m a little puzzled about this since the paper is pretty comprehensive. But maybe this is an effect of reading the first version, not the final version. So I would call this neutral on the approach.

  • Reviewer 3: No decision.

Summary

The takeaway from this gambit is mixed.

I think if it had been executed better (by me) I could have gotten the final approval through by the time the grant reviewers were looking at it and then a lot of the hesitation and negative feelings would have gone away. Of course, this would be dependent on having paper reviewers that were as quick as those that I got- which certainly isn’t a sure thing.

I think that the views of biologists on preprints, post-publication review, and other ‘alternative’ publishing options are changing. Hopefully more biologist will start using these methods- because, frankly, in a lot of cases they make a lot more sense than the traditional closed-access, non-transparent peer review processes.

However, the field can be slow to change. I will probably try this, or something like this, again. Honestly, what do I have to lose exactly? Overall, this was a positive experience and one where I believe I was able to make a contribution to science. I just hope my next grant is a better substrate for this kind of experiment.

Other posts on this process:

 

 

Therapy

I’ve been thinking lately about how events in your academic life can lead to unintended, and often times unrecognized, downstream effects. Recently I realized that I’m having trouble putting together a couple of papers that I’m supposed to be leading. After some reflection I came to the conclusion that at least one reason is I’ve been affected by the long, tortuous, and somewhat degrading process of trying to get a large and rather important paper published. This paper has been in the works, and through multiple submission/revision cycles, for around five years. And it starts to really wear on your academic psyche after that time, though it can be hard to recognize. I think that my failure to get that paper published (so far) is partly holding me back on putting together these other papers. Partly this is about the continuing and varied forms of rejection you experience in this process, but partly it’s about the fact that there’s something sitting there that shouldn’t be sitting there. Even though I don’t currently have any active tasks that I have to complete for that problem paper it still weighs on me.

The silver lining is that once I recognized that this was a factor things started to seem easier with those projects and the story I was trying to tell. Anyway, I think we as academics should have our own therapists that specialize in problems such as this. It would be very helpful.

therapy_comic

Writing Yourself Into A Corner

I’ve been fascinated with the idea of investment, and how it can color your thoughts, feelings, and opinions about something. Not the monetary sense of the word (though probably that too) but the emotional and intellectual sense of the word. If you’ve ever been in a bad relationship you might have fallen prey to this reasoning- “I’m in this relationship and I’m not getting out because reasons so admitting that’s it’s absolutely terrible for me is unthinkable so I’m going to pretend like it’s not and I’m going to believe that it’s not and I’m going to tell everyone that I’m doing great”. I really believe this can be a motivating factor for a big chunk of human behavior.

And it’s certainly a problem in science. When you become too invested in an idea or an approach or a tool- that is, you’ve spent a considerable amount of time researching or promoting it- it can be very difficult to distance yourself from that thing and admit that you might have it wrong. That would be unthinkable.

Sometimes this investment pitfall is contagious. If you’re on a project working together with others for common goals the problem of investment can become more complicated. That is, if I’ve said something, and some amount of group effort has been put into this idea, but it turns out I was wrong about it, it can be difficult to raise that to the rest of the group. Though, I note, that it is really imperative that it is raised. This can become more difficult if the ideas or preliminary results you’ve put forward become part of the project- through presentations made by others or through further investment of project resources to follow up on these leads.

I think this sometimes happens when you’re writing an early draft of a document- though the effect can be more subtle here. If you write words down and put out ideas that are generally sound and on-point it can be hard for you, or others who may edit the paper after you, to erase these. More importantly a first draft, no matter how preliminary or draft-y, can establish an organization that can be hard to break. Clearly if there are parts that really don’t work, or don’t fit, or aren’t true, they can be removed fairly easily. The bigger problems lie in those parts that are *pretty good*. I’ve looked back at my own preliminary drafts and realized (after a whole lot of work trying to get things to fit) that the initial overall organization was somehow wrong- and that I really need to rip it all apart and start over, at least in terms of the organization. I’ve also seen this in other people’s work, where something just doesn’t seem right about a paper, but I really can’t place my finger on what- at least not without a bunch of effort.

Does this mean that you should very carefully plan out your preliminary drafts? Not at all. That’s essentially the route to complete gridlock and non-productivity. Rather, you should be aware of this problem and be willing to be flexible. Realize that what you put down on the paper for the first draft (or early versions of analysis) is subject to change- and make others you are working with aware of this explicitly (simply labeling something as “preliminary analysis” or “rough draft” isn’t explicit enough). And don’t be afraid to back away from it if it’s not working out. It’s much better if that happens earlier in the process than later- that is, it’s better to completely tear down a final draft of a paper than to have reviewers completely miss the point of what you’re trying to say after you’ve submitted it.

writing_corner_v1

Proposal gambit

I am currently (this minute… well, not THIS minute, but just a minute ago, and in a minute) in the throes of revising a resubmission of a previously submitted R01 proposal to NIH. This proposal generally covers novel methods to build protein-sequence-based classifiers for problematic functional classes- that is, groups of proteins that have a shared function but either are very divergent in their sequence (meaning that they can’t be associated by traditional sequence similarity approaches) or have a lot of similar sequences with divergent functions (and the function that’s interesting can’t be easily disambiguated).

I got good feedback from reviewers on the previous version (though I did not get discussed- for those who aren’t familiar with the process, to get a score- and thus a chance at funding- your grant has to be in the top 50% of the grants that the review panel reads, then it moves on to actual discussion in the panel and scoring). Their main complaint was that I had not described the novel method I was proposing in sufficient detail, and so they were intrigued but couldn’t assess if this would really work or not. The format of NIH R01-level grants (12 pages for the research part) means that to provide details of methods you really need to have published your preliminary results. Also- if it’s published it really lends weight to the fact that you can do it and get it through peer review (or pay your way into a publication in an fly-by-night journal).

So anyway. I’ve put this resubmission off since last year and I’m not getting any younger and I don’t have a publication to reference on the method in the proposal yet. So here’s my gambit. I’ve been working on the paper that will provide preliminary data and it was really nearly finished it just needed a good push to get it finalized, which came in the form of this grant. My plan is to finish up the last couple of details on the paper and submit it to F1000 Research because it offers online publication immediately with subsequent peer review. I’ve been intrigued by this emerging model recently and wanted to try it anyway. But this allows me to reference the online version very soon after I upload it (maybe tomorrow) and include it as a bona fide citation for my grant. The idea is that by the time it’s reviewed (3 months hence) it will have passed peer review and will be an actual citation.

But it’s a gambit. It’s possible that the paper will still be under review or will have received harsh reviews by the time the reviewers look at it. It’s also possible that since I won’t have a traditional journal citation in text for the proposal- I’ll need to supply a URL to my online version- that the reviewers will just frown on this whole idea and it might even piss them off making them think I’m trying to get away with something (which I totally am, though it’s not unethical or against the rules in any way that I can see). However, I’m pretty sure that this is a lot more common on the CS side (preprint servers, and the like) so I’m betting on that flying.

Anyway, I’ll have an update in 3+ months on how this worked out for me. I actually have high hopes for this proposal- which does scare me a little. But I’m totally used to dealing with rejection, as I’ve mentioned before on numerous occasions. Wish me luck!

Big Data Showdown

One of the toughest parts of collaborative science is communication across disciplines. I’ve had many (generally initial) conversations with bench biologists, clinicians, and sometimes others that go approximately like:

“So, tell me what you can do with my data.”

“OK- tell me what questions you’re asking.”

“Um,.. that kinda depends on what you can do with it.”

“Well, that kinda depends on what you’re interested in…”

And this continues.

But the great part- the part about it that I really love- is that given two interested parties you’ll sometimes work to a point of mutual understanding, figuring out the borders and potential of each other’s skills and knowledge. And you generally work out a way of communicating that suits both sides and (mostly) works to get the job done. This is really when you start to hit the point of synergistic collaboration- and also, sadly, usually about the time you run out of funding to do the research.
BigDataShowdown_v1

Regret

Well, there probably ARE some exceptions here.

Well, there probably ARE some exceptions here.

So I first thought of this as a funny way of expressing relief over a paper being accepted that was a real pain to get finished. But after I thought about the general idea awhile I actually think it’s got some merit in science. Academic publication is not about publishing airtight studies with every possibility examined and every loose end or unconstrained variable nailed down. It can’t be. That would limit scientific productivity to zero because it’s not possible. Science is an evolving dialogue, some of it involving elements of the truth.

The dirty little secret (or elegant grand framework, depending on your perspective) of research is that science is not about finding the truth. It’s about moving our understanding closer to the truth. Often times that involves false positive observations- not because of the misconduct of science but because of it’s proper conduct. You should never publish junk or anything that’s deliberately misleading. But you can’t help publishing things that sometimes move us further away from the truth. The idea in science is that these erroneous findings will be corrected by further iterations and may even provide an impetus for driving studies that advance science. So publish away!

A Fine Trip Spoiled

I had a dream the other night that inspired this comic. My dream was about waiting for a connecting flight. I decided to take it easy and do something fun, then realized that my flight was leaving soon and I was nowhere near the gate. Then I got on a train and realized I was going the wrong direction. Anyway, I woke up to the realization that I’d relaxed and done fun stuff most of the weekend (I did work some in the evenings) and that I had an unfinished grant that was still due this week. As it turned out I finished up my grant quite nicely despite the slacking off- or maybe even because of the slacking off. But it gave me the inspiration for this comic.

You see, writing and submitting a grant proposal is a lot like planning for a vacation that you’ll probably never get to take. The work you’re proposing should be fun and interesting (otherwise, why are you trying to get money to do it, right?) but your chances are pretty slim that you’ll ever get to do it- at least in the form that you propose it. I’ve started to think of the grant process as a long game (see this post from one DrugMonkey)- one where the act of writing a single grant is mainly just positioning for the next grant you’ll write down the line. Writing grants give you opportunity to come up with ideas, to consolidate your thoughts, and think through the science that you want to do and how you want to do it. The process can push you to publish your work so that you can cite it as preliminary data. And it can forge long-lasting collaborations that go beyond failed proposals (though funded proposals certainly help to cement these relationships in a much more sure way).

I think “A Fine Trip Spoiled” may be the title of my autobiography when I get rich and famous.

GrantWritingTravel_2

Literature Search Party

Continuing on my adventure metaphor theme: has this ever happened to you? You have a great idea, it’s brilliant, it’s revolutionary, it’s a thing that will change the way that people think about other things. You work on it, sometimes feverishly. And get… great results! Then you think, “hey – wait a minute. If this is such a great idea and so simple, why hasn’t anyone ever thought of it before?” Pause about 10 minutes. “Ohhhhhh… no. They probably have.” A quick PubMed search turns up that seminal paper from 1995 demonstrating what you’ve just ‘discovered’. My diagram on how to do science highlights this point.

Anyway, why does this problem happen and how can you avoid it. I don’t have the answers but here are some general ideas.

For me this often happens because, in coming up with a brilliant new idea you’re pushing your knowledge and experience past it’s limits. In the early stages this means that your ideas are not very well formed; you don’t have a clear idea of what you’re thinking about and how it might relate to other things. And you don’t know the area you’re moving in to. So even doing a literature search at this point can be useless. I’ve had the situation where what I was searching for actually had been done before, but I didn’t know what to call it- so PubMed was useless.

After you’ve started to get some legs to the project, maybe doing a few tests to see if it would even work and getting positive results, excitement can take over. Then you just want to get through it and get the good results. Even then you may not be able to see your idea in a greater context to be able to know what to look for.

Finally, in the later stages of the project you can suffer from “investment blindness”. You may ignore the issue of searching the literature because what if you found that you weren’t doing something new? You’d put SO much work in to it, it would be unthinkable to have to abandon it all! And you’re on a roll- the good results are coming in, the implications are starting to fall into place, and the shape of the thing, the idea you’ve had, is starting to make itself clear. It’s generally at this point that that creeping, nagging, suspicious feeling comes up. Yep, you’re pretty sure somebody MUST have done this before.

Sometimes you’re wrong. Other times you’re right, but the spin you’ve put on things and the results you’ve gotten are actually novel and you can still get a story out (this is the most common actually). Then there are the times when there’s just nothing you can do. Your exact idea has been done somewhere else and published in Nature or Science or Cell, Nature, and Science.

I guess the idea is that you know your field so well that you can see the gaps and know when you are trying something new. That’s true of a number of different projects I’ve initiated. Generally, these are not the most interesting or groundbreaking. Sometimes they’re downright boring, small steps forward.

How can you prevent this? I guess by being aware of that three-step progression I outlined above, and trying at each step to do your literature searches with that in mind. Also, be pessimistic: always start from the point of view that someone has done it before. You’re then not surprised if they have done it, and you can start to evaluate how different and novel your approach is from theirs. Approaching your literature search from the point of view that you’re looking for something will make it more likely that you will find something.

Also, consult friends and colleagues who are working in similar areas. Sometimes they may know what you’re talking about – that is, that someone has already done and they know the name of what it is you’re doing. Sometimes they might just be able to provide you with a sounding board for your idea that will allow you to clarify your thoughts.

Above all, be flexible. If it turns out that someone has done it before read their paper carefully and any follow-on papers you can find. Look for the gaps and ask how what you’ve done can answer a critical question they’ve left open.

Dude. You want a beer or something? It's hot work making it all the way up here.

Dude. You want a beer or something? It’s hot work making it all the way up here.