Therapy

I’ve been thinking lately about how events in your academic life can lead to unintended, and often times unrecognized, downstream effects. Recently I realized that I’m having trouble putting together a couple of papers that I’m supposed to be leading. After some reflection I came to the conclusion that at least one reason is I’ve been affected by the long, tortuous, and somewhat degrading process of trying to get a large and rather important paper published. This paper has been in the works, and through multiple submission/revision cycles, for around five years. And it starts to really wear on your academic psyche after that time, though it can be hard to recognize. I think that my failure to get that paper published (so far) is partly holding me back on putting together these other papers. Partly this is about the continuing and varied forms of rejection you experience in this process, but partly it’s about the fact that there’s something sitting there that shouldn’t be sitting there. Even though I don’t currently have any active tasks that I have to complete for that problem paper it still weighs on me.

The silver lining is that once I recognized that this was a factor things started to seem easier with those projects and the story I was trying to tell. Anyway, I think we as academics should have our own therapists that specialize in problems such as this. It would be very helpful.

therapy_comic

Proposal gambit – Update 1

Last week I posted about my strategy for a proposal I’m just submitting. Pretty simple really, just using a publication in a post-publication peer review journal (F1000 Research) as the crucial piece of my preliminary data in my grant. Here’s an update on the process.

So, if you’re going to predicate an R01 submission on having a citation to a paper with a crucial set of preliminary data in it… don’t leave it until the last minute. I submitted my paper to F1000 Research on Thursday (one week prior to the submission date for my grant). They responded very quickly – next day, with requests for some minor changes and to send the figures separately (I had included them in the document). No problems, but then the weekend came up and I ended up getting everything back to them on Sunday evening. Fine. Monday came and went and I didn’t have a link. Also on Monday I was surprised because I was erroneously told that I had to have the absolute final version of my grant to our grants and contracts office that day. With no citation. I scrambled to make myself an arXiv account so that I could get it out that way (a good thing in any case). But turns out it was incorrect and I could still make minor modifications after that.

So yesterday (Tuesday) I pinged F1000 Research, politely and with acknowledgment that this was a short turnaround time, and mentioned that I wanted to put the citation in the grant. They replied on Wednesday morning apologizing for the delay (nice, but there was no delay- I was really trying to push things fast) and saying that the formatted version should be ready in a couple of days and GIVING ME A DOI for the paper! Perfect. That’s what I really needed to include in the grant.

So today the updated grant was actually submitted- a whole day early, probably a first. Now it’s just a matter of settling in until June when it will be reviewed. Of course, I still need to get my paper reviewed, but I think that won’t be a huge problem.

Overall this process is going swimmingly. And I’ve been really pleased with my interactions with F1000 Research so far.

Regret

Well, there probably ARE some exceptions here.

Well, there probably ARE some exceptions here.

So I first thought of this as a funny way of expressing relief over a paper being accepted that was a real pain to get finished. But after I thought about the general idea awhile I actually think it’s got some merit in science. Academic publication is not about publishing airtight studies with every possibility examined and every loose end or unconstrained variable nailed down. It can’t be. That would limit scientific productivity to zero because it’s not possible. Science is an evolving dialogue, some of it involving elements of the truth.

The dirty little secret (or elegant grand framework, depending on your perspective) of research is that science is not about finding the truth. It’s about moving our understanding closer to the truth. Often times that involves false positive observations- not because of the misconduct of science but because of it’s proper conduct. You should never publish junk or anything that’s deliberately misleading. But you can’t help publishing things that sometimes move us further away from the truth. The idea in science is that these erroneous findings will be corrected by further iterations and may even provide an impetus for driving studies that advance science. So publish away!

Literature Search Party

Continuing on my adventure metaphor theme: has this ever happened to you? You have a great idea, it’s brilliant, it’s revolutionary, it’s a thing that will change the way that people think about other things. You work on it, sometimes feverishly. And get… great results! Then you think, “hey – wait a minute. If this is such a great idea and so simple, why hasn’t anyone ever thought of it before?” Pause about 10 minutes. “Ohhhhhh… no. They probably have.” A quick PubMed search turns up that seminal paper from 1995 demonstrating what you’ve just ‘discovered’. My diagram on how to do science highlights this point.

Anyway, why does this problem happen and how can you avoid it. I don’t have the answers but here are some general ideas.

For me this often happens because, in coming up with a brilliant new idea you’re pushing your knowledge and experience past it’s limits. In the early stages this means that your ideas are not very well formed; you don’t have a clear idea of what you’re thinking about and how it might relate to other things. And you don’t know the area you’re moving in to. So even doing a literature search at this point can be useless. I’ve had the situation where what I was searching for actually had been done before, but I didn’t know what to call it- so PubMed was useless.

After you’ve started to get some legs to the project, maybe doing a few tests to see if it would even work and getting positive results, excitement can take over. Then you just want to get through it and get the good results. Even then you may not be able to see your idea in a greater context to be able to know what to look for.

Finally, in the later stages of the project you can suffer from “investment blindness”. You may ignore the issue of searching the literature because what if you found that you weren’t doing something new? You’d put SO much work in to it, it would be unthinkable to have to abandon it all! And you’re on a roll- the good results are coming in, the implications are starting to fall into place, and the shape of the thing, the idea you’ve had, is starting to make itself clear. It’s generally at this point that that creeping, nagging, suspicious feeling comes up. Yep, you’re pretty sure somebody MUST have done this before.

Sometimes you’re wrong. Other times you’re right, but the spin you’ve put on things and the results you’ve gotten are actually novel and you can still get a story out (this is the most common actually). Then there are the times when there’s just nothing you can do. Your exact idea has been done somewhere else and published in Nature or Science or Cell, Nature, and Science.

I guess the idea is that you know your field so well that you can see the gaps and know when you are trying something new. That’s true of a number of different projects I’ve initiated. Generally, these are not the most interesting or groundbreaking. Sometimes they’re downright boring, small steps forward.

How can you prevent this? I guess by being aware of that three-step progression I outlined above, and trying at each step to do your literature searches with that in mind. Also, be pessimistic: always start from the point of view that someone has done it before. You’re then not surprised if they have done it, and you can start to evaluate how different and novel your approach is from theirs. Approaching your literature search from the point of view that you’re looking for something will make it more likely that you will find something.

Also, consult friends and colleagues who are working in similar areas. Sometimes they may know what you’re talking about – that is, that someone has already done and they know the name of what it is you’re doing. Sometimes they might just be able to provide you with a sounding board for your idea that will allow you to clarify your thoughts.

Above all, be flexible. If it turns out that someone has done it before read their paper carefully and any follow-on papers you can find. Look for the gaps and ask how what you’ve done can answer a critical question they’ve left open.

Dude. You want a beer or something? It's hot work making it all the way up here.

Dude. You want a beer or something? It’s hot work making it all the way up here.

Excavation

I finished up revising a figure that I’d put together for a paper the other day with such a feeling of finality and satisfaction. Then I realized that I had to do the same thing to two other figures in the same manuscript- and that each one involved a different set of analyses. Each analysis requires me to dig around and figure out where I was and what files I needed to be looking at and even how I’d done the analysis in the first place. It made me feel like I was unearthing a huge pyramid or something- or in a competitive eating contest (as I’ve written about before). So that was the inspiration for today’s comic.

I was giving serious consideration to having the guy have to fight off Nazis, outrun a boulder, and duke it out with a big dude under an airplane after he got the scroll. But I couldn't figure out how to draw faces melting off of the stick figures...

I was giving serious consideration to having the guy have to fight off Nazis, outrun a boulder, and duke it out with a big dude under an airplane after he got the scroll. But I couldn’t figure out how to draw faces melting off of the stick figures…

Career Strategy

Including this on my actual CV could be a problem though...

Including this on my actual CV could be a problem though…

I Tweeted this last week as a brilliant idea for a career move.

The largest barrier I see to this actually working is that it will be difficult to include it on my  printed CV. And have the same effect at least. I’m working on it. Also it has a similar bad taste as the “research” group that bought their own journal to publish their ridiculous paper on Sasquatch DNA.

Turns out, of course, that I’d been beaten to the idea by actual publishers:  

Screen Shot 2014-09-02 at 9.30.18 AM home_cover Screen Shot 2014-09-02 at 9.32.46 AM

As pointed out:

I think I’ve still got something novel with the whole “Other High Impact” journal idea. *BRILLIANT!!!*

Dealing with Academic Rejection

Funny, it feels like I’ve written about exactly this topic before…

I got rejected today, academically speaking*. Again. I was actually pretty surprised at how

"Not Discussed", again

“Not Discussed”, again

nonplussed I was about the whole thing. I’ve gotten mostly immune to the being rejected- at least for grant proposals and paper submissions. It certainly could be a function of my current mid-career, fairly stable status as a scientist. That tends to lend you a lot of buffer to deal with the frequent, inevitable, and variably-sized rejections that come as part of the job. However, I’ve also got a few ideas about advice to deal with rejection (some of which I’ve shared previously).

Upon rejection:

  1. Take a deep, full breath: No, it won’t help materially- but it’ll help you feel better about things. Also look at beautiful flowers, treat yourself to a donut, listen to a favorite song, give yourself something positive. Take a break and give yourself a little distance.
  2. Put things in perspective: Run down Maslow’s hierarchy of needs. How you doing on there? I’ll bet you’ve got the bottom layers of the pyramid totally covered. You’re all over that. And it’s unlikely that this one rejection will cause you to slip on this pyramid thing.
  3. Recognize your privilege: In a global (and likely local) perspective you are extremely privileged just to be at this level of the game. You are a researcher/academic/student and get to do interesting, fun, rewarding, and challenging stuff every day. And somebody pays you to do that.
  4. Remember: science is ALL about failure. If you’re not failing, you’re not doing it right. Learn from your failures and rejections. Yes, reviewers didn’t get you. But that means that you need to do a better job of grabbing their attention and convincing them the next time.
  5. Recognize the reality: You are dealing with peer review, which is arbitrary and capricious. Given the abysmal levels of research funding and the numbers of papers being submitted to journals it is the case that many good proposals get rejected. The system works, but only poorly and only sometimes. And when everyone is scraping for money it gets worse.
  6. Evaluate: How do YOU feel about the proposal/submission: forget what the reviewers said, forget the rejection and try to put yourself in the role of reviewer.
    This is YOU on the steps of the NIH in 6 months! Winning!

    This is YOU on the steps of the NIH in 6 months! Winning!

    Would YOU be impressed? Would YOU fund you? If the answer is ‘no’ or ‘maybe’ then you need to reevaluate and figure out how to make it into something that you WOULD or decide if it’s something you should let go.

  7. Make plans: Take what you know and plan the next step. What needs to be done and what’s a reasonable timeline to accomplish this. This step can be really helpful in terms of helping you feel better about the rejection. Instead of wallowing in the rejection you’re taking ACTION. And that can’t be a bad thing. It may be helpful to have a writing/training montage to go along with this since that makes things more fun and go much faster. Let me suggest as the theme to Rocky as a start.

I’m not saying you (or I) can do all of these in a short time. This process can take time- and sometimes distance. And, yes, I do realize that some of this advice is a little in the vein of the famous Stuart Smalley. But, gosh darn it, you ARE smart enough.

stuart_smalley

*For those interested, I submitted an R01 proposal to the NIH last February. It was reviewed at the NIH study section on Monday and Tuesday. The results of this review were updated in the NIH submission/tracking system, eRA commons, just this morning. I won’t know why the proposal was ‘not discussed’ for probably a week or so, when they post the summary of reviewers’ written comments. But for now I know that it was not discussed at the section and thus will not be funded.

At this point I’ve submitted something like 8 R01-level proposals as a PI or co-PI. I’ve been ‘Not Discussed’ on 7 of those. On the eight I got a score, but it was pretty much the lowest score you can get. Given that NIH pay lines are something around 10% I figure that one of the next 2 proposals I submit will be funded. Right? But I’ve been successful with internal funding, collaborations, and working on large center projects that have come to the lab- so I really can’t complain.

How to know when to let go

This post is a story in two parts. The first part is about the most in-depth peer review I think I’ve ever gotten. The second deals with making the decision to pull the plug on a project.

Part 1: In which Reviewer 3 is very thorough, and right.

Sometimes Reviewer 3 (that anonymous peer reviewer who consistently causes problems) is right on the money. To extend some of the work I’ve done I’ve done to predict problematic functions of proteins I started a new effort about 2 years ago now. It went really slowly at first and I’ve never put a huge amount of effort in to it, but I thought it had real promise. Essentially it was based on gathering up examples of a functional family that

"At last we meet, reviewer 3, if that is indeed your real name"

“At last we meet, reviewer 3, if that is indeed your real name”

could be used in a machine learning-type approach. The functional family (in this case I chose E3 ubiquitin ligases) is problematic in that there are functionally similar proteins that show little or no sequence similarity by traditional BLAST search. Anyway, using a somewhat innovative approach we developed a model that could predict these kinds of proteins (which are important bacterial virulence effectors) pretty well (much better than BLAST). We wrote up the results and sent it off for an easy publication.

Of course, that’s not the end of the story. The paper was rejected from PLoS One, based on in-depth comments from reviewer 1 (hereafter referred to as Reviewer 3, because). As part of the paper submission we had included supplemental data, enough to replicate our findings, as should be the case*. Generally this kind of data isn’t scrutinized very closely (if at all) by reviewers. This case was different. Reviewer 3 is a functional prediction researcher of some kind (anonymous, so I don’t really know) and their lab is set up to look at these kinds of problems- though probably not from the bacterial pathogenesis angle judging from a few of the comments. So Reviewer 3’s critique can be summed up in their own words:

I see the presented paper as a typical example of computational “solutions” (often based on machine-learning) that produce reasonable numbers on artificial test data, but completely fail in solving the underlying biologic problem in real science.

Ouch. Harsh. And partly true. They are actually wrong about that point from one angle (the work solves a real problem- see Part 2, below) but right from another angle (that problem had apparently already been solved, at least practically speaking). They went on, “my workgroup performed a small experiment to show that a simple classifier based on sequence similarity and protein domains can perform at least as well as <my method> for the envisioned task.” In the review they then present an analysis they did on my supplemental data in which they simply searched for existing Pfam domains that were associated with ubiquitin ligase function. Their analysis, indeed, shows that just searching Reviewer3Attackfor these four known domains could predict this function as well or better than my method. This is interesting because it’s the first time that I can remember where a reviewer has gone in to the supplemental data to do an analysis for the review. This is not a problem at all- in fact, it’s a very good thing. Although I’m disappointed to have my paper rejected I was happy that a knowledgeable and thorough peer reviewer had done due diligence and exposed this, somewhat gaping, hole in my approach/results. It’s worth noting that the other reviewer identified himself, was very knowledgeable and favorable to the paper- just missing this point because it’s fairly specific and wrong, at least in a particular kind of way that I detail below.

So, that’s it right? Game over. Take my toys and go home (or to another more pressing project). Well, maybe or maybe not.

Part 2: In which I take a close look at Reviewer 3’s points and try to rescue my paper

One of the hardest things to learn is how to leave something that you’ve put considerable investment into and move on to more productive pastures. This is true in relationships, investments, and, certainly, academia. I don’t want to just drop this two year project (albeit, not two solid years) without taking a close look to see if there’s something I can do to rescue it. Without going into the details of specific points Reviewer 3 made I’ll tell you about my thought process on this topic.

So, first. One real problem here is that the Pfam models Reviewer 3 used were constructed from the examples I was using. That means that their approach is circular: the Pfam model can identify the examples of E3 ubiquitin ligases because it was built from those same examples. They note that four different Pfam models can describe most of the examples I used. From the analysis that I did in the paper and then again following Reviewer 3’s comments, I found that these models do not cross-predict, whereas my model does. That is, my single model can predict the same as these four different individual models. These facts both mean that Reviewer 3’s critique is not exactly on the mark- my method does some good stuff that Pfam/BLAST can’t do. Unfortunately, neither of these facts makes my method any more practically useful. That is, if you want to predict E3 ubiquitin ligase function you can use Pfam domains to do so.

Which leads me to the second point of possible rescue. Reviewer 3’s analysis, and my subsequent re-analysis to check to make sure they were correct, identified around 30 proteins that are known ubiquitin ligases but which do not have one of these four Pfam domains. These are false negative predictions, by the Pfam method. Using my method these are all predicted to be ubiquitin ligases with pretty good accuracy. This is a definite good point then to my method, meaning that my method can correctly identify those known ligases that don’t have known domains. There! I have something useful that I can publish, right? Well, not so fast. I was interested in seeing what Pfam domains might be in those proteins other than the four ligase domains so I looked more closely. Unfortunately what I found was that these proteins all had a couple of other domains that were specific to E3 ubiquitin ligases but that Reviewer 3 didn’t notice. Sigh. So that means that all the examples in my E3 ubiquitin ligase dataset can be correctly identified by around 6 Pfam domains, again rendering my method essentially useless, though not incorrect. It is worth noting that it is certainly possible that my method would be much better at identification of new E3 ligases that don’t fall into these 6 ‘families’ – but I don’t have any such examples, so I don’t really know and can’t demonstrate this in the paper.

So where does this leave me? I have a method that is sound, but solves a problem that may not have been needed to be solved (as Reviewer 3 pointed out, sort of). I would very much like to publish this paper since I, and several other people, have spent a fair amount of time on it. But I’m left a bit empty-handed. Here are the three paths I can see to publication:

  1. Experimental validation. I make some novel predictions with my method and then have a collaborator validate them. Great idea but this would take a lot of time and effort and luck to pull off. Of course, if it worked it would demonstrate the method’s utility very solidly. Not going to happen right now I think.
  2. Biological insight. I make some novel observations given my model that point out interesting biology underpinning bacterial/viral E3 ubiquitin ligases. This might be possible, and I have a little bit of it in the paper already. However, I think I’d need something solid and maybe experimentally validated to really push this forward.
  3. Another function. Demonstrate that the general approach works on another functional group- one that actually is a good target for this kind of thing. This is something I think I have (another functional group) and I just need to do some checking to really make sure first (like running Pfam on it, duh.) I can then leave the ubiquitin ligase stuff in there as my development example and then apply the method to this ‘real’ problem. This is most likely what I’ll do here (assuming that the new example function I have actually is a good one) since it requires the least amount of work.

So, full disclosure: I didn’t know when I started writing this post this morning what I was going to do with this paper and had pretty much written it off. But now I’m thinking that there may be a relatively easy path to publication with option 3 above. If my new example doesn’t pan out I may very well have to completely abandon this project and move on. But if it does work then I’ll have a nice little story requiring a minimum of (extra) effort.

As a punchline to this story- I’ve written a grant using this project as a fairly key piece of preliminary data. That grant is being reviewed today- as I write. As I described above, there’s nothing wrong with the method- and it actually fits nicely (still) to demonstrate what I needed it to for the grant. However, if the grant is funded then I’ll have actual real money to work on this and that will open other options up for this project. Here’s hoping. If the grant is funded I’ve decided I’ll establish a regular blog post to cover it, hopefully going from start to (successfully renewed) finish on my first R01. So, again, here’s hoping.

*Supplemental data in a scientific manuscript is the figures, tables, and other kinds of data files that either can’t be included in the main text because of size (no one wants to read 20 pages of gene listings in a paper- though I have seen stuff like this) or because the information is felt to be non-central to the main story and better left for more interested readers.

The Map of Manuscript-earth

Adventures in manuscript-land. For those who don’t get it this is imagining the path of writing an academic manuscript from the point of view of a map of a fantasy realm. There’s just so much material here- I may have to do another like this. What I didn’t really capture here is the very difficult aspect of getting rejecting, revising, resubmitting, and getting rejected again (damn you Reviewer 3!). It’s like going over those distant mountains then finding yourself back on the road somewhere still trying to get there. For a case in point of something that is still on this road see my stalled series on a computational biology project.

This comic is just crying out for a board game I think. Not sure anyone would play it though. Too painful.

The Map of Manuscript-earth

The Map of Manuscript-earth

Nature Transient Findings

You know, I think the glamour publishers could really benefit from a journal to publish these kinds of results. Far less messy and then they won’t get confused with real science. Also, a bonus is that the title of the papers could be already buzzfeed-ready, no editing involved.

Also I’ve officially titled my series of academia-themed comics Red Pen/Black Pen (see previous post for something of an explanation)

Too good to be true or too good to pass up?

Too good to be true or too good to pass up?

This comic was inspired by this wonderful parody, which was circulating awhile back but unfortunately I don’t know proper attribution.

Because you tried really hard for a really long time.

Because you tried really hard for a really long time.