A case for failure in science

If you’re a scientist and not failing most of the time you’re doing it wrong. The scientific method in a nutshell is to take a best guess based on existing knowledge (the hypothesis) then collect evidence to test that guess, then evaluate what the evidence says about the guess. Is it right or is it wrong? Most of the time this should fail. The helpful and highly accurate plot below illustrates why.

Science is about separating the truth of the universe from the false possibilities about what might be true. There are vastly fewer true things than false possibilities in the universe. Therefore if we’re not failing by disproving our hypotheses then we really are failing at being scientists. In fact, as scientists all we really HAVE is failure. That is, we can never prove something is true, only eliminate incorrect possibilities. Therefore, 100% of our job is failure. Or rather success at elimination of incorrect possibilities.

So if you’re not failing on a regular repeated basis, you’re doing something wrong. Either you’re not being skeptical and critical enough of your own work or you’re not posing interesting hypotheses for testing. So stretch a little bit. Take chances. Push the boundaries (within what is testable using the scientific method and available methods/data, of course). Don’t be afraid of failure. Embrace it!

How much failure, exactly, is there to be had out there? This plot should be totally unhelpful in answering that question

How much failure, exactly, is there to be had out there? This plot should be totally unhelpful in answering that question

 

 

Us versus them in science communication

This Tweet got me thinking about my grandfather. Gideon Kramer was a great thinker who read widely and was very spiritual and philosophical. He also placed a great emphasis on science, but did not consider himself to be a scientist. When he was alive he would continually challenge me to make my science more approachable by a broader audience. He still does. He once suggested that all scientists should publish a lay version of every technical paper they published so that he (and, of course, others who are interested in science but don’t have the full background) could understand. Something I’m still interested in doing- but totally challenged by. How do you communicate a large amount of assumed knowledge in a way that’s accessible to everyone? He also suggested that I could write a scientific paper not in prose, but in poetry- an idea that is pretty antitheitic to the standard by-the-book scientific paper. Also a challenge I’m still wrestling with.

To a certain extent this is the role that scientific journalism plays – distilling the essence of a scientific study down to easily readable terms and placing it in the broader context of the field and previous research. Some journals (PLoS journals, for example) now require a synopsis of the papers to be provided that will be accessible to a wider audience. I believe for exactly this purpose. This is a more general problem since it does not just pertain to the scientist-layperson  divide, but also within the sciences. I am highly educated. I spent something like 22 years of my life being formally educated in one form or another- and another five in post-doctoarl training, and I’m still educating myself. The problem is, I, like every other scientist I know, have a pretty narrow focus of what I know and what I’m comfortable with. I can’t read physics papers, or chemistry papers, or neuroscience papers, and immediately know what the important parts are or even how to interpret the results from sometimes highly specialized methods of exploring the universe around us. I’m essentially in the same boat as a ‘layperson’ when reading and evaluating these kinds of papers. Of course, just knowing the scientific method and how to read a technical paper in general helps immensely.

So, back to the point of the Tweet: this is certainly a problem. The “them versus us” issues is alive and well. On one side we consider scientists to be living in ivory towers, isolated and above everyone else- and maybe being disconnected from real-world problems (who can support research on duck mating habits?). On the other side we consider laypeople to be slack jawed ignoramuses ready to lay aside the wealth of scientific evidence available for the extremely important issues that confront our world (why don’t people see what a problem the emergence of antibiotic resistance is?). So the divide is as real as we choose to make it.

But here’s the thing: the divide is not nearly as pronounced as we (either side) would seem to make it out. There are plenty of “laypeople” who understand as much, or more, about physics, psychology, or soil ecology, than I do. And there are plenty of “scientists” who think about many things: economics, politics, gender equality issues, and are thought leaders in these areas. There is a great need for better communication though- perhaps through Twitter or similar social media. In fact, there have been several recent social media events that have challenged these boundaries, making science and the process of doing science more real to the general public. I’m talking about the #overlyhonestmethods hashtag (as well as several other similar events), which was criticized for laying things too bare in places, but that I think was a boon to this relationship.

We are human. We make human mistakes. We think about human problems. We do not exist in an ivory tower. We are also athletes, foodies, hipsters, enthusiasts, wives, husbands, partners, parents, lovers, artists, humorists, and trolls. I can only think, and hope, that this will bring down walls rather than putting up more of them.

What if I were my own post-doc mentor?

Recently I’ve  had time, and reason, to reflect upon what was expected of me during the early portion of my post-doc and what I was able to deliver. It started me thinking: how would I judge myself as a post-doc if I (the me right now) were my own mentor?

My post-doc started 12 years ago and completed when I landed my current job, 7 years ago. I’ve given a short introduction that includes some context; where I was coming from and what I settled on for my post-doc project.

Background: I did my PhD in a structural virology lab in a microbiology and immunology department. I started out solidly on the bench science side then worked my way slowly into image analysis and some coding as we developed methods for analysis of electron microscopy images to get structural information.

May 2001: Interviewed for a post-doc position with Dr. Ram Samudrala in the Department of Microbiology at UW. Offered a position and accepted soon after. My second day on the job, sitting in an office with a wonderful panoramic view of downtown Seattle from tall tower to tall tower, was September 11th 2001.

First idea on the job: Was to develop a one-dimensional cellular automaton to predict protein structure. It didn’t work, but I learned a lot of coding. I’m planning on writing a post about that and will link to it here (in the near future).

Starting project: My starting project that I finally settled on was to predict structures for all the tractable proteins in the rice, Oryza sativa, proteome, a task that I’m pretty sure has never been completed by anyone. The idea here is that there are three classes of protein sequence: those which have structures that have been solved for that specific protein, those that have significant sequence similarity to proteins with solved structures, and those that are not similar to sequences with known structures. Also, there’s a problem with large proteins that have many domains. These need to be broken up into their domains (structurally and functionally distinct regions of the protein) before they can be predicted. So I started organizing and analyzing sequences in the rice proteome. This quickly took on a life of it’s own and became my post-doc project. I did still work some with structure but focused more on how to represent data, access it, and use it from multiple levels to make predictions that were not obvious from any of the individual data sources. This is a area that I continue to work in in my current position. What came out of it was The Bioverse, a repository for genomic and proteomic data, and a way to represent that data in a way that was accessible to anyone with interest. The first version was coded all by me from the ground up in a colossal, and sometimes misguided, monolithic process that included a workflow pipeline, a webserver, a network viewer, and a database, of sorts. It makes me tired just thinking of it. Ultimately the Bioverse was an idea that didn’t have longevity for a number of different reasons- maybe I’ll write a post about that in the future.

Publishing my first paper as a post-doc: My first paper was a short note for the Nucleic Acids Research special issue on databases on the Bioverse that I’d developed. I submitted it one and a half years after starting my post-doc.

Now the hard part, what if I were my own mentor: How would mentor me view post-doc me?

How would I evaluate myself if I were my own mentor? Hard to say, but I’m pretty sure mentor me would be frustrated at post-doc me’s lack of progress publishing papers. However, I think mentor me would also see the value in the amount and quality of the technical work post-doc me had done, though I’m not sure mentor me would give post-doc me the kind of latitude I’d need to get to that point. Mentor me would think that post-doc me needed mentoring. You know- mentor me needs to DO something, right? And I’m not sure how post-doc me would react to that. Probably it would be fine, but I’m not sure it’d be helpful. Mentor me would push for greater productivity, and post-doc me would chafe under the stress. We might very well have a blow up over that.

Mentor me would be frustrated that post-doc me was continually reinventing the wheel in terms of code. Mentor me would push post-doc me to learn more about what was already being done in the field and what resources existed that had similarities with what post-doc me was doing. Mentor me would be frustrated with post-doc me’s lack of vision for the future: did post-doc me consider writing a grant? How long did post-doc me want to remain a post-doc? How did post-doc me think they’d be able to land a job with minimal publications?

Advice that mentor me would give post-doc me? Probably to focus more on getting science done and publishing some of it than futzing around with (sometimes unnecessary) code. I might very well be wrong about that too. The path that I took through my post-doc and to my current independent scientist position might very well be the optimal path for what I do now.

I (mentor me) filled out an evaluation form that is similar to the one I have to do for my current post-docs (see below). Remember, this was 12 years ago- so it’s a bit fuzzy. I (post-doc me) comes out OK- but having a number of places for improvement.

This evaluation makes me realize how ideas and evaluations of “success”, “progress”, and even “potential as an independent scientist” can be very complicated and can evolve rapidly over time for the same person. As a mentor there is not a single clear path to promote these qualities in your mentees. In fact, mentorship is hard. Too much mentorship and you could stifle good qualities. Too little and you could let those qualities die. And here’s the kicker: or not. What you do as a mentor might not have as much to do with eventual outcomes of success as you’d like to think.

SelfEvaluation

How would mentor me rate post-doc me if I had to evaluate using the same criteria that I now use for my own post-docs?

How would mentor me rate post-doc me if I had to evaluate using the same criteria that I now use for my own post-docs?

Goodbye to two good friends

(Note: this post isn’t nearly as sad as it might seem from the title or the introduction below)

Yesterday I lost two close friends. We had been friends for five years, though our relationships had extended a tumultuous 10 or so months before that. Given that we still have unfinished business I expect our friendships to straggle on a little longer. But really, it’s over. My friends have helped me grow in a number of important ways- become more mature, deal with different personalities, forced me to communicate more clearly and to take criticism in a constructive light. The friendships both challenged me in different ways and supported me through a fragile time in my life. I will miss both of these friends for some different reasons- and some of the same reasons.

Like many friendships they have ended because of what other people thought about them. A small number of people had comments on our friendship- some of the comments, upon reflection, were probably well-placed, others certainly were not. But that outside influence is what really broke us apart. I hope that we can become friends again in the future- but we both will have changed so much in the intervening time that we may well be unrecognizable to each other. Still it would be nice to continue this friendship.

Bye Bye

Farewell Systems Biology of Enteropathogens and Systems Virology Centers – you will be missed but not forgotten.

Here are a few mementos of our time together….

  1. Ansong C, Schrimpe-Rutledge AC, MitchellH, Chauhan S,Jones MB, Kim Y-M, McAteerK, Deatherage B, Dubois JL, Brewer HM, Frank BC, McDermottJE, Metz TO, Peterson SN, Motin VL, Adkins JN. A multi-omic systems approach to elucidating Yersinia virulence mechanisms.Molecular Biosystems 2012. In press.
  2. McDermott JE, Corley C, Rasmussen AL, Diamond DL, Katze MG, Waters KM: Using network analysis to identify therapeutic targets from global proteomics dataBMC systems biology 2012, 6:28.
  3. Yoon H, Ansong C, McDermott JE, Gritsenko M, Smith RD, Heffron F, Adkins JN: Systems analysis of multiple regulator perturbations allows discovery of virulence factors in SalmonellaBMC systems biology 2011, 5:100.
  4. Niemann GS, Brown RN, Gustin JK, Stufkens A, Shaikh-Kidwai AS, Li J, McDermott JE, Brewer HM, Schepmoes A, Smith RD et alDiscovery of novel secreted virulence factors from Salmonella enterica serovar Typhimurium by proteomic analysis of culture supernatantsInfect Immun 2011, 79(1):33-43.
  5. McDermott JE, Yoon H, Nakayasu ES, Metz TO, Hyduke DR, Kidwai AS, Palsson BO, Adkins JN, Heffron F: Technologies and approaches to elucidate and model the virulence program of salmonellaFront Microbiol 2011, 2:121.
  6. McDermott JE, Shankaran H, Eisfeld AJ, Belisle SE, Neumann G, Li C, McWeeney SK, Sabourin CL, Kawaoka Y, Katze MG et alConserved host response to highly pathogenic avian influenza virus infection in human cell culture, mouse and macaque model systemsBMC systems biology 2011, 5(1):190.
  7. McDermott JE, Corrigan A, Peterson E, Oehmen C, Niemann G, Cambronne ED, Sharp D, Adkins JN, Samudrala R, Heffron F: Computational prediction of type III and IV secreted effectors in gram-negative bacteriaInfect Immun 2011, 79(1):23-32.
  8. McDermott JE, Archuleta M, Thrall BD, Adkins JN, Waters KM: Controlling the response: predictive modeling of a highly central, pathogen-targeted core response module in macrophage activationPLoS ONE 2011, 6(2):e14673.
  9. Aderem A, Adkins JN, Ansong C, Galagan J, Kaiser S, Korth MJ, Law GL, McDermott JG, Proll SC, Rosenberger C et alA systems biology approach to infectious disease research: innovating the pathogen-host research paradigmMBio 2011, 2(1):e00325-00310.
  10. Buchko GW, Niemann G, Baker ES, Belov ME, Heffron F, Adkins JN, McDermott JE (2011). A multi-pronged search for a common structural motif in the secretion signal of Salmonella enterica serovar Typhimurium type III effector proteinsMolecular Biosystems. 6(12):2448-58.
  11. Lawrence PK, Kittichotirat W, Bumgarner RE, McDermott JE, Herndon DR, Knowles DP, Srikumaran S: Genome sequences of Mannheimia haemolytica serotype A2: ovine and bovine isolatesJ Bacteriol 2010, 192(4):1167-1168
  12. Yoon H, McDermott JE, Porwollik S, McClelland M, Heffron F: Coordinated regulation of virulence during systemic infection of Salmonella enterica serovar TyphimuriumPLoS Pathog 2009, 5(2):e1000306.
  13. *Taylor RC, Singhal M, Weller J, Khoshnevis S, Shi L, McDermott J: A network inference workflow applied to virulence-related processes in Salmonella typhimuriumAnnals of the New York Academy of Sciences 2009, 1158:143-158.
  14. *Shi L, Chowdhury SM, Smallwood HS, Yoon H, Mottaz-Brewer HM, Norbeck AD, McDermott JE, Clauss TRW, Heffron F, Smith RD, and Adkins JN. Proteomic Investigation of the Time Course Responses of RAW 264.7 Macrophages to Salmonella Infection. Infection and Immunity 2009, 77(8):3227-33.
  15. *Shi L, Ansong C, Smallwood H, Rommereim L, McDermott JE, Brewer HM, Norbeck AD, Taylor RC, Gustin JK, Heffron F, Smith RD, Adkins JN. Proteome of Salmonella Enterica Serotype Typhimurium Grown in a Low Mg/pH Medium. J Proteomics Bioinform. 2009; 2:388-397.
  16. *Samudrala R, Heffron F, McDermott JE: Accurate prediction of secreted substrates and identification of a conserved putative secretion signal for type III secretion systemsPLoS Pathog 2009, 5(4):e1000375.
  17. *McDermott JE, Taylor RC, Yoon H, Heffron F: Bottlenecks and hubs in inferred networks are important for virulence in Salmonella typhimuriumJ Comput Biol 2009, 16(2):169-180.
  18. *Ansong C, Yoon H, Norbeck AD, Gustin JK, McDermott JE, Mottaz HM, Rue J, Adkins JN, Heffron F, Smith RD: Proteomics Analysis of the Causative Agent of Typhoid FeverJ Proteome Res 2008.

*these were really from slightly before our time- but I’ll count them there anyway

Science figures we could do without

Everyone knows them- Figure 3 in <big name journal> that’s supposed to be telling you something important, or a lot of important things, but instead is either uninterpretable or just plain misleading. An excellent compendium of bad figures can be found at Bad Figure. Others have made collections of similar things: scientific figures, infographics.

I thought I’d add to this by trying to define some classes of bad figures and give some examples.

Note: the examples I’ve chosen ARE NOT a statement about the quality of the originating paper in terms of their results or conclusions. They’re just examples to illustrate these different classes of poor figures.

The “Trump Up”:

This kind of figure could be expressed as a Table or as a single panel of another figure, or even just reported as numbers in the text. However, it would look REALLY COOL as some incomprehensible figure with lots of lines.

This beauty of an example comes from a Nature Genetics paper, “Bayesian method to predict individual SNP genotypes from gene expression data”, Nature Genetics 44, 603–608. Take some simple results and add a bunch of lines. It looks like a bunch of snow angels. It does look like there’s some more complicated information on the predicted genotypes side- but is there actual usable information there?

(a–c) Sample IDs were sorted for each semicircle (right, predicted genotypes; left, observed genotypes; numbers on the outside of the semicircles represent indexed sample numbers). Results are shown for experiments in which RYGB liver was used as the training set for HLC liver (a), HLC liver was used as the training set for RYGB liver (b) and RYGB adipose was used as the training set for HLC liver (c). In the case of a correct pairing (with adjusted minimum Pi,j of <1 × 10−5), the connection between the semicircles was a straight line passing the circle center (blue lines). In the case that no match for a given individual was identified, no line existed: for example, tick A in a–c. The blue curves outside of the right semicircles denote adjusted minimum Pi,j (−log10 transformed) for matching predicted genotype vectors to observed genotype vectors. For convenience, this value was capped at 16. If the value was <5, the curve is shown in red, indicating lack of statistical support for any match. (d) Matching was performed in the HLC liver set to which RNA-DNA mispairing and orphan samples had been added. In the case of a mispairing detected at adjusted minimum Pi,j of <1 × 10−5, the line connecting the semicircles will not be straight (red connections). The predicted genotype of subject 31 (tick A) best matches the observed genotype of subject 98 (tick D). There was no line connecting the observed genotype of subject 31 (tick C). In the case of orphan RNA (for example, subject 137), there was no connection between the predicted genotype (tick B) and observed genotype (tick E). The green curve outside the right semicircle show adjusted −log10 (Pi,i).

(a–c) Sample IDs were sorted for each semicircle (right, predicted genotypes; left, observed genotypes; numbers on the outside of the semicircles represent indexed sample numbers). Results are shown for experiments in which RYGB liver was used as the training set for HLC liver (a), HLC liver was used as the training set for RYGB liver (b) and RYGB adipose was used as the training set for HLC liver (c). In the case of a correct pairing (with adjusted minimum Pi,j of

The “Glamor Cram”:

Many “glamor” journals like Science and Nature have strict limits on the number of pages and figures in a paper. Because some studies being published are large and extremely complicated this can result in some funny, odd, and disturbing outcomes in terms of paper organization. For example, the study of ovarian cancer from The Cancer Genome Atlas published a couple of years ago was 5 pages in the journal and 130 pages of supplemental methods and results (not counting larger tables and data files in the supplemental results). Another effect is that figures can sometimes be crammed with information, because, you know, there’s a LOT to share. An example is from another TCGA paper on breast cancer. It IS interpretable, and even elegant in its own way. But, man, is it complicated- with multiple levels of color and borders having different meanings. Whew. Exhausting.

Mutual exclusivity modules are represented by their gene components and connected to reflect their activity in distinct pathways. For each gene, the frequency of alteration in basal-like (right box) and non-basal (left box) is reported. Next to each module is a fingerprint indicating what specific alteration is observed for each gene (row) in each sample (column). a, MEMo identified several overlapping modules that recapitulate the RTK–PI(3)K and p38–JNK1 signalling pathways and whose core was the top-scoring module. b, MEMo identified alterations to TP53 signalling as occurring within a statistically significant mutually exclusive trend. c, A basal-like only MEMo analysis identified one module that included ATM mutations, defects at BRCA1 and BRCA2, and deregulation of the RB1 pathway. A gene expression heat map is below the fingerprint to show expression levels.

Mutual exclusivity modules are represented by their gene components and connected to reflect their activity in distinct pathways. For each gene, the frequency of alteration in basal-like (right box) and non-basal (left box) is reported. Next to each module is a fingerprint indicating what specific alteration is observed for each gene (row) in each sample (column). a, MEMo identified several overlapping modules that recapitulate the RTK–PI(3)K and p38–JNK1 signalling pathways and whose core was the top-scoring module. b, MEMo identified alterations to TP53 signalling as occurring within a statistically significant mutually exclusive trend. c, A basal-like only MEMo analysis identified one module that included ATM mutations, defects at BRCA1 and BRCA2, and deregulation of the RB1 pathway. A gene expression heat map is below the fingerprint to show expression levels.

 The “Ridiculome” (aka The Hairball):

This is one that I’ve been guilty of- so I’ll use an example from one of my own papers (again: this isn’t a critique of the quality of these papers, which is impeccable in this example of course). Lots of interactions? Why not show them! All. At. Once. Because that will really illustrate your point about your work being super complicated.

From “Enhanced functional information from predicted protein networks.”, Trends in Biotechnology 22:60-62. A rainbow-colored Death Star exploding? A psychedelic pincushion? Who can tell?

Figure 1. Comparison of predicted protein networks for E. coli. (a) Protein pairs and their mutual information scores based on phylogenetic profiling were used to generate a network for E. coli. Figure generated using data from [4, supplementary information] (b) Protein interactions were predicted using Bioverse [7] based on finding pairs of proteins similar in sequence to proteins from a database of experimentally determined interactions. Figure generated using data from Bioverse (http://bioverse.compbio.washington.edu). For both networks, nodes representing proteins are colored based on their gene ontology (GO) [19] category and the 220 proteins present in both networks are outlined in blue. Edges represent the predicted relationships between proteins [functional linkages in (a) and protein interactions in (b)] and are colored by confidence (a) or mutual information score (b).

Figure 1. Comparison of predicted protein networks for E. coli. (a) Protein pairs and their mutual information scores based on phylogenetic profiling were used to generate a network for E. coli. Figure generated using data from [4, supplementary information] (b) Protein interactions were predicted using Bioverse [7] based on finding pairs of proteins similar in sequence to proteins from a database of experimentally determined interactions. Figure generated using data from Bioverse (http://bioverse.compbio.washington.edu). For both networks, nodes representing proteins are colored based on their gene ontology (GO) [19] category and the 220 proteins present in both networks are outlined in blue. Edges represent the predicted relationships between proteins [functional linkages in (a) and protein interactions in (b)] and are colored by confidence (a) or mutual information score (b).

The “Token Network Figure”

Why, yes, Reviewer 3, we DO have a license for MetaCore/Ingenuity Pathway Analysis. We’ll get our post-doc right on that. A “systems-level” figure you say? Certainly- right away. What does it mean? Well there are molecules. And they’re connected. They’re ALL connected see? I mean to say that it’s complicated. And some of these things depicted were actually found by us to be significant. We’ll let you guess at which those are. But one of our favorites shows up smack in the middle and looks really meaningful. I’m sure there’s something good here. There must be. There are so many different interactions. My god, it’s full of proteins.

Yes. There are interactions here.

Yes. There are interactions here.

The “Included-By-Popular-Demand”:

At least, that’s what I’m guessing. The paper has been returned from review and the reviewers want to see “a figure showing relationship X included”. The data doesn’t look *that* great, but the reviewers are asking for it so it’s included as a figure. The reviewers are satisfied and give the paper a go-ahead. If they had reviewed that same figure the first time around they would have suggested it be removed. Sigh. An example of this is from this paper: “Evolutionary Rate in the Protein Interaction Network” Science 296 (5568): 750-752 

While it’s true that a “trend” shown here could be significant (given enough data points) this figure totally doesn’t show that. Why would the authors think that this supports their conclusions solidly, or why it should be Figure 1 in their Science paper is beyond me.

Figure 1 The relation between the number of protein-protein interactions (I) in which a yeast protein participates and that protein's evolutionary rate, as estimated by the evolutionary distance (K) to the protein's well-conserved ortholog in the nematode C. elegans.

Figure 1
The relation between the number of protein-protein interactions (I) in which a yeast protein participates and that protein’s evolutionary rate, as estimated by the evolutionary distance (K) to the protein’s well-conserved ortholog in the nematode C. elegans.

 The “Really Should Be A Better Way to Show This”:

This is where it gets personal. I really don’t like these, but do admit that under certain circumstances with certain kinds of data they can be effective. The spider plot or radar charts. They attempt to show differences between multiple categories of data, for multiple variables. And they do so by changing directions in dizzying fashion.

This post presents an excellent critique of radar charts and some alternatives that work better to present these kinds of data.

Here's an example of a spider plot. It's not even that bad as these plots go- but still very hard to interpret.

Here’s an example of a spider plot. It’s not even that bad as these plots go- but still very hard to interpret.

Fin

This list is woefully incomplete. Of course, there are many other categories that could be added here. Please suggest some.