Vince Buffalo – Vince Buffalo

Why Do Species Get a Thin Slice of π? Revisiting Lewontin's Paradox of Variation

Wed, 01 Sep 2021 00:00:00 +0000

The Great Obsession of population geneticists, to borrow John Gillespie’s words, is genetic variation. As an evolutionary biologist, it’s rather hard to not be obsessed with genetic variation, for it’s the ultimate source of the two most striking features of life on earth: the mind-boggling diversity of species, and adaptations so utterly clever they look as though they were assembled by a designer. Both life’s dizzying diversity and cunning adaptations are the result of evolutionary processes like natural selection and numerous historical accidents, overlaid on one another like brush strokes on canvas, to give us the snapshot of life on earth today.

Population geneticists’ obsession with genetic variation is a result of the way we look at evolution: evolution as the change in the genetic composition of a population through time. By carefully studying genetic variability, we hope we have some chance of figuring out which evolutionary processes played out in the past. Some biologists dismiss this view as “beanbag genetics”, since population geneticists like to reduce a population down to the simplest components: the frequencies of various genetic variants, or alleles, in that population. While surely this is an oversimplified view, the upside is that it is particularly amenable to thinking about mathematically. Indeed evolution often occurs so slowly that to understand it we need to use mathematics to figure out what a population looked like long before we were born, or what it will look like long after we’re dead.

Our field has constructed a rich mathematical theory of evolution over the last one-hundred years, but it was only a half-century ago we got our first actual estimates of the genetic variation in a fruitfly species named Drosophila pseudoobscura, from the work of Richard Lewontin and Jack Hubby. After a glimpse of the data, one theory of evolution was slain, and its rival theory was in disarray. Within a few years, evolutionary biologists were measuring genetic variation in all kinds of species during the “find ’em and grind ’em” era, named for the unceremonious way numerous flies, crabs, plants, and individuals from other species met their fate in the quest to measure variability.

Richard Lewontin's seminal 1974 book, The Genetic Basis of Evolutionary Change. Source: Wikipedia.

With new data of levels genetic variability across species, another theory was soon on chopping block. This theory was the neutral theory, and Dick Lewontin’s 1974 synthesis of the find ’em and grind ’em era pointed out a seeming contradiction between this theory and the new estimates of genetic variability. He named this the “The Paradox of Variation”, and it’s an enduring riddle in my field of evolutionary genetics. My recent paper published in eLife attempts to make some progress on this longstanding paradox. I’ve written this blog post to introduce a general audience to these topics and why we study them, and then give an overview of my paper (you may wish to skip ahead and start there).

Lewontin and Hubby’s Quest to Measure Genetic Variability

Before Lewontin and Hubby’s work measuring genetic variability in Drosophila pseudoobscura, evolutionary biologists were uncertain of how high or low the genetic variability was within populations. As I write this, I am surrounded by two examples of the extremes of this genetic variability spectrum: my Monstera deliciosa houseplant (also known as the Swiss cheese plant for it’s deeply fenestrated leaves) and the Drosophila fruit flies that have invaded the kitchen compost bin I’ve procrastinated emptying. While its incorrect to think of the monsetera plants in people’s homes as a “population” since they do not interbred with one another, they serve as a good example a group of organisms with low genetic diversity. This is because most monstera houseplants are clones of one another, created by taking a clipping of one plant, letting it root and grow, then taking a clipping of this plant, and so on. Unlike parent and offspring which are separated by a generation, all monstera plants are identical siblings; perhaps some mutations lead to small amounts of genetic variability, but this genetic variability is minuscule compared to what’s created by free mating in a sexually-reproducing species.

The gel electrophoresis of the esterase-5 gene in Drosophila pseudoobscura. Each bar is where a sample's protein ended up after being drawn through a thick gel with electrical current. The different bars represent different samples, and their varying positions reflect the different protein variants, which due to their different shapes and charges, move through the gel at different speeds.

By contrast, my compost bin population of Drosophila melanogaster harbors massive amounts of genetic diversity. Part of the reason why Drosophila melanogaster have such high genetic variability is because their population sizes are so large. Drosophila melanogaster originated from equatorial Africa, yet followed humans around the globe living off our trash heaps and spoiled fruit. One way to measure genetic diversity is to take two random chromosomes and count the differences between their DNA sequences, then repeat this for another two random chromosomes, and so forth. Population geneticists call this “pairwise” measure of genetic variability the Greek letter $\pi$ (I imagine much to the disappoint of mathematicians). For Drosophila, $\pi_\text{flies} \approx 1%$, or one difference per 100 DNA basepairs, and for humans, $\pi_\text{humans} \approx 0.1%$, or one difference per 1,000 basepairs. Humans have much lower genetic diversity than fruitflies. Lewontin and Hubby explain the drastic importance of this simple quantity:

In a sense, a description of the genetic variation in a population is the fundamental datum of evolutionary studies; and it is necessary to explain the origin and maintenance of this variation and to predict its evolutionary consequences. […]

Thus, one end goal of measuring variability within and across species is to answer the fundamental question of evolutionary genetics: what evolutionary processes are compatible with the observed levels of genetic variation?

The Evolutionary Theories Killed by Measuring Genetic Variation

¹ This is how Lewontin himself sets up the problem in his book The Genetic Basis of Evolutionary Change.
² If this view sounds almost vaguely political, you’re in good company. Lewontin, who never missed an opportunity to place scientific theories in their historical context, writes: “A basis for racism may also flow from the concept of a wild type, since if there is genetic type of the species, those who fail to correspond to it must be less perfect. Platonic notions of type are likely to intrude themselves from one domain to another, and Dobzhanksy (1955) was clearly conscious of this problem when he attacked the concept of a wild type.”
³ For example, one observation that was at odds with the balance theory was the observed rate of inbreeding depression was far too slow if most fitness variation was maintained by overdominance.

Before I introduce Lewontin’s Paradox of Variation and the neutral theory, it’s worthwhile to set the stage with the theories of genetic variability that preceded it.¹ At the time of Lewontin and Hubby’s work, proponents of the classical theory believed there wasn’t much genetic variation between individuals. This was because selection was thought to be extremely powerful: a new mutation that increased survival or the number of offspring quickly replaced all alternatives. The genetic composition of a population was more or less uniform, as it moves from one perfectly adapted state to another under the steady marching orders of natural selection. Of the variation that existed, it was primarily from harmful mutations that ‘broke’ organisms from their archetypal “wild-type”.²

By measuring their genetic variability, Lewontin and Hubby showed genetic variability was far too abundant for the classical theory to be correct. The alternative view, the balance theory, held that natural selection maintained vast amounts of genetic variation through a variety of mechanisms; aspects of this theory were successfully challenged through other experiments and arguments³. How evolutionary processes, whether random change, natural selection, historical accidents (e.g. a population bottleneck), combine to determine the central quantity of evolutionary genetics, genetic variation, remained a mystery.

The Neutral Theory

This brings us to the architect of the third view of genetic variation: Motoo Kimura. Kimura’s view was that the majority of genetic variation was selectively neutral: it did not impact the survival or number of offspring individuals had. Free from the marching orders of selection, such neutral variation lingered in populations, drifting up or down in frequency due the vagaries of which chromosomes were passed down to offspring (Mendelian segregation), and random differences in survival and offspring number that didn’t have a genetic cause. We call this process genetic drift. Kimura’s neutral theory was in some ways an extension of Sewall Wright’s view that evolution was governed as much, or more, by random chance as it was by natural selection. While here we’re focused on Kimura’s work on neutral polymorphism within a population, much of Kimura’s motivation for his neutral theory was to explain observations in molecular evolution, such as the steady clock-like ticking of amino acid differences between species.

Suddenly under neutral theory, the mystery of why there was so much genetic variation wasn’t such a mystery. If the variation was neutral, natural selection couldn’t act on it. Surely, some beneficial mutations would enter the population —adaptations of course had a genetic basis— but these mutations were rare, and they would quickly replace their alternative and didn’t persist for long in the population. Lewontin called neutral theory the “neoclassical theory” because of this similarity.

Setting Goddess Tyche in charge of evolutionary change has another benefit. If we can ignore natural selection, and thus all the uncertainty about the strength and direction of natural selection, the mathematics of evolution become much simpler. Under a model where $N$ individuals freely mate, new variation is created through mutation at rate $\mu$ per generation per basepair (i.e. there’s a $100 \times \mu$% chance that any one basepair mutates in a given generation), and new variation is lost through random drift at rate proportional to $1/N$ (i.e. slower in larger populations, fast in small populations). Ultimately, like the level of water in a dam, a balance is met between new mutations entering and old mutations going extinct in the population: this is the equilibrium genetic diversity. A bit of mathematics says that under this toy model, equilibrium genetic diversity should be $\pi \approx 4N\mu$. In a big population (think of all the fruitflies in the world), there should be high genetic variability. If population sizes are small (think of a small island), soon everyone becomes everyone else’s cousin, and genetic diversity is lost.

Lewontin’s Paradox of Variation

So while neutral theory is mathematically alluring, and not at odds with the high level of genetic variability found in various species, Lewontin noticed a critical problem. Under Kimura’s neutral theory, the heterozygosity at a locus should depend on the product of mutation rate and population size, $N \mu$, such that:⁴

\[ \pi = \frac{4N\mu}{1 + 4 N \mu} \]

However, the range of genetic variability across species was surprisingly narrow. Lewontin lays out why this is a problem visually in his book:

A figure from Lewontin's 1974 book The Genetic Basis of Evolutionary Change on the "Paradox of Variation" (p. 209). The curved line is the hypothetical heterozygosity (another term for genetic variability) given by $\pi = 4N\mu / (1 + 4N\mu)$. The range of variation observed in early studies ranges between $\pi \approx 6\%$ to $\pi \approx 20\%$, suggesting the product $N\mu$ is between 0.015 and 0.06. For a gene mutation rate of $\mu = 10^{-6}$, this suggests that $N$ ranges from 3,750 to 15,000 --- a minuscule factor of four difference.

As Lewontin explains,

The observed range of heterozygosities over all the species […] lies in the sensitive region, between 0.056 and 0.184. This range corresponds to values of $N\mu$ between 0.015 and 0.057. Since there is no reason to suppose that mutation rate has been specially adjusted in evolution to be the reciprocal of population size for higher organisms, we are required to believe that higher organisms including man, mouse, and Drosophila and the horseshoe crab all have population sizes within a factor of 4 of each other. […] The patent absurdity of such a proposition is strong evidence against a neutralist explanation of observed heterozygosity.

While Lewontin’s estimated range of heterozygosities only included 13 species (Table 22, p. 117), this problem wasn’t just an artifact of small sample size. Throughout the 1970s, estimating protein heterozygosities with gel electrophoresis became a cottage industry. In the decade since Lewontin’s book, over a thousand species had published estimates of their protein variability, which confirmed the narrow range of genetic variability found by Lewontin. Nevo and colleagues published a survey of estimates of protein heterozygosities for 1,111 different species, finding that heterozygosity ranges only from 0% to 30%:⁵

Figure 2c from Nevo et al. The Evolutionary Significance of Genetic Diversity: Ecological, Demographic and Life History Correlates (1984).

So Lewontin’s Paradox of Variation seemed to be a rather serious problem for Kimura’s neutral theory, and soon folks were looking for other evolutionary processes that were consistent with the observed data.

Lewontin’s Paradox and the Hitchhiking Model

Lewontin wrote his book while on sabbatical in John Maynard Smith’s lab at the University of Sussex. The neutral theory was under attack from multiple angles, but especially so in Britain, where a cultural that treasured naturalism and adaptive storytelling fostered the conviction that every genetic variant must have some adaptive benefit.⁶ As Maynard Smith described,

The whole tradition of British population biology had been, if you find a genetic variability, it must have some kind of selective explanation, and if at first you don’t find it, you must try, try, and try again until you do.

Perhaps in the spirit of this tradition, Maynard Smith and John Haigh tried to find a selective explanation for the observed narrow range of diversity diversity. In 1974, they published their masterwork The hitch-hiking effect of a favourable gene, which develops a mathematical model of how a selected variant can decrease its surrounding genetic variability. When a new strongly selected mutation occurs, it does so on a random chromosome in the population. As this mutation increases in frequency, either by improving an individual’s odds of survival or number of offspring, it drags along its neighboring stretches of DNA as it spreads within the population. To understand why this is, it may be helpful to think of our DNA as bit like a spool of 8mm film and you can consider our genes as random frames of the motion picture. Most neighboring frames during a scene of the film end up together into the final production, much like if I pass on a genetic variant I inherited from my father, there’s a good chance that variant’s neighbor would also end up in my child. However, occasionally the film editor comes along and cuts the scene short and splices in another scene. So too can a cell’s recombination machinery cut the stretch of my father’s chromosome I’m passing on and splice in my mother’s; suddenly her genetic variants and their neighbors go on to the next generation. Our genomes, after all, are exactly half of each our parents’ genomes, but a random mosaic of our four grandparents’ genomes.

A Griswald Junior Film Splicer splicing together two bits of 16mm film (this still is from this short film).

This is recombination, and it’s a vitally important part of the evolutionary process. In particular, how often recombination occurs —how often the scenes of a film are cut and spliced to different scenes— varies across organisms and along the chromosome itself. In regions of high recombination, beneficial variants are quickly spliced off and away from their neighbors, allowing the surrounding genetic variation to persist in the population undisturbed. By contrast, in low recombination regions, a beneficial variant won’t be spliced away from even its furthest neighbors for quite some time, and vast neighborhoods of genetic diversity are wiped out whenever a beneficial mutation comes along and takes over the population. It is precisely this process that Maynard Smith and Haigh thought could be reducing genetic variability along the genome, and explain the narrow range of genetic diversity seen across species. As they say,

[This investigation] can therefore be regarded as a last ditch attempt to save the neutral mutation theory by showing that there is another process which can account for the uniformity of [heterozygosity] between species.

Nearly fifty years of population genetics has indeed shown that hitchhiking can strongly impact genetic diversity within a species. The textbook example is from work a member of my dissertation committee, Dave Begun, did as a PhD student with Chip Aquadro. They showed that there was a striking correlation between recombination rates and genetic diversity in Drosophila melanogaster, which is precisely what one would expect under pervasive hitchhiking. Genetic diversity is highest in high recombination regions, and lowest in low recombination region across the Drosophila melanogaster genome:

The correlation between pairwise diversity (a measure of genetic variability) and recombination rate in Drosophila melanogaster, from Begun and Aquadro (1992).

Later, theoretic work showed that a similar reduction in diversity can happen if new mutations are selected against, rather than selected for. This process is called background selection, as it’s continually occurring in the background of the genome. Population geneticists are still debating and developing new ways to estimate the relative strengths of hitchhiking and background selection, which collectively we call linked selection. While hitchhiking and background selection can reduce genetic diversity, and have been shown to do so in many species, the central question remained unanswered: are these selection processes strong enough to constrain genetic diversity across species to the narrow range observed? Or is there some other explanation?

The Neutralist View on Lewontin’s Paradox

Kimura and others had a simpler explanation for the narrow range of diversity. First, we have known since Sewall Wright that what is relevant to the rate of genetic drift, and thus the level of genetic variability, isn’t a species population census size (i.e. the total number of individuals in the population), but rather its effective population size. Effective population size is a bit of an amorphous concept in population genetics, but it’s fundamentally a way to account for the complexities of real populations of breeding individuals. For many populations, the idealized random mating we assume in our mathematical models is overly simplistic and ignores the realities of ecology and demography. Fortunately, we have found that in the vast majority of cases, we can account for the complexities of real populations by simply rescaling the population size, $N$, to a new “effective population size”, $N_e$.

Think of it this way: imagine 500 male and 500 female crabs on an island. There crabs freely and randomly mate. Still, it being an island, after a few generations, every crab becomes every other crab’s cousin — remember, the rate at which this happens is proportional to $1/N$. However, now imagine that a ruthless male despot crab comes along and battles the other males, and forbids them from mating. While there are still $N=1,000$ individuals on this island, within a generation every crab is the despot crab’s kin. The rate at which every crab on the island becomes a cousin is now much faster. Although this situation seems quite different to the free crab love island, it only requires adjusting $N$; specifically, we use an effective population size $N_e$. In this case, it works out that the adjusted $N_e \approx 4$.⁷

Importantly, effective population also depends on population bottlenecks. If instead of the male despot crab, the island goes 99 generations undisturbed, but then suffers a severe generation-long bottleneck as seagulls descend on the island. If only 2 crab survive, the effective population size is now reduced from $N=1,000$ to $N_e = 170$. Since it’s effective population sizes, and not census sizes, that determine levels of genetic variability, one can see how frequent population bottlenecks could explain Lewontin’s Paradox. The evolutionary history of many species also includes range expansions and colonization and extinctions — when a few individuals make it a new patch of habitable land, procreate, but ultimately may die. These dynamics also decrease effective population size and capture the messy reality of many species.

The earth looked very different over the last 20,000 years than it did today. Numerous ice sheets formed and forced species into refugia, where they may have diverged into different species or subspecies (for example, the hooded and carrion crows). This figure is from Hewitt (2000), The genetic legacy of the Quaternary ice ages.

Kimura argued that the only “serious problem that remains to be explained” with Lewontin’s Paradox was that observed heterozygosities never exceeded 30%. Considering Drosophila species, which were at the upper end of the population size and heterozygosity ranges, Kimura argued that their numbers were severely depressed by the last continental glaciation. Genetic variability takes a while to recover from a severe bottleneck, and longer for larger populations (as new mutations must escape rarity and work up to intermediate frequencies), so this severe, lasting bottlenecks during glaciations is the primary neutralist hypothesis for Lewontin’s Paradox.

Recent Work on Lewontin’s Paradox

Technological improvements in genome sequencing revolutionized population genomics, and have provided evidence that diversity within species is shaped by a mix of past demographic events, such as bottlenecks, as well as by natural selection. Lewontin’s Paradox, however, remains unresolved through the genomics era. There are a few nice pieces of recent work that set up some context for my recent paper. The first of which was Ellen Leffler et al. (2012) , which compiled estimates of diversity from genomic data. This was one of the first papers I read as I grew interested in population genomics, and certainly inspired me to continue in the field. In this paper, Leffler and colleagues survey diversity for 167 species, and find an 800-fold difference between the highest diversity species (the sea squirt Ciona savignyi) to the lowest diversity species (the wild cat, Lynx lynx). Overall, they point out the narrow range of diversity is still an open and important mystery.

The second paper is J. Romiguier et al. (2014) , which surveyed diversity and life-history characteristics across 72 species. These authors find that diversity levels across species are highly-correlated to ecological strategy. In particular they find diversity is highest when species have lots of offspring they invest little in, and lower when species have few offspring they invest more in. Intriguingly, these results suggest ecological processes are predictive of genetic diversity across species. While these ecological correlates are an important piece of the puzzle, it’s still uncertain how mechanistically such ecological processes would act to constrain genetic diversity.

More recently, Russ Corbett-Detig, et al. (2015) tested the hypothesis of whether linked selection could explain Lewontin’s Paradox. Since population sizes are very difficult to estimate for many species, Russ and colleagues considered how the strength of linked selection varies with two proxies of population size: range size and body size. Large-bodied animals (e.g. whales) have smaller population densities than small-bodied animals (e.g. mice), in part because of the energy requirements needed to sustain life are much higher than in larger-bodied animals. Similarly, we’d expect species with larger ranges to have larger population sizes than species with small ranges. Then, the authors fit a statistical model to estimate the strength of linked selection across population genomic samples for 40 species. As a former bioinformatician, I should point out this is a monumental task worthy of praise. Interestingly, they find that the strength of linked selection does seem to scale with these proxies of population size. I read this paper in my first year of graduate school and discussions about it with my PhD advisor Graham Coop were foundational to my early interest in Lewontin’s Paradox.

Graham ultimately found a limitation of Corbett-Detig et al. (2015), which he shared on BioRxiv (Coop 2016 ). While Corbett-Detig et al. do indeed find that linked selection is stronger in species with large population sizes, it still may not be strong enough to explain why the range of genetic diversity is so narrow. While linked selection was reducing diversity some 60%-80% in species with large population sizes, this isn’t enough to explain why there’s only an 800-fold difference between fruit flies and Lynx, when is likely well upwards of a million-fold difference in their population sizes. The mystery remains open, and at the end of Graham’s article it seemed a bit more likely that repeated bottlenecks and other non-selective processes could also be reducing genetic diversity to the narrow range we observe.

Estimating Census Population Sizes

This brings us to my recent paper in eLife ⁸ that hopefully adds a few more pieces to this longstanding puzzle. My paper tries to chip away at this problem from a few angles, which I’ll detail below. I also tried to include a quick review of the history and relevant literature in my article. Lewontin’s Paradox touches on lots of interesting, and often missed, parts of evolutionary genetics.

The first component of my paper was to try to actually estimate the population sizes across various animal species, and quantify the relationship between genetic diversity and population size. Early work from Soulé (1976), Frankham (1996), and Nei and Graur (1984) has looked at this relationship before, using early protein heterozygosity data:

(A) Frankham (1996) using the data of Soulé (1976), and (B) Nei and Graur (1984). The solid line is neutral expectation if the mutation rate is $\mu = 10^{-7}$.

These papers used population size estimates from the literature, or back-of-the-envelope calculations. To get a modern look at the relationship between genomic estimates of genetic diversity from previous studies (Leffler et al. 2012, Romiguier et al. 2014, and Corbett-Detig et al. 2015) wanted to find a way to roughly approximate census population sizes for hundreds of species in an automated way. One approach is to take the product of population density (i.e. how many individuals there are per square kilometer) and species range size (i.e. how wide a species range is), to get a very crude estimate of population size. The downside of this approach though, is that population densities are unknown for most species too.

Luckily, the field of macroecology provides a way out. As mentioned, animals have energetic needs that scale with their body sizes. Large-bodied animals require more energy that they procure by hunting or grazing a certain area; competition between individuals for such resources means that large-bodied animals can only live at lower densities. The shocking result is that population densities are surprisingly well correlated with body mass. Damuth (1981, 1987) quantified this; here is a figure from my paper (Figure 1–figure supplement 1 ):

The relationship between population density and population body mass from Damuth's 1987 data.

Using this relationship, I predict population density using body mass. In practice, I use body length, since this is much easier to collect and reported in a more standard way than body mass. Using a statistical routine, I predict body mass from body length, and population density from body mass.

Next, I need to estimate range size. A common approach is to use occurrence data —records of the latitude and longitude of where animals were observed— to infer the range. Using the Global Biodiversity Information Facility database, I downloaded occurrence data for the animals species I had genetic diversity and population density estimates for. I then wrote some R code to automatically infer the ranges from this occurrence data.

The GBIF occurrence data (red points) and inferred range (green polygons) of the common honeybee (Apis mellifera). This shows some of the limitation of this course-grained approach: there is some uncertainty whether the some regions have sparse observations and the algorithm is properly filling in the range, or whether these are aberrant observations in regions where honeybees don't normally live. Still, for Lewontin's Paradox, we do not need precise estimates, but rather a rough look. Compare the range of the honeybee to that of Apis cerana, which lives only in South Asia.

With the population densities and ranges estimated, I take their product to get an approximate population size (see Figure 1 of the paper for a look at the distribution of these ranges by phylum). There’s quite a bit of validation I do to ensure the numerous approximations I’ve made here are reasonable. For example, I make sure my census sizes don’t lead to predictions of the total biomass that are unreasonably small or large (see Population Size Validation and Table 1 ), since estimates of the total biomass on earth of certain animal groups are available thanks to the study of Bar-On et al. (2018) . I also do lots of other consistency checks in Appendix 3 .⁹

The Relationship Between Genetic Diversity and Population Size

Next, I merged the approximate population sizes with the genetic diversity data from Leffler et al. (2012), Romiguier et al. (2014), and Corbett-Detig et al. (2015). These data allow for a nice visualization of Lewontin’s Paradox of Variation, through the relationship between genetic diversity and population size:

The relationship between genetic diversity ($\pi$) and approximate population size ($N_c$) for 172 animal species. Note that genetic diversity varies just over three orders of magnitude, while census sizes vary over 12 orders of magnitude. The gray ribbon indicate the expected neutral diversity for a range of mutation rates ($10^{-9} \lt \mu \lt 10^{-8}$) were the diversity to be determined entirely by census size under the neutral model. Lewontin's Paradox wouldn't be a paradox if the diversity estimates fell in this thin gray area; instead they do not scale with population size, and are mostly constrained to within three orders of magnitude. The eLife article has numerous supplementary figures relevant to this figure here.

What this relationship tells us is that population size appears to impact genetic diversity in the way we’d expected (there is higher genetic diversity in species with larger population sizes – this is shown by the dashed gray line of best fit in the figure above), but as Lewontin first pointed out, genetic diversity doesn’t increase as fast as we’d expect if solely census population sizes and neutral evolution determined genetic diversity.

Is the Diversity–Population-Size relationship meaningful?

Darwin's sketch of a phylogenetic tree in his First Notebook on Transmutation of Species, 1837. Source: Wikimedia commons.

We might look at the relationship above and conclude that there is a significant relationship between population sizes and genetic diversity. Unfortunately, there’s a statistical conundrum that crops up whenever we draw lines of best fit through data points like species. Species are connected by an underlying phylogenetic tree; certain species share common ancestors more recently than others. This was one of Darwin’s brilliant ideas, sketched out in 1837 in his notebook (figure above). In a seminal article in 1985, Joe Felsenstein pointed out that the underlying species tree creates a statistical problem whenever we want to make comparisons across species (as I’m doing here), and this statistical issue should be accounted for.

We can understand this statistical issue, known as phylogenetic non-independence, with a story. Imagine going to a large family reunion, where there are two sides of the family: all the descendents of your great-great-grandfather, and all the descendents of his sister, your great-great-aunt. You look around at all your family members, and observe that there seems to be a statistical relationship between having freckles and being tall. Your tallest relatives tend to have the most freckles, including your great-great aunt’s husband, who towers over you. Other relatives are more average height, and do not have freckles. If you were to collect data and quantify this relationship, you may conclude that somehow these two traits are correlated in a meaningful way; perhaps there’s some biological process that underpins both characteristics.

However, this ignores something: all the relatives that are tall and have freckles descend from your great-great aunt and uncle, and all the average height relatives without freckles descend from your great-great grandfather (who is shorter and doesn’t have freckles). There isn’t necessarily a meaningful relationship between these two traits — it could just be an accident of who shares ancestry (and thus the genes that determine traits) with whom and what traits they happen to have. The family tree here breaks the statistical assumption of independence, since everyone’s related and thus not independent of one another.

This same problem occurs when we compare traits across species too, traits like genetic diversity and census size. John Gillespie, who I introduced at the start of this article, suspected that early pictures of the diversity–population-size relationship may be entirely a spurious artifact of this shared ancestry. In his 1991 book, he has the following figure:

John Gillespie suspected that the relationship between population size and genetic diversity found by Nei and Graur (1984) could be entirely an artifact of two groups of animals: carnivores and fruitflies (1991).

While some previous work has addressed this statistical quandary in clever ways, it’s been limited by the difficulty of building big species phylogenetic trees. In my paper, I use a tool called datelife to build these trees, and then account for this tree structure using something known as Phylogenetic Comparative Methods¹⁰. These models account for the way certain groups of related species may all differ from the line of best fit in a systematic way, which is a violation of this statistical model. Overall, I find that even accounting for the species tree, the relationship between population size and diversity is significant statistically. In fact, diversity is also well-predicted by range and body mass (see Supplementary Figure 2–figure supplement 3 .

The estimated ancestral population sizes and diversity levels across species (colors), overlaid on the phylogenetic tree of all species for which data was available.

While the relationship between diversity and population size is statistically significant after accounting for shared ancestry, my work finds Gillespie’s concerns are indeed substantiated. Athropods (insects, spiders, and crustaceans) and vertebrates do indeed form to clusters as he suspected. Insects typically have large population sizes and high genetic diversity, while vertebrates typically have smaller population sizes and lower genetic diversity. Still, when I look at the diversity–population-size relationship within each of these groups, I find it is still statistically significant .

There are a few other analyses I did from this macroevolutionary perspective which I won’t discuss here for brevity’s sake. Overall, I came away from this part of the project thinking that considering the Lewontin’s Paradox from a macroevolutionary perspective will be critical in resolving this puzzle. In particular, it may be important to consider how diversity and population size change along the species tree, and how the speciation and extinction processes that lead that give rise to the species we see today interact with genetic diversity and population size.

Can Selection Explain Lewontin’s Paradox?

The final component of my paper is investigating whether selection could explain the shortfall between the observed genetic diversity levels across species, and the diversity we would expect under the neutral theory if effective population sizes were equal to census population sizes (e.g. the gray ribbon in this figure). As described earlier, one of the determinants of how strongly selection can impact genetic diversity is how much recombination there is. The level of recombination varies across species, so if I have any hope roughly predicting how strong selection can get, I need to know how much recombination there is in each species.

Luckily, Jessica Stapley and colleagues published a very nice survey of recombination across animal species. Simply put, the last part of this project wouldn’t have worked without this nice dataset, so I am quite grateful for this work. Using this dataset, I investigated the relationship between recombination map length (the expected number of times chromosomal breaks occur per generation) and my population size estimates. I find that species with large population sizes (such as fruitflies) typically have less recombination (shorter map lengths) than species with smaller population sizes (such as humans). I show this in Figure A below:

(A) The relationship between recombination map length and population sizes. These triangle points are eusocial/social species like ants that have adaptively longer map lengths (see Wilfert et al., 2007 ). (B) The relationship between genetic diversity and population size, with the predicted diversity under hitchhiking and background selection overlaid over it (blue ribbon). The ribbon is the diversity for a variety of mutation rates ($10^{-9} \lt \mu \lt 10^{-8}$).

While recombination map lengths are a critical parameter that mediates the strength of selection, there are a few other parameters we unfortunately do not have good estimates of for most species. For example, the rate that new deleterious mutations flow into a population determines diversity, since if more harmful mutations have to be purged from the population, any linked genetic diversity would be purged as well, leading to a net loss of genetic variability. Similarly, we’d also need an estimate of how many beneficial mutations flow into a population each generation, since these also act to reduce diversity through the hitchhiking effect.

While we don’t have estimates of these key parameter for the majority of species in my study, we do have very good estimates for the darling species of evolutionary biology: the fruitfly. In a very nice study by Eyal Elyashiv and colleagues , they developed a sophisticated statistical approach to estimate these key selection parameters for Drosophila melanogaster. Drosophila, with its very large population sizes, has been known to experience some of the strongest selection of any species we’ve looked at. So while I cannot predict the strength of selection for each species, I can predict another useful quantity: if selection were as strong in all species as it is in Drosophila melanogaster, would selection sufficiently reduce diversity to recreate the diversity–population-size relationship I see in the data? It turns out, the answer is no (see Figure (B) above). Essentially, there is too much recombination in species with moderate population sizes — even if we assume they experience absurdly high levels of selection, similar to what we expect in Drosophila, the reduction in genetic diversity caused by selection is severely weakened by the large amount of recombination they experience. It seems that using our current models of linked selection, there is not a plausible way that Lewontin’s Paradox could be solved by selection. In this sense, my work extends Graham’s work: not only do current estimates of the strength of selection seem incapable of explaining the narrow range of diversity, no plausible estimates seem like they could. Indeed, I even try to increase all the key selection parameters ten-fold (Figure 4–figure supplement 3 , and still fail to see that predicted diversity matches the observed diversity).

So How Do We Solve Lewontin’s Paradox?

So where does this leave us? From my perspective, my work strengthens the arguments against selection as an explanation for Lewontin’s Paradox. Alternative hypotheses, such as the effect of complex demographic histories like repeated bottlenecks, appear to be a more likely explanation for Lewontin’s Paradox. I also suggest in the discussion that some interaction of macroevolutionary processes, like speciation and extinction dynamics, could play a role in the pattern of diversity we see across species today. At the heart of the problem is that the genetic diversity across species we see today is the result of multiple overlaid processes occurring at very different timescales: ecological, evolutionary, and historical. As I argue in the conclusion, Lewontin’s Paradox may not be fully resolved for some time because the explanation requires synthesis and model building at so many different disciplines.

Richard Lewontin at the chalkboard. It looks like he's explaining the interaction---and inseparability---of genotype and environment. Source: Why Evolution is True.

Finally, I should mention that while this paper was in its second round of reviews, Dick Lewontin passed away. I was deeply saddened by this news, as were countless of his colleagues, students, and a large number of other younger scientists that were deeply inspired by his work and approach to science. Lewontin was not just a preeminent scientist, but an activist and outspoken opponent of the misuse of genetics for racist ends. He practiced science in a way that was always keenly aware of its larger social context. This is now much more commonplace than it used to be, in part thanks to him. I suggest reading his takedown of Arthur Jensen’s racist misuse of genetics , The Apportionment of Human Diversity , and the The Analysis of Variance and The Analysis of Causes . I’ll never forget reading Lewontin and Cohen (1969) when I took Sebastian Schreiber’s population ecology course during graduate school, nor Lewontin’s chapter in the textbook Building a Science of Population Biology (which I reference at the end of my article). The obituaries in the New York Times , Nature , and the Society for Molecular Biology and Evolution are well-worth reading. Jerry Coyne’s stories of the lab cultural and life of Lewontin are particularly fun to read.¹¹

The Genome-wide Signal of Linked Selection in Temporal Data

Thu, 20 Aug 2020 00:00:00 +0000

The last chapter of my dissertation with Graham Coop was recently published in PNAS (pdf , bioRxiv ) last week. In an effort to communicate my research to a broader audience, I have written two blog posts on our work. The first post , is meant to introduce the historical context and concepts like linked selection and polygenic adaptation to a non-scientist, and the second post, below, describes our work on temporal covariance as a signature of polygenic linked selection and its application to four evolve-and-reseqeunce data sets.

In the last post , I explained two longstanding problems in the field of evolutionary genetics. The first is detecting adaptation on polygenic traits from temporal genomic data. Temporal data is gathered by sampling a population through the generations and sequencing these samples, and provides us with a immense amount of information about the evolutionary process over short timescales. Yet even with this amount of data, distinguishing allele frequency changes caused by polygenic selection from that random genetic drift is a challenge. A population could be adapting over short timescales —we might even observe drastic changes in a trait over a few generations— yet it could be impossible to see the signature of such strong selection at the DNA level.

The second related problem is how we quantify the roles of drift and linked selection in determining genome-wide allele frequency changes. Since the debates between Fisher and Ford, and Wright, evolutionary geneticists have disagreed on the relative roles played by genetic drift and natural selection in determining allele frequency changes. Since their time, we have learned selection at a site exerts a strong influence on its linked neighbors, known as linked selection. This perturbs the frequency trajectories of alleles through time, and to have a complete view of how populations evolve we need to understand this process. With temporal data, we could perhaps directly estimate the effects of linked selection on drift on allele frequency change.

The Quantitative Genetics View of Linked Selection and Temporal Covariance

In our first paper (Buffalo and Coop, 2019 ), Graham and I proposed that one promising signal that could be used to detect polygenic selection in temporal genomic data is temporal covariance. Our work builds of many excellent papers but three in particular: Robertson (1961), and Santiago and Cabarello (1995, 1998)¹.

**Figure 1** The first discussion of linked selection in a quantitative genetics context was F.H.W. Morley discussing selection in Australian sheep for merino wool: "in a flock exposed to selection, the genetically superior individuals will tend to be most inbred. As a corollary, selection increases the approach to homozygosity, not only at loci carrying genes determining the character in question but at all loci." ([1954](https://www.publish.csiro.au/cp/AR9540305)). (*image source: [Wikipedia](https://commons.wikimedia.org/wiki/File:Macarthur_stamp_sheep_1934.jpg)*)

These papers (and others) establish what I’ll refer to as a quantitative genetics view of linked selection, and implicitly describe temporal covariance. While the classic linked selection work (i.e. hitchhiking and background selection, which I explain in the first post ) describes how a neutral allele behaves when it is a close neighbor of a new mutation that affects fitness, the quantitative genetics view of linked selection often seeks to understand how a neutral allele behaves when it is much more distant —perhaps even on a different chromosome— from the genetic variation that is selected upon. Furthermore, this view supposes the genetic variation that determines fitness is polygenic, and results from standing variation, meaning it is present at appreciable frequencies in the population (i.e. not the rare new mutations of the classic linked selection work).

Figure 2 Wright's (1940) figure of extinction and recolonization in a population. Population lineages are expanding through time from left to right and space (top to bottom), with groups going extinct and re-establishing. The genetic drift in a population with such a complex breeding process can be represented by standard population genetic models with a rescaled effective population size.

One of Sewall Wright’s ingenious ideas was to recognize that many different breeding structures we might see in nature (e.g. organisms capable of self-fertilization, or the extinction-recolonization dynamics depicted above) can still be described by standard models of genetic drift, as long as the population size is rescaled appropriately. We call this effective population size², and the early work of Robertson first described the long-run effect of selection on a polygenic trait exerts on a neutral site as a reduction in effective population size. This, in essence, means selection can make it seem genetic drift is running faster since changes are larger per unit time. The fact that many types of selection have effects quite similar to random genetic drift occurring in a smaller population is one reason why the effects of selection and drift can be hard to distinguish. This is precisely why Fisher and Ford first had to estimate the population size to test whether selection caused the decline in frequency of the dark wing color variant: they needed to determine the magnitude of genetic drift to see if the frequency changes were too drastic to be caused by drift alone, and were more likely to be caused by selection.

While Robertson, and Santiago and Cabarello’s work expressed the linked selection effects felt by a neutral allele during polygenic selection as a reduction in effective population size, their work tells us there is one key difference between a neutral allele randomly drifting in the population, and a neutral allele affected by linked selection: with linked selection, the neutral allele’s frequency changes between two generations are correlated with the changes between later generations. Imagine following a neutral allele as it descends down lineages, forward through the generations towards the present. In one generation, the neutral allele finds itself in a father carrying genes very well suited for his environment, and as a result, he leaves many offspring. Each child inherits some fraction of his well-suited genes, and possibly, the neutral allele as well. Within this family, the neutral allele’s frequency increases. As long as the genes that suited him well in this environment are also beneficial in his children’s generation, they too will leave more offspring. So too will the neutral allele’s frequency also rise in this generation, as long as it remains associated with a fraction of genes well-suited for this environment.

**Figure 3** Whether a neutral allele finds itself on a beneficial background (blue) or disadvantageous background (orange), the direction of the allele frequency changes between times will be the same (as long as the neutral allele is still associated with its background). This creates positive temporal covariance that we can detect in populations, which can tell us that there is heritable fitness variation in the population that is perturbing allele frequency changes.

If we were to track this neutral allele’s frequency trajectory through time in the entire population, we would see it rise between the two first consecutive generations, and then rise again between the next two, as long as it remains associated with a fraction of the father’s beneficial genes. The neutral allele’s frequencies between the second and first generation (mathematically, $\Delta p_1 = p_2 - p_1$), and the third and second generation ($\Delta p_2 = p_3 - p_2$) would both change in the same direction when they’re associated with this good background, creating temporal covariance in the frequency trajectory. This same effect happens if the neutral allele were to instead be associated with a disadvantageous background (see figure above). In contrast, genetic drift cannot create temporal covariance in neutral allele frequency trajectories. Another way to say this is that when different chromosomes have heritable fitness differences, the frequency change of an allele at one time interval can be predictive of the changes at later generations, as long as the allele is still associated with its fitness background, and that background has the same effect on fitness.

In Buffalo and Coop (2019 ), we proposed using temporal genomic data to quantify the amount of temporal covariance in a population, and use it as a means of detecting polygenic selection over short timescales. We also develop a mathematical theory of what determines the strength of temporal covariance, and find it is determined by how much additive genetic variance for fitness there is in a population (a key quantitative genetics parameter), and how strong the neutral allele’s association is with the genetic fitness variation (which is mediated by the level of recombination and the strength of initial association, known as linkage disequilibrium).

Detecting Temporal Covariance in Evolve-and-Resequence Studies

With a basic understanding of how temporal covariance functions as a signal of polygenic selection, let’s look our recent paper, Buffalo and Coop (2020 ). In this study, we applied the methods we developed in our Genetics paper to four evolve-and-resequence studies. Our work relied entirely on the accessibility of the data from these previous studies, and we both are grateful for the authors' support and openness with their data.

The first study we analyzed was from Barghi et al. (2019) , an evolve-and-resequence study in Drosophila simulans. In this experiment fruit flies were evolved in a hot laboratory environment for 60 generations, across ten independent replicates. We re-analyzed this well-designed study using our temporal covariance methods. We found extensive evidence of temporal covariance through time (Figure A below), consistent with the original author’s findings that the populations were adapting to their new warmer environment. Furthermore, these temporal covariances declined through time, just as we expected, since the temporal covariances weaken as the associations between neutral sites and fitness variation decay through time.

Figure 4 (A) Temporal covariance in the Barghi et al. (2019) data set, averaged over all ten replicate populations. Each line depicts the covariances between some initial allele frequency change, and later frequency change through time (which represents the rows of the covariance matrix, right). (B) A lower-bound on the proportion of total variance in allele frequency change due to drift, $G$.

In Buffalo and Coop (2019 ), we proposed that we could use temporal covariances to estimate what proportion of total variation in allele frequency change is caused by linked selection². We called this proportion $G$; if $G = 0$, all the variance in allele frequency change is caused by genetic drift, whereas if $G = 1$, all the allele frequency change is caused by linked selection. We calculated $G$ as it accumulates through the generations (and thus call it $G(t)$ where $t$ represents time), shown in the Figure B above. Because samples in Barghi et al. (2019) were sequenced every ten generations, our $G$ estimate is a lower bound estimate, meaning the actual proportion of variation in allele frequency change due to linked selection is higher than our $G$ estimates. Still, we find that over short timescales, at least 20% of the variation in allele frequency change is due to linked selection. This provides us with the first glimpse of how linked selection determines frequency changes over very short timescales.

Given we find such evidence of linked selection over short timescales, two questions arise: (1) is this really polygenic selection? and (2) where is the signal coming from? First, to investigate whether this was indeed polygenic selection, rather than selection on a few mutations that have large effect, we calculated temporal covariances along windows in the genome. If our genome-wide signal of linked selection was driven by a few regions under strong selection, we should expect to see these regions as outliers. Instead, we see that the whole distribution of windowed covariances is enriched for positive covariances, indicating the signal we’ve detected is spread across the entire genome.

Second, how can we detect a signal of linked polygenic selection when the effect at each site is so weak? Drift and sampling variance introduce considerable noise that can swamp the signal of temporal covariance, as well as create spurious covariances. However, these sources of noise do not share random change in the same direction, whereas temporal covariances do, leading to a signal that can be readily distinguished from random drift.

Convergent Correlations

One common study design of evolve-and-resequence experiments is to evolve multiple indepedent populations under the same (or different) environments, and look for evidence of convergent (or divergent) selection. We hypothesized that we should be able to detect something analogous to temporal covariance across replicate populations exposed to the same selective pressures. This is because replicate populations created from the same founding population will, by chance, share some of the same fitness variation. Neutral alleles associated with the same advantageous genetic backgrounds would then be expected to increase in frequency through time, in both replicate populations. This creates what we call between-replicate covariance, and we can measure this much like we do temporal covariance.

Figure 5 A measure of between-replicate covariance, convergence correlations, calculated on the ten replicates of the Barghi et al. data. Each line represents a row in the correlation matrix pictured on the right, showing how the convergence correlation averages across all pairwise comparisons between replicates changes through time.

The Barghi et al. study evolves ten replicate populations independently, which provided us with a great data set to see if we could detect between-replicate covariances. To measure the extent of between-replicate covariances, we use what we call convergence correlations, which are simply the covariance in allele frequency changes between replicates scaled by the standard deviation in allele frequency change. As with our temporal covariances, we calculate these for all time intervals, and find that they are relatively week and decay quickly through the generations. This tells us that while early on, the selection occurring across replicates is similar, each replicate quickly goes its own way (and this confirms a major finding of the original Barghi et al. study, that different loci across replicates contribute to adaptation).

**Figure 6** The convergence correlations between each pair of replicates (A, B, and C) of the Kelly and Hughes (2019) study. The 95% confidence intervals, estimated by a block bootstrap approach are shown (but look quite small on this y-axis scale).

A benefit of convergence correlations is that unlike temporal covariances, they only require evolve-and-resequence studies with two timepoints and replicate populations. This allowed us to re-analyze two other elegant evolve-and-resequence studies. The first is Kelly and Hughes (2019 ), which evolved three replicate populations of Drosophila simulans to a novel lab environment. Similar to our re-analysis of the Barghi et al. study, we calculated convergent correlations across each pairwise combination of the replicate populations and find that all these convergent correlations are statistically significant and stronger than those we see in the Barghi et al. study³. Furthermore, using an approach like $G$ described above, we found that at least 37% of the total variation in allele frequency change was shared between replicates, which is a pretty sizable proportion considering these lab populations are rather small and strongly affected by genetic drift.

The second study is Longshanks selection experiment of Castro et al. (2019 , see also Marchini et al. 2014 ), where over twenty generations, two independent replicate population lines of mice were selected for longer tibiae lengths relative to body size. The study also has a control line, where mice were bred randomly. Remarkably, over twenty generations of selection, tibiae length increased about five standard deviations.

**Figure 7** The convergence correlations between the two Longshanks selection lines (LS1 and LS2) and the control line (Ctrl). The black lines represent 95% confidence intervals calculated on genome-wide data, and the blue lines represent 95% confidence intervals calculated on the same data excluding the chromosomes where Castro *et al.* found large-effect loci. The signal despite excluding these chromosomes shows the extent to which selection for large tibiae length was polygenic.

Since the Longshanks study has a control line, it provided a powerful test of our convergence correlations: we should expect significant convergence correlations between the two Longshanks selection lines, but not between each Longshanks selection line and the control line, since these two do not share convergence selection pressure. This is precisely what we find (shown above). Furthermore, the original Castro et al. study found two large effect loci, one on chromosome 5 and the other on chromosome 10. In the original paper, they show that while the loci on these chromosomes show a signal of convergent selection, the trait itself is highly polygenic. In sharing our preliminary results with the authors of Castro et al., they wondered the extent to which our convergence correlations were driven just be these large-effect loci. We decided to exclude these large-effect loci by leaving out the entire chromosomes they reside on (in part, because these loci could have far-reaching effects on linked variation). We show the convergence correlations sans these chromosomes above in blue — they are also statistically significant, indicating the signal we are detecting is polygenic.

The presence of the control line also allowed us to do a fun little calculation to partition the total variation in allele frequency changes into drift, shared selection, and unique selection components. Since the mating of mice is random in the control line, the total variance in allele frequency change we see is due only to genetic drift. Any additional variance in allele frequency change we see in the two control lines is then caused by selection (due to exactly the same effect that Robertson 1961 described). Furthermore, we can estimate the fraction of variance shared between the Longshanks selection lines, since this is just the covariance in frequency changes between Longshanks lines. Finally, the remaining part of the variance due to selection is that which is unique to each selection line. We find that at least 32% of the variance in allele frequency change is due to selection, and of this 32%, 17% is due to shared selection pressures and the remaining 14% is due to unique selection pressures or associations unique to a particular replicate population³.

Shifts in Temporal Covariance

**Figure 8** In one environment, where a trait is beneficial if it is larger (blue background), the temporal covariance (gray points) through time is positive. If the direction of selection changes such that small values of the trait are beneficial (yellow background), the temporal covariance becomes negative. Negative observed temporal covariance can tell us about reversals in the direction of selection dynamics. Figure from Buffalo and Coop ([2019](https://www.genetics.org/content/213/3/1007)).

In our Genetics paper, we suggested that fluctuating selection could create negative temporal covariance. The intuition here is that if a neutral allele rises in frequency on a beneficial genetic background, a change in the environment that leads this background to become disadvantageous would cause a decline in frequency. Since in the first generations $\Delta p_t > 0$ and in the later generations $\Delta p_s < 0$, the temporal covariance would be negative (see figure above). We confirmed this hunch with simulations in our Genetics paper, and were curious if we’d see this pattern in real data.

If you look closely at Figure 4 (A) above, you’ll see that at later timepoints, we do observe negative covariances that are statistically significantly different from zero. This is consistent with a reversal in the fitness of certain genetic backgrounds. We wondered to what extent these reversals at later time intervals were common, but obscured since we were average over the entire genome.

One thought we had was that perhaps we could look for such reversals happening in smaller chunks, or windows, of the genome. While this could allow us to detect subtle, local reversals in selection dynamics, it introduces two problems. First, the temporal covariance estimates are incredibly noisy, since we’re averaging over fewer sites. Second, we would need a way to estimate what the distribution of these temporal covariances across windows would look like under just drift, as a null hypothesis. We devised a simple way to do this by randomly permuting the allele frequency changes across each window, and looking at the entire distribution of windowed temporal covariances. The permutation approach essentially breaks the directionality of temporal covariances caused by selection, and gives us an estimate of the distribution of temporal covariances as they would be under just genetic drift, which we could compare to our observed distribution. Below, I show the distributions of the windowed temporal covariances between different time intervals:

**Figure 9** Each figure shows the observed distribution of windowed temporal covariances (orange) and the sign-permuted null distribution (blue). (A\) Windowed temporal covariances twenty generations apart show that these are positive (the right shoulder of the orange distribution, compared to the blue null distribution). (B\) Forty generations apart, we see the shoulder has shifted towards the left side, indicating a reversal in the selection dynamics across the genome.

In Figure 9 (A) above, the temporal covariances are separated by two timepoints (20 generations), and we see an enrichment of positive temporal covariances across the genome, consistent with the findings mentioned above. In Figure 9 (B) above, the temporal covariances are 40 generations apart and we see an enrichment of negative covariances, compared to the null distribution in blue. Our paper and the appendix shows more figures illustrating this point. Seeing the signal of such selection dynamics over short timescales was an unexpected and interesting surprise. Note that the environment in the Barghi et al. study was kept constant; there was not an any outside factor that was intentionally changed that could lead to such a reversal in the direction of selection. This tells us that rather complex selection dynamics could be common over short evolution, and these selection dynamics may go unnoticed in the long-run despite having a considerable impact on genome-wide frequency changes.

Conclusions

Overall, we find that as sizable proportion of allele frequency change, over short timescales, is due to selection (likely through its effects on linked sites). Our approach was overall extremely conservative, so the actual effect could be much greater. As further evolve-and-resequence studies are designed and conducted, I hope to apply and improve these methods to get better estimates of the impact of polygenic selection on genome-wide frequency changes. Overall, I believe our view of evolutionary genetics will be greatly advanced if we continue to try to understand genomic evolution over short timescales and try to reconcile these with population genomic studies using samples from a single contemporary timepoint.

Our paper contains much more detail, and many analyses I’ve omitted here for brevity. I have left out our entire section on the extensive simulations we have conducted using SLiM, which confirm that our measures of temporal covariance, $G$, and convergence correlations work as intended. Graham and I are both quite grateful to our reviewers and our editor for a recommendations and a review process that made our paper much stronger.

Finally, I should point out that this entire project was only possible because the authors of the original studies conducted careful, well-designed experiments, and were open with their data. I am extremely grateful for this openness and I hope our re-analyses bring their excellent papers more readers. Following their lead, I have made my analyses and the intermediate data I produced openly available on Github . Each analysis was conducted in a Jupyter Lab notebook using open source tools, and I have tried to make my analyses as reproducible as possible. Collectively, our knowledge of evolution will grow faster if we all embrace the same open science mindset.

The Problem of Detecting Polygenic Selection from Temporal Data

Thu, 20 Aug 2020 00:00:00 +0000

The last chapter of my dissertation with Graham Coop was recently published in PNAS (pdf , bioRxiv ) last week. In an effort to communicate my research to a broader audience, I have written two blog posts on our work. The first post, below, is meant to introduce the historical context and concepts like linked selection and polygenic adaptation to a non-scientist, and the second post describes our work on temporal covariance as a signature of polygenic linked selection and its application to four evolve-and-reseqeunce data sets.

Natural Selection is Rapid

For nearly seventy years, we have known Charles Darwin was wrong about a key aspect of natural selection: that it was slow acting. In the first edition of The Origin of Species, Darwin wrote of natural selection,

We see nothing of these slow changes in progress, until the hand of time has marked the long lapse of ages, and then so imperfect is our view into long past geological ages, that we only see that the forms of life are now different from what they formerly were.

While Darwin knew artificial selection¹ could rapidly change an organism’s traits, it was only until ecological geneticists E.B. Ford and Bernard Kettlewell showed rapid changes in moth wing coloration, that it was recognized that natural selection could act over very short timescales. Since, evolutionary biologists have shown rapid adaptation is remarkably common in a variety of species including sticklebacks², Anolis lizards³, soapberry bugs⁴, guppies⁵, and field mustard⁶.

Natural Selection versus Drift

The frequency trajectory of the medionigra wing color variant in Panaxia dominula, scarlet tiger moth,
observed in Oxford by Fisher and Ford (1947).

In one classic study, E.B. Ford and R.A. Fisher tracked the frequency of dark wing color variant in a population of scarlet tiger moths around Oxford for nearly a decade. The dark wing variant declined quickly in frequency (see figure above), which they believed was caused by natural selection against this variant. However, to demonstrate that natural selection drove this change, they had to rule out another possible explanation: genetic drift. Genetic drift is another factor that leads the frequency of alleles⁷ in populations to vary through time, caused by random chance. Some individuals in a population may get lucky, and encounter an abundance of resources that allow them to leave more offspring; some may experience random hardships that force them to leave fewer. Unlike with natural selection, the underlying causes of variation in survival and family size due to drift are not genetic, but random; consequently they are not inherited by the next generation. Another major source of randomness is Mendelian segregation: if you carry two different copies of a gene (one you inherited from your father, one you inherited from your mother), the copy you pass to your child is determined essentially by a biological coin flip.

These random changes in allele frequency across families lead to a random behavior of allele frequencies within a population, which could, purely by chance, also lead to the decline of a particular wing coloration through time that Ford observed. To discern natural selection from random drift, Ford collaborated with R.A. Fisher, who developed a statistical method that used population size estimates to determine the strength of genetic drift (genetic drift is the strongest in small populations, where a single family’s reproductive fortunes have a larger proportional impact). Their 1947 paper argued the change in the frequency of the dark wing variant through time was caused by natural selection, not genetic drift; this later lead to a vitriolic debate between Ford and Fisher and Sewall Wright, the leading proponent of genetic drift at the time.

We can study natural selection through its effects on linked sites

Since this early work using temporal data, there was a relative lull as efforts shifted towards detecting selection from single present-day population genetic samples, until the tremendous growth in DNA sequencing technology over the last two decades. Now, researchers can directly observe allele frequency changes through time all across an organism’s genome (millions of sites), rather than at a few sites that affect traits (e.g. darker wing color). This has spurred the further development of statistical methods that can differentiate allele frequency changes caused by selection from those caused by random drift⁸. Simultaneously, new statistical methods were discovering the footprints left by natural selection in our DNA and the DNA of many other species, by the effect selection has on its neighboring sites.

When a new beneficial mutation arises in a population, its comparative advantage (either through increasing survival or in leaving more offspring) causes it to quickly rise in frequency. Alleles reside on our chromosomes, and are linked, or coupled to their neighbors because large contiguous stretches of our chromosomes are inherited together. A process known as recombination does shuffle up chromosomes a bit, since each chromosome we inherit from our father is a patchwork of our paternal grandparent’s chromosomes (and likewise with our maternal chromosomes). Consequently, when beneficial mutations rise in frequency, they drag along neighboring alleles that happen to be lucky enough to reside upon the same chromosome that the beneficial mutation arose (we call this “genetic hitchhiking”⁹ since these alleles hitch a ride along with the beneficial mutation). We can detect these events because they wipe out genetic variation in spots along the genome, since, in essence, everyone derives their ancestry in this region from the individual in which the beneficial mutation first arose.

A selective sweep in malaria, Plasmodium falciparum, conferring drug resistance. Data from Nash et al. (2005), figure produced by Graham Coop in his population genetics notes.

One remarkable finding of the last three decades was that mutations that are disadvantageous, or deleterious (that is, they leave fewer offspring or lower odds of survival) also extinguish genetic variation in a region. This effect is known as background selection. Collectively, the hitchhiking effect and background selection are types of linked selection, and this is a primary focus of Graham and my work. There is a great deal of mathematical theory that predicts the extent to which, and over how long a stretch of chromosome, genetic variation is wiped out by genetic hitchhiking and background selection. From this body of work, we predict regions of high recombination (i.e. an allele is very likely to be randomly shuffled off its chromosome background) reduce linked selection’s effects on genetic variation, while in regions of low recombination (i.e. an allele is very unlikely to be shuffled off its chromosome background) are drastically affected by linked selection. These predictions have been confirmed in numerous studies, perhaps most famously by Begun and Aquadro (1992) in the fruit fly Drosophila melanogaster, where they find a strong correlation between the amount of recombination and the level of pairwise diversity, a common measure of genetic variability.

The correlation between pairwise diversity (a measure of genetic variability) and recombination rate in Drosophila melanogaster, from Begun and Aquadro (1992).

Over the last fifty years, we have come to recognize that linked selection itself introduces a source of randomness, much like genetic drift, into evolution; we call this genetic draft. Since new selected mutations (whether advantageous or injurious) arise on random chromosomes in the population, and since recombination randomly shuffles these chromosomes through the generations, an allele’s frequency may jiggle about through time not only due to genetic drift, but by the fitness of whatever background it happens to find itself on. Our understanding of evolutionary genetics will not be complete until we can differentiate between the randomness of genetic drift from the randomness of linked selection. This open problem is reminiscent of the same debates that Fisher and Ford were having over seventy years ago.

The Problem of Polygenic Selection

The distribution of human heights, a continuous trait, among 143 University of Connecticut students. Image from Schilling et al. (2012).

Over the last decade, modern genomic data sets have advanced the field of quantitative genetics as well. Quantitative genetics studies the nature of continuous traits, such as height, including how selection operates on these traits and how variability for these traits arises in populations. One of the triumphs of the Modern Synthesis¹⁰ was finding that the same Mendelian genetic system that determines discrete traits, like wing coloration, also determines continuous traits like height. The difference is that while a single gene may determine wing coloration, hundreds to thousands of genes each have a small effect on height; such traits are known as polygenic traits. These small differences across numerous genes, as well as environmental factors, add up to determine one’s height; across a population, the genetic differences between individuals lead to a smooth distribution of traits that looks normally distributed (i.e. having a bell curve shape, such as the distribution of heights pictured above).

Selection also acts on continuous traits, which has enabled humans to continually increase crop yields, breed cows that produce more milk, and so forth. Like population genetics, a rich body of mathematical theory can predict the response to selection and many other aspects of quantitative traits. However, unlike population genetics theory, the mathematical theory of quantitative genetics does not concern itself with the allele frequency changes at each of the sites that determine a trait’s value, but rather takes a macroscopic approach that focuses on a trait’s mean and variance, much like the macroscopic view of the ideal gas law in physics. Bridging this macroscopic quantitative genetics view with the microscopic population genetics view of individual genes has proved to be an arduous task for many reasons.

One key difficulty is that when a polygenic trait like height is selected for, the effects of selection are distributed across all the hundreds or thousands of sites that contribute to the trait’s value. Consequently, the effect of selection at any one site is minuscule, and its frequency trajectory through time will not look like the rapid change in frequency that Fisher and Ford saw with wing coloration in scarlet tiger moths. Worse, such small changes in allele frequency change can easily be mistaken for the random changes caused by drift.

In sum, evolutionary geneticists are left with an interesting quandary. We can readily observe rapid adaptation in natural populations, as traits change through time to better suit a new environment. Yet, despite the present abundance of population genomic data, it can be difficult to detect selection on polygenic traits from temporal data at the DNA level, since the allele frequency changes are minor and hard to differentiate from random genetic drift. Furthermore, since these allele frequency changes are seemingly indistinguishable from genetic drift, even with temporal data we are not able to discern what fraction of allele frequency changes are due to drift, and what fraction are due to selection. These are the problems my work with Graham Coop is working to address.

Continue to Part II

Understanding Snakemake

Wed, 04 Mar 2020 00:00:00 +0000

Heraldic snake from Flickr (CC Licensed).

Each day, data scientists, computational biologists, astronomers, and other folks that spend far too much time in front of a computer screen spend hours doing somewhat horrible, monotonous tasks. Scientific programming, when done right, is supposed to prevent us from doing these monotonous tasks, and this is certainly true when we compare what we do today to what the tireless programmers and human computers of the 1950s did: inverting matrices by hand, writing code to calculate the t-statistic and corresponding p-value, and so forth. All of these monotonous tasks, thankfully, are implemented now in modern libraries like BLAS/LAPACK, GNU Scientific Library, numpy, R, eigen, etc. However, the problem has just shifted: today’s monotonous tasks are tidying messy data, applying a linear model to tens of thousands of data sets, or assessing prediction accuracy across statistical model parameters using cross validation.

Some of these monotonous tasks have been made easier by very clever abstractions. Consider R’s tidyverse, which has simplified numerous monotonous tasks most R users do by realizing these tasks fit a pattern: import data, tidy that data, explore the data, and communicate the findings (see Hadley Wickham’s book R for Data Science ). Similarly, scikit-learn has slick classes to do the tedious task of model selection . Additionally, data projects usually involve lots of repetitive file system work: downloading datasets, running command line tools to pre-process raw data, running simulation software, etc. These tasks are often repetitive because, throughout the course of a project, you’ll likely need to re-run the same steps multiple times. On page 9 of my book Bioinformatics Data Skills I explain why:

You will almost certainly have to re-run an analysis more than once, possibly with new or changed data. This happens frequently because you’ll find a bug, a collaborator will add or update a file, or you’ll want to try something new upstream of a step. In all cases, downstream analyses depend on these earlier results, meaning all steps of an analysis need to be re-run.

What abstraction and accompanying software tools allows us to avoid these humdrum repetitive tasks we do in the Unix shell? Make and Snakemake.

Make

I think the best way to learn Snakemake (our ultimate goal) is to first get a rough sense of how its predecessor, Make, works. I still use Make for simple tasks, and you’ve already likely used it to compile software at some point. Make is a software tool and language that’s been around since 1976, and is still widely used. Originally it was designed as a way to automate building software, but the central problem of compiling software is quite like the problem described above: some input file will change, and all downstream files that depends on this file needs to be updated by re-running code. Compiling software is relatively time consuming, so Make’s designers took advantage of a simple idea: if we declare what files depend on other files, we only need to run the steps downstream of the files that have changed. In computer science lingo, we write out the code compiling steps as a directed acyclic graph, and only the paths connected to a changed file need to be re-run. This will become clear with some examples.

First, Make (the software) looks for file named Makefile in a directory when we run the make command¹. In a Makefile, we specify the rules describing the steps to run to turn input files into output files. Specifically, a Makefile consists of:

¹ For clarity, I’ll try to be consistent in how I stylize various words when describing Make and Snakemake. Make is the name of the software and language, Makefiles are the files full of code describing what to do, make is Make’s command line tool, and a file named Makefile is what the make tool looks for in a directory when its run.

A target, the thing to build with this rule. This is the output file.
A single or list of dependencies files. These are the files needed by the rule to create the target.
Commands, or the list of Unix commands needed to convert the dependencies into the target.

The format of a rule is:

file_target.txt: dependency_1.txt dependency_2.txt
	unix_command dependency_1.txt dependency_2.txt > file_target.txt

Note that the command line must begin with a tab character, not spaces. Make’s a bit of a grumpy about this. Still, Make is quite clever, in that it will only re-run each rule if (1) the input files changed, or (2) the target does not exist and needs to be generated.

I still use Make for the simplest redundant file tasks: usually downloading data from the web, and doing some minimal pre-processing. For example, in a recent project I wanted to download the genome of Drosophila melanogaster and create a file containing the lengths of all the sequences using bioawk . Here’s what a Makefile running these steps looks like²:

all: Dmel_BDGP6.28_seqlens.tsv

Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz: 
  wget ftp://ftp.ensembl.org/pub/release-99/fasta/drosophila_melanogaster/dna/Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz

Dmel_BDGP6.28_seqlens.tsv: Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz
  # note we need the double dollar signs here, since the $ indicates a variable in Make
  bioawk -c fastx '{print $$name "\t" length($$seq)}' Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz > Dmel_BDGP6.28_seqlens.tsv

You can give this a try for yourself by copying and pasting this into a file called Makefile in an empty directory (or downloading it from Github ) and typing:

$ make all

This looks for a file named Makefile, and runs the all target. Note that Make is a declarative language , meaning it doesn’t execute code from top to bottom like a normal program’s control flow. Instead you declare what needs to be done, and it executes things in the right order. Stepping through the Makefile code, there are three rules, which I’ll explain in the order Make works through them:

all, which is the primary target. It’s the starting place; typing make all tells Make to create all’s dependencies. In this case, there’s only one dependency: Dmel_BDGP6.28_seqlens.tsv.
Now, make is looking for all’s dependency, Dmel_BDGP6.28_seqlens.tsv. Since this file does not exist, Make looks for a rule to create this target. It finds this rule, but this rule requires the file Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz. This file isn’t in this directory yet either, so Make goes looking for a rule to build that.
The second rule (after all) declares how to generate Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz. This target requires no dependencies, so Make proceeds straight into executing the rule: use wget to download a file from the web.
One this file downloads, the dependency for the Dmel_BDGP6.28_seqlens.tsv file is available. Make, working backwards, runs this rule now, calling bioawk to summarize the sequence lengths of this genome. This generates the file Dmel_BDGP6.28_seqlens.tsv.
Finally, make has satisfied dependency of all: Dmel_BDGP6.28_seqlens.tsv. Since it’s got everything it needs, it quits.

The real magic is what happens if we delete Dmel_BDGP6.28_seqlens.tsv, or change Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz. Unlike a bash script, which will re-run everything start to finish, Make will only re-run what it needs to build files depending on the changed files:

$ rm Dmel_BDGP6.28_seqlens.tsv
$ make all  # since the genome hasn't been changed or deleted, only the last rule is run!
bioawk -c fastx '{print $name "\t" length($seq)}' Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz > Dmel_BDGP6.28_seqlens.tsv

We can emulate changing an input file in this example by using touch to change the timestamp of the Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz file:

$ touch  Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz
$ make all  # runs all downstream steps
bioawk -c fastx '{print $name "\t" length($seq)}' Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz > Dmel_BDGP6.28_seqlens.tsv

Make gets a lot, lot more complicated than this simple example. The language is rich, but its a rather tedious language for all about the simplest tasks. As you use Make more, I’d recommend learning more about its automatic variables , which allow us to avoid redundantly typing out target and dependency filenames. The two I use most are $@, which is a placeholder for the filename of the target, and $<, the name of the first prerequisite. This would simplify our earlier Makefile like so³:

all: Dmel_BDGP6.28_seqlens.tsv

Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz: 
  wget ftp://ftp.ensembl.org/pub/release-99/fasta/drosophila_melanogaster/dna/Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz

Dmel_BDGP6.28_seqlens.tsv: Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz
  bioawk -c fastx '{print $$name "\t" length($$seq)}' $< > $@

Finally, it’s worth mentioning that Make is trivially parallelizable. Since Makefiles describe the chain of rules needed to create a file, indepenent chains can be run across different cores simulultaneously. The example above is too simple to run steps in parallel, but if did have independent chains that could be run independently, we’d do this with:

$ make -j 4  # run the Makefile on 4 cores

Make is definitely finicky and old; after all, it first emerged in 1974. I still use Make for the simplest of data downloading and pre-processing tasks, much like Mike Bostock describes . While there was a time when I would dig deep into the Make documentation, using Make’s functions to write complicated Makefiles to process hoards of data, now I prefer Snakemake.

Snakemake

Snakemake is a new, Python-based build automation software program. Unlike Make, which was intended to be used to automate compiling software, Snakemake’s explicit intention is to automate command line data processing tasks, such as those common in bioinformatics. You can install Snakemake with Conda (instructions here ). Much like Make, running the command line program snakemake looks for a Snakefile, named Snakefile in the directory. And much like Make, the format of the Snakefile has rules defined by targets (known in Snakemake as outputs), dependencies (Snakemake calls these inputs), and rules (and a lot more is possible here with Snakemake, as we’ll see). Let’s translate our earlier Makefile to a Snakefile⁴:

rule all:
  input:
    "Dmel_BDGP6.28_seqlens.tsv"

rule genome:
  output: 
    "Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz"
  shell: 
	  "wget ftp://ftp.ensembl.org/pub/release-99/fasta/drosophila_melanogaster/dna/Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz"

rule seqlens:
  input:
    "Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz"
  output:
    "Dmel_BDGP6.28_seqlens.tsv"
  shell:
	  """bioawk -c fastx '{{print $name "\t" length($seq)}}' {input} > {output}"""

The key changes are:

Slightly different rule format, and all rules are named (e.g. all, genome, etc.).
Filenames and shell commands must be quoted (note that Python’s triple quotes can be used to avoid escaping quotes in cases where single and double quotes are used in the rule).
Rules running shell commands are specified in shell blocks. Snakemake also supports running Python in run blocks.
Rather than awkward special variables like Make’s $@ and $<, Snakemake uses Python’s formatted strings (i.e. the braces in the last line) and clear names like {input} and {output}. However, since braces are now special in Snakemake, we need to escape them when using them in our bioawk line; a literal brace is specified by using two of them, e.g. {{ and }} (this is analogous to how we had to escape the $ in Make by using $$!).

This is then executed much like a Makefile. Before executing it though, let’s do a dry run with snakemake --dryrun or snakemake -n. This doesn’t execute any steps, it just shows what Snakemake would do if run⁵:

⁵ Make does have a dry run option too, by the way; it’s make --dry-run or make -n.

$ snakemake --dryrun
Building DAG of jobs...
Job counts:
	count	jobs
	1	all
	1	genome
	1	seqlens
	3

[Thu Mar  5 11:37:28 2020]
rule genome:
    output: Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz
    jobid: 2


[Thu Mar  5 11:37:28 2020]
rule seqlens:
    input: Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz
    output: Dmel_BDGP6.28_seqlens.tsv
    jobid: 1


[Thu Mar  5 11:37:28 2020]
localrule all:
    input: Dmel_BDGP6.28_seqlens.tsv
    jobid: 0

Job counts:
	count	jobs
	1	all
	1	genome
	1	seqlens
	3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

Now, let’s execute these steps – on the command line, enter:

$ snakemake

Unlike Make, Snakemake has a really nice progress reporting (I’ve omitted this output above for brevity).

Using Expand to Build up all Filesnames and Parameter Combinations

The real strength of Snakemake is how easy it makes applying rules across multiple files that share a similar filename structure (this is why it is so important to have a consistent file name scheme!). I’ll demonstrate this incrementally with a few Snakefiles, which because we can program in Python with Snakemake, allow us to see what’s happening with print() statements. First consider this Snakefile⁶:

chrom_filename = "Drosophila_melanogaster.BDGP6.28.dna.chromosome.{chrom}.fa.gz"

chroms = ['2L', '2R', '3L', '3R', 'X', '4']

chrom_fa_files = expand(chrom_filename, chrom=chroms)

print(chrom_fa_files)

Now, rather than downloading the entire Drosophila melanogaster genome, we’re going to download some individual chromosome sequences⁷. We exploit these well-named files, and build all the chromosome sequence filenames that need to be downloaded programmatically with Snakemake’s powerful expand() function. expand() builds a list of strings by replacing the string {chrom} in chrom_filename with each of the chromosome names in the chroms list. We use print() on the last line to look at the resulting list of filenames.

While above we used expand() to build up chrom_fa_files, populating it with values from just the chroms list, it works with more than one input list too, and generates all combinations (a Cartesian Product ) of the input values. This makes expand() exceedingly powerful because it can be used to build up all possible parameter combinations for a series of simulations. Consider⁸:

import numpy as np
Ns = [100]
selcoefs = 10**np.linspace(-3, -1, 3)
rbps = 10**np.linspace(-8, -7, 2)
nreps = np.arange(20)

sim_results_pattern = "sim_{N}N_{selcoef}s_{rbp}rbp_{rep}rep.tsv"

sim_results = expand(sim_results_pattern,
                     N=Ns, selcoef=selcoefs,
                                          rbp=rbps, rep=nreps)

print(sim_results)

Running this, we see we have a list of all results files for all parameter combinations:

$ snakemake
['sim_100N_0.001s_1e-08rbp_0rep.tsv', 'sim_100N_0.001s_1e-08rbp_1rep.tsv',
 'sim_100N_0.001s_1e-08rbp_2rep.tsv', 'sim_100N_0.001s_1e-08rbp_3rep.tsv',
 'sim_100N_0.001s_1e-08rbp_4rep.tsv', 'sim_100N_0.001s_1e-08rbp_5rep.tsv',
  ... 
 'sim_100N_0.1s_1e-07rbp_14rep.tsv', 'sim_100N_0.1s_1e-07rbp_15rep.tsv',
 'sim_100N_0.1s_1e-07rbp_16rep.tsv', 'sim_100N_0.1s_1e-07rbp_17rep.tsv',
 'sim_100N_0.1s_1e-07rbp_18rep.tsv', 'sim_100N_0.1s_1e-07rbp_19rep.tsv']
Building DAG of jobs...
Nothing to be done.
Complete log: /Users/vinceb/projects/snakemake-tutorial/example-05/.snakemake/log/2020-03-05T190027.307441.snakemake.log

There’s nothing special about by my file name scheme here, though it is one that I often use. The filenames could be parsed by downstream programs so the parameters are known, but I prefer usually to have the simulation software write a metadata string at the top of the results files (e.g. as a comment line a TSV/CSV beginning with #).

Using Wildcards

In the previous section, we automatically created a bunch of filenames of simulation results we want after running all our simulations, representing all combinations of parameters. Now, we need to write a rule that describes how to actually run all these simulations and pass the appropriate parameters to the command line tool responsible for running the simulations. What’s elegant about Snakemake is that since each file is the result of running a simulation once with particular parameters, we can write one special general rule that describes how to generate all the simulation results. The trick to do this is to use wildcards.

Understanding wildcards was, for me, the hardest part of understanding Snakemake. They’re just magic enough to be confusing, but also really useful. The best way to grok wildcards is to understand that they match parts of a rule’s output file. I think it’s easier to explain this through a simple example, after which we’ll continue the simulation example described above.

Here’s a simple example of wildcards⁹:

results = "file_{sample}.txt"

all_results = expand(results, sample = [1, 2, 3])

rule all:
  input:
    all_results

rule sims:
  input:
  output:
    "file_{sample_name}.txt"
  run:
    with open(output[0], 'w') as f:
      f.write(f"the sample name is {wildcards.sample_name}")

Note that rather than using the shell block, we’re using a run block which is just pure Python code – this is a beautiful feature of Snakemake. In this block, the variable output is automatically set by Python, and is a list of all files in output. However, since we’re using wildcards, Snakemake is passing in the files from all_results one at a time, so this list contains just a single file. We grab the only file in the list, output[0] and open it for writing. In that file, we write the contents of {wildcards.sample_name}, which Snakemake also automatically sets for each output filename.

If this is still unclear, it’s important to remember that Snakemake is working backwards. The all target is first run, and Snakemake looks for this rule’s inputs: the list of files in all_results. Then, since these files don’t exist, Snakemake looks for a rule to generate them. The sims rule’s output matches the filenames needed – "file_{sample_name}.txt" is treated like "file_*.txt" would be by Unix. The difference is that the matching section is assigned to wildcards.sample_name and can be used by the rule’s shell or run block.

With this simple example hopefully making wildcards clearer, let’s continue our simulation example. For this example, I use the population genetics forward simulation software SLiM from the Messer Lab , but the basic idea extends broadly to bioinformatics and data science tasks. I’m simulating evolution of a stretch of chromosome, where selected mutations pop in the population, but only in a small region (emulating a gene) in the middle of the chromosome. The details of the simulation are on Github , and I use the Snakemake file to try different parameters, in this case selection coefficients and the level of recombination. I also use Snakemake to generate a lot of independent replicate results. The Snakemake file¹⁰ looks like:

import numpy as np
Ns = [100]
selcoefs = 10**np.linspace(-3, -1, 3)
rbps = 10**np.linspace(-8, -7, 2)
nreps = np.arange(40)

sim_results_pattern = "sim_{N}N_{selcoef}s_{rbp}rbp_{rep}rep.tsv"

sim_results = expand(sim_results_pattern, 
                     N=Ns, selcoef=selcoefs, 
                     rbp=rbps, rep=nreps)

rule all:
  input:
    sim_results

rule sims:
  input:
  output:
    sim_results_pattern
  shell:
    # split across two lines, to make this easier to fit on screen:
    ("slim -d s={wildcards.selcoef} -d rbp={wildcards.rbp} " + 
     "-d N={wildcards.N} -d rep={wildcards.rep} sim.slim")

Here, Snakemake captures the wildcards and passes them directly into the command line call to SLiM. We can run this across four cores with:

$ snakemake --cores 4

This creates a lot of simulation results. Processing these files isn’t within the scope of this tutorial, but you can see the R script I used to do so on Github . Additionally, I’ve included another Snakemake file for this example showing how Snakemake can also be used to run scripts to make figures, using the simulation results generated by another part of Snakemake. It’s Snakes all the way down! Our final result is a figure:

SLiM simulation results showing the effects of recurrent sweeps. The results are noisy because only 40 simulations were averaged over, and the population size is rather small (N = 100). Still, one sees the effect of increasing recombination (weak sweep effect) and changing the selection coefficient.

Future

I hope this has convinced you that Snakemake is a powerful tool that should be in your computational toolbox, and it is clearer how some of the more powerful features of Snakemake (expand() and wildcards) work. For what it’s worth I still use Make for small tasks, and will continue to do so.

While I use Snakemake a fair amount now, I expect this to continue to increase. Why? Once you become acquainted with Snakemake, you start to see increasingly many areas in a project you can use it in (e.g. generating figures, parsing collating raw data, running unit tests). Snakemake becomes fun to use because it prevents the monotony of running the same steps repeatedly in a project. I think too many years coding up analyses have made me realize that 70% of the computational work of an analysis or project is the same as every other project. This shared component of computational work is not intellectually stimulating, and if some tool or library can serve as a higher-level abstraction that makes these repetitive tasks easier, it frees up time to work on intellectually stimulating parts of a project – the stuff I enjoy more. Hopefully Snakemake will help you do more monotonous data tasks in less time with less effort too.

A Genealogical Look at Shared Ancestry on the X Chromosome

Sun, 03 Apr 2016 00:00:00 +0000

An example of a present-day female's X material being broken up across her X ancestors in her X genealogy back through the generations.

My article with Steve Mount and Graham Coop, A Genealogical Look at Shared Ancestry on the X Chromosome has been published in Genetics. In the spirit of both outreach and continuing Graham’s terrific series of blog posts¹ on genetic genealogy, I’m writing about our paper on X chromosome genealogy and recent ancestry. Before diving into the details of X chromosome ancestry work, I’ll review the concepts of genealogies and ancestry. Then, in the next section we’ll look at how one’s genetic ancestors —the subset of ancestors that you share genetic material with— vary back through the generations. With these concepts reviewed, we’ll look at the genealogy that includes all of our X ancestors, which due to the special inheritance pattern of the X chromosome is only a subset of one’s genealogy. The embedded X genealogy has some properties that impact how segments of DNA are shared between individuals with recent common ancestry (e.g. 6^th degree cousins), which we look at through a simple probability model. Finally, we’ll look at what we can learn about the relationships of individuals that share sections of their X chromosome due to sharing a recent common ancestor.

Genealogies

Each human, as a sexually reproducing species with two sexes, has two parents. You have two parents, four grandparents, eight great-grandparents, 16 great-great grandparents, and $k$ generations back have $2^k$ great^$(k-2)$ grandparents, and in general $2^k$ ancestors $k$ generations back. An example genealogy back five generations is shown below:

Of course, these genealogical ancestors are not necessarily all distinct individuals; as you go further back through the generations, some of these $2^k$ individuals aren’t unique—they’re the same person. Intuitively, this occurs when one’s two parents are actually related some number of generations back. For example, one’s two parents could be 9^th degree cousins—e.g. if we assume a generation time of about 30 years, this means these parents shared an ancestor around 270 years ago. This phenomenon is known as pedigree collapse, and it’s the same thing as inbreeding. The further back through the generations you go back, pedigree collapse must happen—it’s exceedingly unlikely that 20 generations ago, your 1,048,576 ancestors are all distinct.² While pedigree collapse definitely occurs, throughout the rest of this blog post (and in our paper) we ignore it, as we model ancestry that’s recent enough where pedigree collapse isn’t a large problem.

Genetic Ancestry

Since each of us have two parents, we receive ½ of our autosomal (i.e. not including the sex chromosomes) genetic material from each parent. We share ½ of our genome with our mother, and ½ with our father. Since your mother shares ½ her genetic material with her two parents, you share ¼ of your genetic material with each grandparent. In general, on average you’ll share ½^k of your genome with an ancestor $k$ generations in the past. Since the number of crossovers per chromosome is limited, close relatives are likely to share large contiguous segments of their genetic material; a beautiful visualization of this is Morgan’s 1916 illustration of crossing over:

When we look at how much DNA two relatives share, we see it occurs in large blocks like the black and white segments above. For example, using 23andme ’s Ancestry Tools I can see how much DNA my grandmother and I share—around 14 Morgans spread across 21 long segments. Essentially, the fact that on average only one crossover occurs per chromosome³ per generation limits how much the genome is broken up through the generations. While on average ¼ of my DNA should be identical to my grandmother’s DNA (we say such genetic material is identical by descent) there’s variance around this ¼ because the genome is of finite length and recombination is limited. In other words, the fraction of my genome that derives from my grandmother isn’t like randomly sampling 6.6 billion marbles independently (the number of basepairs in a diploid human genome), a quarter of which are colored red (i.e. come from my grandmother) and the rest white (i.e. come from my other ancestors). Rather, a more appropriate model is that these marbles are connected by string that is cut and reattached (much like Morgan envisioned in his illustration)—leading recent ancestry to be blocky and segmented.

Currently, there are computational methods (e.g. Browning and Browning, 2011) that take polymorphism datasets and using probabilistic models, identify large identical by descent (IBD) regions shared between individuals—it’s programs like these that services like 23andme use to infer how far back your relatives share ancestry with you. So if we wish to take genomic datasets and understand the large shared segments between relatives due to their shared ancestry⁴ we need a more appropriate mathematical model than the simple model of sampling marbles. Numerous probabilists and statistical geneticists have tackled this using probability theory and stochastic processes (Donnelly 1983; Huff et al., 2011; Thomas et al., 1994). Some of the mathematical details are rather complex (leading to fun conceptualizations like “a random walk on a hypercube”), but the underlying model can be simplified considerably.

Each generation, we can imagine that a random number $B$ of crossovers breaks the 22 human autosome, creating $B+22$ segments. As in Morgan’s original illustration (above), this leads to complementary gametes, with alternating paternal and maternal segments (the black and white segments in the rightmost figure). Mathematically, tracking these alternate segments is a bit tricky, so we can approximate the process by imaging that each of the segments is passed on to the next generation with probability ½—a flip of a fair coin. Since we don’t actually know how many breakpoints have occurred, we model them as a random process. In our case, we use the Poisson distribution⁵ to assign a probability to the event that some number of breakpoints $B=b$ occurs. This idea of using the Poisson distribution to model recombination has a long history in genetics, going back to Haldane (1919). If we then imagine that this same process across all of the $k$ individuals that connect you and one of your ancestors in the k^th generation, the total number of breakpoints is a Poisson distributed, but with the rate is $k$ times faster. Then, for a segment to survive to be passed from your ancestor in the k^th generation to you, it must survive k independent coin flips—an event that occurs with probability ½^k. By a nice property of Poisson processes known as Poisson thinning, this coin-flipping process can be incorporated directly into the Poisson process by changing it’s rate. Then, the expected number of segments $N$ shared between you and your ancestor in the k^th generation is:

\[\mathbb{E}[N] = \frac{1}{2^k}(22 + 33k)\]

where 33 is the total genetic length of the human autosomes in Morgans, a unit defined as the average number of recombinations that occur (and is named after the Morgan that created the figure above). The formula above can give us a good intuition about what’s going on—the number of segments created by recombination grows linearly with how far back we go ($22 + 33k$), but the survival probability decreases exponentially (½^k). Using the Poisson distribution⁶, we can do more than just find an expression for the average number of segments you share with an ancestor, like calculate probabilities of sharing zero segments (such that your genealogical ancestor is not a genetic ancestor) and calculate the distribution of segment lengths. Additionally, these models can be easily extended to handle the segments shared between cousins.

What’s fascinating about this is that your may not share genetic material with your genealogical ancestors. If you play around with the equation above with different values of $k$, you’ll see around $k=9$ that you’re expected to share less than one segment with your ancestors 9 generations back. We can visualize this using an arc diagram, which depicts a present-day individual in the center as the white half-circle, your two parents, four grandparents, and so forth:

An arc diagram of one's genealogical ancestors and their genetic contributions to the present-day individual. Female ancestors are colored red, and male ancestors are colored blue. This visualization uses simulated genetic ancestry back through the generations, and the opacity of the red or blue arcs grows fainter with the less genetic material shared between that ancestor and you. Completely white arcs are genealogical ancestors that contribute zero genetic material to you. Hover over an ancestor to highlight it and find how much genetic material it has contributed to the present-day individual.

We see that one’s genetic ancestors don’t grow as rapidly one’s genealogical ancestors. There’s a lot more to say about this; see Graham’s terrific blog post on this topic for more information.

X Genealogies

In our paper, we were curious how these processes would play out on the X chromosome. The human genome contains 22 autosome pairs and one sex chromosome pair, give us 23 pairs (i.e. the 23 from 23andme), plus one mitochondrial genome. However, unlike the autosomes, the X chromosome undergoes a special inheritance pattern. Males have only one X chromosome, and a Y chromosome. In contrast, females have two X chromosomes. Each generation, individuals pass a haploid set of chromosomes to their offspring—meaning they take the 23 pairs and pass a combination of each pair. Since males have two different sex chromosomes (the X and the Y), these two different chromosomes don’t recombine like the autosomes (except for over a small region called the pseudo-autosomal region). Instead, the male either passes his X to a daughter or a Y to a son. Females, having two X chromosomes, do pass a recombined X chromosome to their son or daughter. Since the X can only recombine over its entire length in females, we call these female meioses recombinational meioses. Note that with the autosomes, every meiosis is a recombinational meiosis.

What’s fascinating is that this different inheritance pattern leads the X chromosome to have a different genealogy than the one’s biparental genealogy. Since males don’t pass X chromosomes to their sons, one’s X genealogy only includes a subset of one’s total ancestors, and is embedded inside of one’s total genealogy. Below is a genealogy for a present-day female, with her X genealogy shaded in:

Note the number of X ancestors of a present-day female has back through the generations, 2, 3, 5, 8, etc. This sequence is the famous Fibonacci sequence offset by two. Thus, a present-female’s number of X ancestors is the $k+2$ Fibonacci number, $\mathcal{F}_{k+2}$ (if the present-day individual is a male, we offset this by one). This sequence crops up throughout nature and mathematics .

Models for X chromosome recent genetic ancestry

Another feature of X genealogies is that unlike the autosomes, where chromosomes undergo recombination every generation and in every ancestor, the number of X recombinational meiosis vary by lineage. This is because the number of females that occur in a lineage to an X ancestor in the 5^th generation vary depending on the lineage. In the leftmost lineage of the genealogy above, a female occurs each generation. In contrast, the rightmost X lineage (with all shaded individuals) alternates between male and female ancestors. Since the X chromosome only undergoes recombination over its entire length in females, the specific lineage to an X ancestor impacts how quickly genetic relatedness breaks down. In our paper, we sought to characterize this lineage-specific rate and see how it affects genetic relatedness.

Our models are similar to the autosomal models described earlier, except given that we don’t know the particular lineage to an X ancestor, we need to average over the number of possible recombinational meiosis that could occur. We found that the number of lineages to an X ancestor $k$ generations back with $r$ recombinational meioses is:

\[{ r + 1 \choose k-r}\]

We can intuitively understand this by looking at an X genealogy; X genealogies enumerate every possible way to arrange males and females such that no two males are adjacent (since fathers don’t pass an X to their sons). Thus, the number of lineages $k$ generations in the past with with $r$ females can be thought of as the number of ways of ordering $r$ red balls and $k-r$ white balls such that no to white balls are adjacent. The number of ways of ordering red and balls this way is given by the binomial coefficient above.

Since one has $\mathcal{F}_{k+2}$ X ancestors $k$ generations back, the probability of $r$ recombinational meioses occurring is:

\[P_R(R=r) = \frac{{ r + 1 \choose k-r}}{\mathcal{F}_{k+2}}\]

Averaging over this number of recombinational meioses gives us a model for the number and length of segments shared identically by descent on the X. It turns out the Poisson thinning approximation described earlier doesn’t work as well as another model we call the Poisson-Binomial model. I won’t cover the detailed derivation here (see the paper if you’re interested), but we find the distribution of X segment number to be well approximated by:

\[P(N=n \;|\; k, \nu) = \sum_{r=\lfloor k/2 \rfloor}^k \sum_{b=0}^\infty \text{Bin}(N=n \;|\; l=b+1, p=1/2^r) \; \text{Pois}(B=b \;|\; \lambda=\nu r) \; \frac{{r+1 \choose k-r}}{\mathcal{F}_{k+2}} \]

As with the autosomes, it’s possible one’s X genealogical ancestors don’t contribute X genetic material to their present-day descendent. For example, here is a simulated X genealogy with opacity of an ancestor indicating that ancestor’s genetic contribution to the present-day individual:

An X genealogy depicted as an arc diagram. Red ancestors are females, blue are males. The opacity indicates the genetic contribution the present-day individual. White ancestors are those that make no genetic contribution to the present-day individual. Gray arcs are genealogical ancestors that are not X ancestors. Hover over an ancestor to highlight it and find how much X genetic material it has contributed to the present-day female.

To get a sense of how one’s X genealogical ancestry grows back in time, we’ve plotted it below (Figure A) compared to one’s autosomal ancestry, and the growth of both one’s genetic X and autosomal ancestry back through the generations. Using probability models we work through in the next section, we also show (Figure B) the probability of sharing some autosomal genetic ancestry (P(N_auto > 0)) and X genetic ancestry (P(N_X

0), conditional and unconditional on both being an X genealogical ancestor (“X ancestor” and “ancestor”, respectively).

Similarly, we extend these models to model the number of X chromosome segments shared between half- and full-cousins and explore other properties of X cousins. These models get a bit tricky mathematically, as the sex of the cousins’ shared ancestor impacts the number of segments shared between cousins, so we incorporate the probability of the sex of the shared ancestor in our models (see Section 3 of our paper for more details).

What recent ancestry on the X can tell us

Using genetic data to infer relationships between individuals is an important topic—it’s used by services like 23andme for ancestry finding, in forensics in assessing DNA-based evidence, and in anthropology and ancient DNA to learn about the familial relationships among individuals. We were curious what X chromosome segments shared between cousins could tell us about their relationship. We found that X chromosome segments can be quite informative about which of their ancestors they share. This information occurs through two avenues: (1) sharing IBD segments on the X immediately reduces the potential genealogical ancestors two individuals share, since one’s X ancestors are only a fraction of their possible genealogical ancestors, and (2) the varying number of females in an X genealogy across lineages combined with the fact that recombinational meioses only occur in females to some extent leave a lineage-specific signature of ancestry. We’ll talk more about this second point in this section.

The X chromosome is relatively short (compared to the autosomes), leading ancestry signals to decay relatively rapidly. Thus, inferring how far back cousins share an ancestor is best accomplished through looking at segments shared on the autosomes rather than X chromosome, and many methods are available for this purpose (Durand et al., 2014; Henn et al., 2012; Huff et al., 2011). We condition on knowing how many generations back these half-cousins share a common ancestor using this autosomal signal. Then, we use Bayes theorem to invert $P(N=n| R)$ to learn the posterior $P(R | N=n)$, where $R$ is the number of recombinational meioses (and thus number of females) between two half-cousins and $N$ is the observed number of X segments shared between the cousins. These posterior distributions are:

For example, the top right panel shows the posterior distributions for the number of females in the lineage connecting two 3^rd degree half-cousins. Each line represents the posterior distribution for a specific number of observed segments shared between these two half-cousins. If these two 3^rd degree cousins share six segments identically by descent, our models say that a lineage with three females is the most likely genealogical configuration. This information is interesting, as these genealogical details cannot be inferred with the autosomal data alone.

As genomic data sets increase, so will the probability of sampling individuals that share recent ancestry. With large data sets (e.g. 23andme’s users), there’s potential for recent ancestry on the X to shed some light on the genealogical relationships connecting us all.

Using Rcpp and C++ to Count Genotypes

Sat, 05 Dec 2015 00:00:00 +0000

I had a matrix (88662 loci x 2060 genotypes) of maize chromosome 1 genotypes, encoded as 0, 1, 2 (e.g. the number of alternate alleles). I needed genotype counts per row, which at first glance is quite easy to solve: just use apply and table:

counts <- apply(chr1g, 1, table)

What’s the problem with this approach? First, technical: if a row only has genotypes 0 and 2, we don’t get counts for 1, which makes merging into a matrix later on a total nightmare (evident because apply is smart enough to return a list). Second, it ignores NA, which is not good. We can fix this with exclude=NULL:

counts <- apply(chr1g, 1, function(x) table(x, exclude=NULL))

This doesn’t solve our first problem, and it’s a bit slower. apply(chr1g, 1, table) took:

  user  system elapsed
84.638   3.110  87.865

Where as apply(chr1g, 1, function(x) table(x, exclude=NULL)) takes:

   user  system elapsed
108.718   5.708 114.609

(and no, passing exclude=NULL to directly to apply doesn’t make it faster). The way around the first technical issue is to use factors. as.factor removes dimensions, so we’re out of luck converting the whole matrix at once (plus, this would require a copy in memory and these objects are moderately large). So, we could so something like:

count <- apply(chr1g, 1, function(x) table(factor(x, levels=c(0, 1, 2, NA)), exclude=NULL))

This works well, but is also not too fast (chromosome 1 is 15% of our dataset):

  user  system elapsed
95.918   5.746 101.861

Rcpp stands out as a simple solution here: this is very easy to code up (it took me literally five minutes). Looking at the timing first:

  user  system elapsed
 8.427   1.573  10.027

We see we have a clear winner. The whole dataset is 573,392 rows. Overall, this would take (95.918 / 88662)* 573392 = 620.3178 seconds or about 10 minutes to complete on all data. Chances are, I’ll have to run this code a few times as the analysis changes. In contrast, the Rcpp method takes (8.427 / 88662)* 573392 = 54.49882 seconds. That’s under 6 minutes to code and implement a working, faster solution in Rcpp!

The code is quite easy to understand too:

#include <Rcpp.h>

using namespace Rcpp;

// [[Rcpp::export]]
IntegerVector countGenotypes(IntegerVector x) {
    // This method is a specialized version of R's table that counts genotypes
    // encoded as 0, 1, 2 in a vector (and also returns NA) always of length 4,
    // always as numbers of 0, 1, 2, NA. This allows faster usage with apply, as
    // we don't need to convert to factor to get all genotype counts, even if
    // none are present.
    CharacterVector names = CharacterVector::create("0", "1", "2", "NA");
    IntegerVector genocounts(4);
    genocounts.fill(0);
    for (int i = 0; i < x.length(); i++) {
            if (!IntegerVector::is_na(x[i])) {
                    genocounts[x[i]] += 1;
            } else {
                    genocounts[3] += 1;
            }
    }
    genocounts.attr("names") = names;
    return genocounts;
}

MD Tags in BAM Files

Fri, 17 Jan 2014 00:00:00 +0000

I needed to work with the MD tag in BAM/SAM files for a recent project. There’s not too much discussion online about this, so I took some notes as I went through examples.

The MD tag is for SNP/indel calling without looking at the reference. It does this by carrying information about the reference that the read does not carry, for a particular alignment. A SNP’s alternate base is carried in the read, but without the MD tag or use of the alignment reference, it’s impossible to know what the reference base was. Thus, this information is carried in the MD tag. A SNP looks like:

10A3T0T10

Here, there are three SNPs:

11 bases in from the aligned portion of the read, the reference has an A and the read has what ever base is at the 10th position (excluding softclips).
15 bases in there’s a T in the reference.
16 bases in there’s a T in the reference. Note that 0s are use used to indicate positions of neighboring SNPs.

Likewise, a reference would be necessary to know the deleted bases from the reference in an alignment. The MD tag stores this information too:

85^A16

Here, there are 85 matches, 1 deletion from the reference (the reference has an A there and the read doesn’t), and then there are 16 matches.

Example 1: With Insertion

Note that insertions, since they don’t represent a loss of information about the reference, are not stored in MD flag. This has some interested consequences. Let’s look at an example:

read seq length: 101
CIGAR: 89M1I11M
MD: 100

Immediately, the surprising part is that the MD tag represents 100 matches to the reference, but the read length is 101 bases and the CIGAR string is 101. This comes back to the core purpose of MD tags: they only represent information about the read aligned to the reference. There are 100 bases that align to the reference, and one insertion that does not.

Example 2: More Complex Insertion

read length: 101
CIGAR: 9M1I91M
MD: 48T42G8
name: HWI-ST222:4:1105:19266:186667#0 // for my reference

Here, there are SNPs (or errors) in the M (match/mismatch) parts of the alignment. The total MD length is: 48 + 1 + 42 + 1 + 8 = 100, which matches our read length minus insertions. What gets tricky (and in my opinion slightly annoying) is that the match component of the MD tag (the numeric parts) overlaps the insertion, but does not show it. This means that our read sequence has an insertion at the 10th base. However, here is where things get tricky: the mismatch at the 49th base (where the reference is a T according to the MD tag) is actually the 50th base in the read. This is because MD ignores insertions and we have a 1 base insertion upstream of the mismatching T. The same is true with the other mismatch (reference has G): according to MD tag, it’s 48 + 1 + 42 + 1 = 92 bases in, but it’s actually 93 bases in.

As an aside, running samtools calmd with -e, which changes masking bases to = really helps seeing these details. Read inspection in IGV also helps.

Example 3: Deletions

Deletions are stored in the MD tag, because these represent a loss of information with respect to the reference. Let’s look at a simple example:

read length: 101
CIGAR: 56M1D45M
MD: 56^A45
read name: HWI-ST222:4:2101:12455:194028#0

CIGAR string length is: 56 + 1 + 45 = 102. MD length is 56 + 1 + 45 = 102. This case is pretty trivial because deletions are indicated in both the MD tag and CIGAR string.

Example 4: Insertions and Deletions

Here’s a trickier example:

read length: 101
CIGAR: 31M1I17M1D37M
MD: 6G4C20G1A5C5A1^C3A15G1G15
read name: HWI-ST222:4:1208:7027:16535#0

That’s a long one! Let’s look at the total lengths of CIGAR string and MD tag. CIGAR length: 31 + 1 + 17 + 1 + 37 = 87. Parsing this is quite tricky.

Approaches

Initially I thought of a single-pass approach. This would almost surely involve a finite state automaton that manages four states: CIGAR token and MD token, read position, reference position. This is quite tricky, so an easier approach is to rebuild the reference string with the MD tag, and then use it to compare to the align read (following positions from CIGAR string). This way only either MD or CIGAR states need to be kept in focus at same time.

Using Named Pipes and Process Substitution in Bioinformatics

Thu, 08 Aug 2013 00:00:00 +0000

It’s hard not to fall in love with Unix as a bioinformatician. In a past post I mentioned how Unix pipes are an extremely elegant way to interface bioinformatics programs (and do inter-process communication in general). In exploring other ways of interfacing programs in Unix, I’ve discovered two great but overlooked ways of interfacing programs: the named pipe and process substitution.

Why We Love Pipes and Unix

A few weeks ago I stumbled across a great talk by Gary Bernhardt entitled The Unix Chainsaw . Bernhardt’s “chainsaw” analogy is great: people sometimes fear doing work in Unix because it’s a powerful tool, and it’s easy to screw up with powerful tools. I think in the process of grokking Unix it’s not uncommon to ask “is this clever and elegant? or completely fucking stupid?”. This is normal, especially if you come from a language like Lisp or Python (or even C really). Unix is a get-shit-done system. I’ve used a chainsaw, and you’re simultaneously amazed at (1) how easily it slices through a tree, and (2) that you’re dumb enough to use this thing three feet away from your vital organs. This is Unix.

Bernhardt also has this great slide, and I’m convinced there’s no better way to describe how most Unix users feel about pipes (especially bioinformaticians):

Pipes are fantastic. Any two (well-written) programs can talk to each other in Unix. All of the nastiness and the difficulty of inter-process communication is solved with one character, |. Thanks Doug McIlroy and others. The stream is usually plaintext, the universal interface , but it doesn’t have to be. With pipes, it doesn’t matter if your pipe is tab delimited marketing data, random email text, or 100 million SNPs. Pipes are a tremendous, beautiful, elegant component of the Unix chainsaw.

But elegance alone won’t earn a software abstraction the hearts of thousands of sysadmins, software engineers, and scientists: pipes are fast. There’s little overheard between pipes, and they are certainly a lot more efficient than writing and reading from the disk. In a past article I included the classic Samtools pipe for SNP calling. It’s no coincidence that other excellent SNP callers like FreeBayes make use of pipes: pipes scale well to moderately large data and they’re just plumbing. Interfacing programs this way allows us to check intermediate output for issues, easily rework entire workflows, and even split off a stream with the aptly named program tee .

Where Pipes Don’t Work

Unix pipes are great, but they don’t work in all situations. The classic problem is in a situation like this:

$ program --in1 in1.txt --in2 in2.txt --out1 out1.txt \
          --out2 out2.txt > stats.txt 2> diagnostics.stderr

My past colleagues at the UC Davis Bioinformatics Core and I wrote a set of tools for processing next-generation sequencing data and ran into this situation. In keeping with the Unix traditional, each tool was separate. In practice, this was a crucial design because we saw such differences in data quality due to different sequencing library preparation. Having separate tools working together, in addition to being more Unix-y, lead to more power to spot problems.

However, one step of our workflow has two input files and three output files due to the nature of our data (paired-end sequencing data). Additionally, both in1.txt and in2.txt were the results of another program, and these could be run in parallel (so interleaving the pairs makes this harder to run in parallel). The classic Unix pipe wouldn’t work, as we had more than one input and output into a file: our pipe abstraction breaks down. Hacky solutions like using standard error are way too unpalatable. What to do?

Named Pipes

One solution to this problem is to use named pipes. A named pipe, also known as a FIFO (after First In First Out, a concept in computer science), is a special sort of file we can create with mkfifo:

$ mkfifo fqin
  prw-r--r--    1 vinceb  staff          0 Aug  5 22:50 fqin

You’ll notice that this is indeed a special type of file: p for pipe. You interface with these as if they were files (i.e. with Unix redirection, not pipes), but they behave like pipes:

$ echo "hello, named pipes" > fqin &
 [1] 16430
$ cat < fqin
 [1]  + 16430 done       echo "hello, named pipes" > fqin
hello, named pipes

Hopefully you can see the power despite the simple example. Even though the syntax is similar to shell redirection to a file, we’re not actually writing anything to our disk. Note too that the [1] + 16430 done line is printed because we ran the first line as a background process (to free up a prompt). We could also run the same command in a different terminal window. To remove the named pipe, we just use rm.

Creating and using two named pipes would prevent IO bottlenecks and allow us to interface the program creating in1.txt and in2.txt directly with program, but I wanted something cleaner. For quick inter-process communication tasks, I really don’t want to use mkfifo a bunch of times and have to remove each of these named pipes. Luckily Unix offers an even more elegant way: process substitution.

Process Substitution

Process substitution uses the same mechanism as named pipes, but does so without the need to actually create a lasting named pipe through clever shell syntax. These are also appropriately called “anonymous named pipes”. Process substitution is implemented in most modern shells and can be used through the syntax <(command arg1 arg2). The shell runs these commands, and passes their output to a file descriptor, which on Unix systems will be something like /dev/fd/11. This file descriptor will then be substituted by your shell where the call to <() was. Running a command in parenthesis in a shell invokes a seperate subprocess, so multiple uses of <() are run in parallel automatically (scheduling is handled by your OS here, so you may want to use this cautiously on shared systems where more explicity setting the number of processes may be preferable). Additionally, as a subshell, each <() can include its own pipes, so crazy stuff like <(command arg1 | othercommand arg2) is possible, and sometimes wise.

In our simple fake example above, this would look like:

$ program --in1 <(makein raw1.txt) --in2 <(makein raw2.txt) \
          --out1 out1.txt --out2 out2.txt > stats.txt 2> diagnostics.stderr

where makein is some program that creates in1.txt and in2.txt in the original example (from raw1.txt and raw2.txt) and outputs it to standard out. It’s that simple: you’re running a process in a subshell, and its standard out is going to a file descriptor (the /dev/fd/11 or whatever number it is on your system), and program is taking input from that. In fact, if we see this process in htop or with ps, it looks like:

$ ps aux | grep program
  vince  [...] program --in1 /dev/fd/63 --in2 /dev/fd/62 --out1 out1.txt --out2 out2.txt > stats.txt 2> diagnostics.stderr

But suppose you wanted to pass out1.txt and out2.txt to gzip to compress them? Clearly we don’t want to write them to disk, and then compress them, as this is slow and a waste or system resources. Luckily process substitution works the other way too, through >(). So we could compress in place with:

$ program --in1 <(makein raw1.txt) --in2 <(makein raw2.txt) \
          --out1 >(gzip > out.txt.gz) --out2 >(gzip > out2.txt.gz) > stats.txt 2> diagnostics.stderr

Unix never ceases to amaze me in its power. The chainsaw is out and you’re cutting through a giant tree. But power comes with a cost here: clarity. Debugging this can be difficult. This level of complexity is like Marmite: I recommend not layering it on too thick at first. You’ll hate it and want to vomit. Admittedly, the nested inter-process communication syntax is neat but awkward — it’s not the simple, clearly understandable | that we’re used to.

Speed

So, is this really faster? Yes, quite. Writing and reading to the disk comes at a big price — see latency numbers every programmer should know . Unfortunately I am too busy to do extensive benchmarks, but I wrote a slightly insane read trimming script that makes use of process substitution. Use at your own risk, but we’re using it over simple Sickle /Scythe /Seqqs combinations. One test uses trim.sh, the other is a simple shell script that just runs Scythe in the background (in parallel, combined with Bash’s wait), writes files to disk, and Sickle processes these. The benchmark is biased against process substitution, because I also compress the files via >(gzip > ) in those tests, but don’t compress the others. Despite my conservative benchmark, the difference is striking:

Additionally, with the >(gzip > ) bit, our sequences had a compression ratio of about 3.46% — not bad. With most good tools handling gzip compression natively (that is, without requiring prior decompression), and easy in-place compression via process substitution, there’s really no reason to not keep data large data sets compressed. This is especially the case in bioinformatics where we get decent compression ratios, and our friends less, cat, and grep have their zless, gzcat, and zgrep analogs.

Once again, I’m astonished at the beauty and power of Unix. As far as I know, process substitution is not well know — I asked a few sysadmin friends and they’d seen named pipes but not process substitution. But given Unix’s abstraction of files, it’s no surprise. Actually Brian Kernighan waxed poetically about both pipes and Unix files in this classic AT&T 1980s video on Unix . Hopefully younger generations of programmers will continue to discover the beauty of Unix (and stop re-inventing the wheel, something we’ve all been guilty of). Tools that are designed to work in the Unix environment can leverage Unix’s power end up with emergent powers.

If you want more information on Unix’s named pipes, I suggest:

Checking out the tee example and other examples on Wikipedia’s process substitution page.
This 1997 article from Linux Journal on named pipes.

Share your experiences/wisdom/comments/critiques; I’m on Twitter .

Bioinformatics and Interface Design

Sat, 26 Jan 2013 00:00:00 +0000

Day to day bioinformatics involves interfacing and executing many programs to process data. We end up with some refinement of the data from which we extract biological meaning through data analysis. Given how much interfacing bioinformatics involves, this process undergoes very little thought or design optimization.

Much more attention is needed on the design of interfaces in bioinformatics, to improve their ease of use, robustness, and scalability. Interfacing is a low-level problem that we shouldn’t be wasting time on when there are much better high-level problems out there.

More bluntly, interfacing is currently an inconvenient glue that full time bioinformaticians waste too many hours on. There is no better illustration of this than by looking at how much time we waste in file parsing tasks. Parsers are most commonly employed in bioinformatics as crappy interfaces to non-standard formats. We need better designed interfaces and cleaner interface patterns to help.

I’m hardly the only one to complain about this. Fred Ross had this sadly accurate description :

Estimating from my own experience and observation of my colleagues, most bioinformaticists today spend between 90% and 100% of their time stuck in cruft. Methods are chosen because the file formats are compatible, not because of any underlying suitability. Second, algorithms vanish from the field…. I’m worried about the number of bioinformaticists who don’t understand the difference between an O(n) and an O(n^2) algorithm, and don’t realize that it matters.

He’s parting with bioinformatics , leaving our field with one less person to fix things. However, if practices are suboptimal and frustrating now, it’s not because people are unprepared to implement better approaches, it’s because they’re content with the status quo because it does work. But, as I’ll argue, we shouldn’t be wasting our time on this and much more elegant solutions exist.

The Current Interfacing Practice and its Paradigm

The current practice is characterized by system calls from a scripting language to execute larger bioinformatics programs, parse any output files created, and repeat. In my mind, I see this as a paradigm in which state is stored on the file system, and execution is achieved by passing a stringified set of command-line arguments to a system call.

This practice and paradigm has been the standard for bioinformatics for ages. It’s clunky and inelegant, but it works for routine bioinformatics tasks. However, the current practice isn’t well suited for large, embarrassingly parallel tasks that will grow increasingly common as the number of samples increases in genomics projects. Portability of these pipelines is usually terrible, and involves awkward tasks like ensuring all called programs are in a user’s $PATH (good luck with version differences too). Program state is stored on the disk, the slowest component apart from the network. Here’s a better look at how to really understand how costly storing state on a disk can be (zoom into this image) .

Storing state on a slow, highly-mutable, non-concurrent component is only acceptable if it’s too big to store in memory. Bioinformatics certainly has tasks that produce files too large to store in memory. However, if a user had a task with little memory overhead, the complete lack of interfaces other than the command-line to all aligners, mappers, assemblers, etc would require the user to write to the disk. If they’re clever, they can invoke a subprocess from a script and capture standard out, but then they’re back to parsing text as a sloppy interface, rather than handling higher-level models or objects. This needs to change.

While I’m maybe being a little harsh on the current paradigm, I should say some parts are elegant. Unix piping is extremely elegant, and avoids any latency due to writing to the disk between execution steps. Workflows like this example from the samtools mpileup man page are clear and powerful:

$ samtools mpileup -uf ref.fa aln1.bam aln2.bam | bcftools view -bvcg - > var.raw.bcf  
$ bcftools view var.raw.bcf | vcfutils.pl varFilter -D100 > var.flt.vcf

The real drawback of the current system occurs when a user wants to leverage the larger command-line programs for something novel. Most aligners assume you want to align a few sequences, output the results, and then stop. A user that wants an aligner to align a few sequences, and then proceed down different paths depending on the output has the hassle of writing the sequences to a FASTA file or serializing the sequences in the FASTA format, invoking a subprocess, and then either passing it a filename, or (if the tool supports this), passing the serialized string to the subprocess through standard in. If the command-line tool has overheard for starting up, this is incurred during each subprocess call (even if it could be shared and the overhead amortized across subprocess calls).

File Formats and Interfacing

To make matters worse, many bioinformatics formats like FASTA, FASTQ, and GTF are either ill-defined or implemented differently across libraries, making them risky interface formats. In contrast, consider the elegance of Google’s protocol buffers . This allow users to write their data structures in the protocol buffer interface description language , and compile interfaces for C++, Java, and Python. This is the type of high-level functionality bioinformatics needs to interface incredibly complex data structures, yet we’re still stuck in the text parsing stone age.

Foreign Function Interfaces and Shared Libraries

One way to avoid unnecessary clunky system calls is through foreign foreign interfaces (FFIs) to shared libraries, using thin wrappers in a user’s scripting language of choice. Large bioinformatics programs like aligners, assemblers, and mappers are most commonly written in C or C++, which allows code to be compiled as a shared library relatively painlessly.

FFIs could solve a few problems for bioinformaticians. First, they would allow much more code recycling in low-level projects: rather than writing your own highly-optimized C FASTA/FASTQ parser, you can ejust link against a shared library with that routine. Additionally, that shared library can be separately developed and improved.

Second, FFIs allow modularity and high-level access to low-level routines. Genome assemblers are packed to the gills with useful functions. So are aligners. Yet unless the developer took the time to separate this out via subcommands or an API (like git or samtools), you’re unlikely to ever be able to access this functionality. Developers with an eye for better program design can write higher level functions that could be utilized through a FFI. Now, novel bioinformatics tasks that may require some sequence assembly, or a few parallel calls to an aligner can be tackled without the system call rubbish, or re-implementing all the low-level algorithms.

For higher-level functionality with FFIs and shared libraries, wrappers work beautifully. Rather than wrapping entire programs through the command line (as BioPython does ), scripting language libraries could interact more directly with low-level programs. In cases in which the current paradigm just doesn’t fit, we’d have the option to avoid it by calling routines directly. Tools like samtools are very successful because they have a powerful API that allow programs like pysam to call their routines.

Imagine now that you could also load adapter and quality trimmers wrappers around shared libraries. Rather than using Unix pipes or bash scripts to write quality control pipelines, and have every program in the execution chain read, parse, and then write FASTA formatted files, it could be done once, using object abstractions of data.

import sys
import biolib
import sickle
import scythe

for read in biolib.read_fasta(open(sys.argv[1])):
    read_tmd = scythe.trim(read.seq, read.qual, prior=0.3)
    biolib.write_fasta(sickle.trim(read_tmd.seq, read_tmd.qual, qual=20), stdout)

In this artificial example, reads could then be sent directly to aligners rather than to standard out. Working with higher-level models of data (in this case, a read) allows easier real-time debugging, statistics gathering, and parallelization. Imagine being able to put this entire block in a try statement, and have exceptions handled at a higher level. An error could invoke a debugger, and a bioinformatician could inspect the culprit interactively in real-time. This is impossible in the old paradigm (and we’ve all spent ages using streaming tools to track down such bugs).

Note that I’m not arguing that your average biologist should suddenly start trying to understand position-independent code and compile shared libraries to avoid making systems calls in their scripts. Sometimes a system call is the right tool for the job. But bioinformatics software developers should reach for a system call not because it’s the only interface, but because it’s the best interface for a particular task. Maybe someday we’ll even see thin wrappers coming packaged with bioinformatics tools (even if under contrib/ and written by other developers) — I can dream, right?

FFI in Practice

Here’s a real FFI example: I needed to assemble many sets of sequences for a pet project. I really wanted to avoid needlessly writing FASTA files of these sets of sequences to disk, as this is quite costly. However, most assemblers were designed solely to be interfaced through the command-line. The inelegance of writing thousands of files, making thousands of system calls to an assembler, then reading and parsing the results not appealing, so I wrote pyfermi , a simple Python interface to Heng Li’s Fermi assembler (note that this software is experimental, so use it with caution). First off, I couldn’t have done this without Heng’s excellent assembler and code, so I owe him a debt of gratitude. The Fermi source has a beautiful example of high-level functions that can be interfaced with relative ease: see the fm6_api_* functions used in example.c .

I wrote a few extra C functions in pyfermi (mostly to deal with void * necessary because Python’s ctypes can’t handle foreign types AFAIK), and compile Fermi as a shared library. I was able to do all this in far less time than it would have taken me to go down the route of writing thousands of files, making syscalls, and handing file parsing.

The Importance of Good Tools

Overall, bioinformaticians need to be more conscious of the design and interfacing of our tools. I strongly believe tools and methods shape how we approach and solve problems. A data programmer only trained in Perl will likely at some point wage a messy war trying to produce decent statistical graphics. Likewise, a statistician only trained in t-tests and ANOVA, will only see normally distributed data (and apply every transformation under the sun to force their data into this shape). I’m hardly the first person to argue that this occurs: this idea is known as the law of the instrument. Borrowing a 1964 quote from Abraham Kaplan (and Wikipedia ):

I call it the law of the instrument, and it may be formulated as follows: Give a small boy a hammer, and he will find that everything he encounters needs pounding.

Earlier in our bioinformatics careers, we’re trained in file formats, writing scripts, and calling large programs via the command-line. This becomes a hammer, and all problems start looking like nails. However, the old practice and paradigms are breaking down as the scale of our data and the complexity of our problems increase. We need new school bioinformatics with a focus on the bigger picture, so let’s start talking about how we do it.

21st Century Science and the Need for Open Data and Open Tools

Sun, 13 Jan 2013 00:00:00 +0000

Note: I’ve been writing this essay in my head for a few months, but I felt it needed to be completed and released after the sad loss of Open Access advocate Aaron Swartz , a hacker and activist I admired.

21st Century Science and the Need for Open Data and Open Tools

Open Science rests upon three core principles: open access, open data, and open tools. However, “open science” doesn’t really imply the importance of this philosophy; it’s perceived as a nice, but not entirely necessary trait of scientific projects. Yet, without access, data, and tools, science is not just not open, but it is not reproducible.

This is a new phenomenon too; hundreds of years ago, a scientific article was sufficient to reproduce an experiment. Methods could be followed to recollect the data, build or collect the necessary tools, and follow the procedures to recreate the output. I believe this is why the Open Access movement has had such a large role in the Open Science movement. People readily accept that reproducibility is a goal of science, and one cannot reproduce a scientific finding without access to scholarly articles.

However, science undergoing a transition that makes the scholarly article alone insufficient to reproduce an experiment. As I’ll argue, Open Science advocates need to quickly start focusing more energy into open tools and open data.

Capital-Intensive Science and the Openness of Data

With large-scale projects like the LHC, new sequencing centers, and Mars Curiousity, it’s clear that some parts of science of becoming increasingly capital intensive. From my perspective in the genomics world, it’s even more apparent: smalls labs are seeking large funds to carry out sequencing efforts. The staggering levels of capital are needed for investment in expensive and quickly-developing new machines, whether sequencer, collider, or GC-MS. Besides the probihitive costs these machines have in common, they also have another trait they share: the create a lot of data.

But the beauty of this new data is that it’s practically free to copy and use. Science labs in the developing world may be decades away from having start of the art sequencers, or GC-MSs, or colliders, but they can benefit from the presence of such technology in Japan, the USA, or Switzerland through access to the data produced, as long as the data is open. In essence, the beauty of big data is not only the information it contains, the complexity it can untangle, or the insights it can foster. It’s that it can be endlessly duplicated and shared, and that any scarcity is entirely artificial . As long as big data remain open, the information created through capital-intensive science can be widely used and studied by all.

Futhermore, the new capital-intensive science could be creating a new era of impossible reproducibility. Small labs cannot afford resequence samples just to recreate data; this is why we see a growth of sequence repositories. As data becomes larger and begins coming from more sources, there will be technical problems and cost issues in maintaining public repositories of open data (such issues have already played out with the Short Read Archive). Open Science must be ready for such battles.

The Importance of Open Source Software

While large new scientific machinery illustrates the new capital-intensive side of science, the growth of bioinformatics, data science, and scientific programming illustrates the other side: labor-intensive science with a high degree of specialization. The science of the past centuries was characterized by a few or a single scientist carrying out an experiment, analyzing the results, and publishing them. However, the new scientific challenges we face require a degree of task specialization that is a break from the past. The necessity to model and understand more complex relationships requires not just bench scientists to organize and run experiments, but statisticians to consults, systems administrators to setup and maintain large computing infrastructure, and programmers that can write and maintain large codebases.

Yet as with big data, the software tools created to analyze data can be duplicated, reused, and adapted for no cost, as long as it’s open source. Open Science must strongly advocate for the use of open source tools, and also the development of open source alternatives to proprietary tools.

A further challenge persists too: open source software communities are almost always factionated. Unlike proprietary software, in which a company develops, releases, and works on one a single product, the open source community frequently recreates software, rather than extends or improves existing software. Even worse, while a company has managers and hierarchy to maintain continuity in functionality, documentation, and quality of a software product while individual developers may leave, open source software projects frequently decay as lead developers switch groups or funding runs short.

Adjusting Our Thinking

Open Access is an ongoing battle, but also point of pride of Open Science advocates. The success of PLoS and the increasing number of open access journals are signs the open science movement is gaining wide support. Yet, the science of the 21st century, characterized by a capital and labor intensive production process, requires further advocacy in open data and open tools.

My First Recommendation to New Scientific Coders: Learn Visualization

Wed, 14 Nov 2012 00:00:00 +0000

Scientists are learning programming at an unprecedented rate. I’ve expressed concern over the fast-paced growth of computing across the sciences and what this could mean for reproducibility and incorrect findings in the sciences. Perhaps the best example that illustrates the severity of this issue is Coombes and Baggerly’s Duke Saga .

I think a lot about how scientists learn programming and how we can change this process to yield a better outcome (fewer errors, more readible and reproducible code). Scientific coders must learn to program in a particular fashion that “stacks the deck” to make errors apparent. On this front, unit tests, following coding standards, and peer code review get a lot of deserved attention. Yet for some reason, visualization does not. This is unfortunate; visualization should be learned to a high degree of competency very early on in a programmer’s career.

Problems Look Differently When You Can Visualize Quickly

Neil DeGrasse Tyson has an excellent saying: “If you are scientifically literate the world looks very different to you.” Similarly, if you have the skills to visualize information quickly, problems start to look different to you. Don’t only learn to visualize, learn to do it so effectively that each time you imagine a visualization, there’s almost no time cost in implementing it.

Why do I stress being efficient at visualization? If there’s no barrier to a coder making a plot —if a coder doesn’t think before each plot, “shit, now I have to remember how to do this”— they’ll more readily apply it to everything, and fewer errors will go unnoticed. If the barrier is high, a coder will hesitate and end up using it less as a tool.

Visualization also drops the barrier for quick interpretation of data. A graphic display of data is often more efficient to interpret than numbers and tables. Trying to interpret a four dimension table requires a lot of mental cycles and time. Look at the Titanic dataset from R:

Age = Child, Survived = No              Age = Adult, Survived = No
       Sex                                             Sex
 Class  Male Female                         Class  Male Female
   1st     0      0                          1st     0      0
   2nd     0      0                          2nd     0      0
   3rd    35     17                          3rd    35     17
   Crew    0      0                          Crew    0      0

Age = Child, Survived = Yes             Age = Adult, Survived = Yes
      Sex                                      Sex
 Class  Male Female                       Class  Male Female
   1st     5      1                         1st    57    140 
   2nd    11     13                         2nd    14     80
   3rd    13     14                         3rd    75     76
   Crew    0      0                         Crew  192     20

Now, consider d3.js this parallel sets visualization by Jason Davies:

Immediately, parallel sets shows us the large numbers that perished in the Titanic’s sinking. Width reveals not only the breakdown or survivors/non-survivors, but also the composition of the ship’s passengers prior to hitting the iceberg. This additional data could only be calculated from the table by manually adding across separate tables, which again incurs a time cost.

Smart Visualization Over Stupid Hypothesis Testing

Imagine you’ve just written a nucleotide sequence processing algorithm and you want to make sure it isn’t being confounded by large sequences or small sequences. Some scientists reach for a hypothesis test. No! Visualize it. A hypothesis test is inherently univariate. Visualization is multidimensional. In this case, I would plot in two dimensions and color by sequence length. Or use density plots and color by sequence length. Then try a logarithmic scale. Try coloring by sequence length in continuous and discrete color scales.

Statisticians are obsessed by confounding variables, but I feel folks writing data processing scripts are not. The example above is what I call color-by-confounder (well, possible confounder). If a variable that should be unrelated to another is forming a colorful cluster in a scatterplot, visualization (and your pattern-finding ape-brain) is much more effective than a clumsily applied, assumption-ridden, old-fashioned, philosophically-troubled hypothesis test.

It’s worth mentioning that while our ape-brains are indeed excellent visual pattern finding machines, they can also be prone to false positives. Getting in the habit of forming hypotheses about how a graphic should look before creating it can help protect us from apophenia . I’ve seen strange patterns emerge in data that scream, “you screwed something up big”, but after heavy thought reveal everything is fine.

Build Tools that Support Visual Output

Developers should try to output data in file formats that are very easily parsed by existing functions or libraries in popular languages. It’s indefensible to make up a file format when your data can be equally well expressed in an existing one. Most data can be expressed in tab-delimited, CSV, JSON, or XML. It takes two lines in R to load any of these file formats with the appropriate library; there is virtually no barrier to loading and visualizing data from these formats.

Recently I had to process some variable-space tabular output from a popular bioinformatics program. The manual had a footnote saying, “contrary to the shrieks of outrage we occasionally receive about this, space-delimited files are just as trivial to parse as tab-delimited files.” Considering the header was across two rows (seriously), data can contain spaces (and is not quoted), and the delimiters could have as few as one space, this is most definitely not safely trivial across datasets.

Attempting to use a human-readable format such as variable-spaced/fixed-width-column formats makes the rather silly assumption that your data will only be looked at by a human. It’s always easier to make human readable data out of computer-readable data than to do the opposite. In today’s big data age, I’m skeptical humans actually process huge data sets in human-readable file formats in any way that’s both meaningful and not horribly inefficient.

Ok, How do I Learn Visualization?

The first requirement to be able to visualize quickly is to know your tools. R is unequivocally the best tool to learn first, and learn the best. Buy Hadley Wickham’s ggplot2 book, and bookmark his website. Learn ggplot2 thoroughly; it scales almost impossibly well to the complexity of problems you throw at it (primarily because it’s built around an ingenious abstraction).

Edward Tufte also has some excellent books on visualization worth investing in. The Visual Display of Quantitative Visualization is probably the best to start with. Other excellent projects worth being aware of are:

Color Brewer for intelligent color choices for different problems.
d3.js is a new-ish Javascript visualization framework I am very excited about.
ggobi is a useful system for high-dimension visualization.
lattice was the first graphics package for R I learned, and even though I use ggplot2 primarily now, it is still useful.
ggbio if you do bioinformatics. Learning to use genome browsers, track formats (BED, WIG, GTF), and read visualization programs (such as IGV ) are also very important skills.

Generally, the best advice about learning visualization is to be patient, and not settle for a subpar graphic. Patience and perfectionism will lead to better graphics and a thorough understanding of the tools.

Conclusion

Visualization is a skill worth investing time in. It’s a low hanging fruit for all programmers. It’s also enjoyable. Fundamentally, developers need to adjust how they think about visualization. It’s not something to brush up on every time a plot is needed for an article or presentation. It should become part of every professional developer’s workflow, right alongside version control, debugging, and unit tests.

Simple Parallel Bioinformatics Pipelines with find, basename, and xargs

Mon, 08 Oct 2012 00:00:00 +0000

Simple Parallel Bioinformatics Pipelines with find, basename, and xargs

Big-Ass Servers and Data Parallelism

A routine operation in bioinformatics is to process a lot of files on a so-called “Big-Ass Server” . In most cases, these have to be processed using the same tools, in the same way, making this a prime example of data-parallism. The unit of data divided across multiple cores is the file.

Note that there’s very little opportunity for task-parallism in bioinformatics file processing pipelines. Consider a common task of most bioinformatics sequencing projects (resequencing, RNA-seq, etc): taking reads from many samples, running quality diagnostics, and running adapter and quality trimming. None of these steps can be done in a task-parallel fashion; they must be pipelined.

Find-Basename-Xargs

Suppose in this example I have genomic sequencing data of 50 human individuals. All individuals’ sequences are in the directory seq/ and are semantically named in the format of {lane-number}-{individual-id}.fastq. In this example, suppose I just want to run each file through my adapter trimming program scythe in parallel, using four cores.

For such tasks, I frequently use a design pattern I refer to as “find-basename-xargs” (FBX hereafter). I doubt this is novel, but I do refer to this specific design pattern frequently. I’ll dissect an example.

Our seq/ directory contains many files we wish to trim with Scythe, in parallel. The first step of FBX is to use the find command to find all relevant files. In our case:

$ find seq -name "*.fastq"
seq/african-1.fastq
seq/african-10.fastq
seq/african-11.fastq
seq/african-12.fastq
# [...]

Next, basename is used to drop the extension and seq/ directory, which leaves us with only the identifying string (I think of this as a key… you’ll see why in a few weeks). Having just the identifying key allows us to specify the output directory and any file suffixes. We use basename with xargs because we’re processing each incoming line from stdin at a time (-n1 specifes one argument will be taken at a time). The command results would look like:

$ find seq -name "*.fastq" | xargs -n1 -I{} basename {} .fastq
african-1
african-10
african-11
african-12
# [...]

The argument -I{} specifes the replacement string. Since we’re the last positional argument of basename is .fastq, this is necesary.

Finally, we do the processing with xargs. GNU parallel also works, and provides nice additional features. Scythe takes some fixed arguments (adapter, prior) and file options dependent on the key (input file, output file). So to run Scythe, on each file in parallel and output the results to the trimmed directory, we’d use xargs with -n1 -P4 -I{}. Our exact command would be:

find seq -name "*.fastq" | \
  xargs -n1 -I{} basename {} .fastq | \
  xargs -n1 -P10 -I{} scythe -a adapters.fasta -p 0.4 -o trimmed/{}.fastq seq/{}.fastq

Note that we re-specify the directories and extensions so Scythe can find and process the appropriate file.

Redirecting Output in Parallel and XBF Chaining

I’m a stickler about gathering and analyzing statistics at all intermediary steps in a bioinformatics processing pipeline. Scythe will output summary statistics on found adapters in the reads, but it prints it out to stderr. With hundreds of files being processed, simple redirecting stderr to a file via 2> is ineffective. Since xargs is calling the program, there’s no way to redirect a specific Scythe calls’ stderr to a specific file.

My work around this is to wrap a command call in a bash shell script. Our shell script would be incredibly simple (a better version would ensure directories exist):

#!/bin/bash
set -o nounset
set -e

IN=$1
echo " started running scythe on file '$IN'..." 1>&2
scythe -a adapters.fasta -p 0.4 -o trimmed/$IN.fastq seq/$IN.fastq 2> stats/$IN-output.txt
echo " completed scythe on file '$IN'." 1>&2
echo $IN

And we could call it in the same fashion.

find seq -name "*.fastq" | \
  xargs -n1 -I{} basename {} .fastq | \
  xargs -n1 -P10 bash run-scythe.bash

A few notes: output message to stderr, not stdout. This allows us to chain the FBX pattern. When a file’s processing is complete, execution will proceed to the echo line and send this file’s key to the next step. This incredibly simple approach allows parallel steps to be pipelined. If we use Sickle to do quality-based trimming, the pattern is similar:

find seq -name "*.fastq" | \
  xargs -n1 -I{} basename {} .fastq | \
  xargs -n1 -P10 bash run-scythe.bash | \
  xargs -n1 -P10 bash run-sickle.bash

Using Bioconductor to Analyze your 23andme Data

Mon, 12 Mar 2012 00:00:00 +0000

Bioconductor is one of the open source projects of which I am most fond. The documentation is excellent, the community wonderful, the development fast-paced, and the software very well written.

There’s a new package in the development branch (due to be released as 2.10 very soon) called gwascat. gwascat is a package that serves as an interface to the NHGRI’s database of genome-wide association studies.

Loading the package with library(gwascat) creates a GRanges instance of SNPs and their diseases. GRanges is a fundamental data structure in Bioconductor (specifically the GenomicRanges package) that is designed to hold ranges on genomes efficiently, as well as metadata about the ranges. In this case, the object gwrngs holds SNP ranges (well, locations) and metadata provided by the GWA studies in NHGRI’s database.

While I really do like 23andme’s interface to one’s genotype information and research, the gwascat package offers some nice data mining power. I’ll briefly introduce it here, and perhaps add additional details later on.

23andme Raw Data

When I was considering 23andme, I ultimately persuaded by the fact that they release their raw genotype calls to users. Unfortunately they do so without SNP call confidence data, but in a personal correspondence with a 23andme representative they stated:

Data reproducibility of our genotyping platforms is estimated at about 99.9%. Average call rate is about 99%. When samples do not meet sufficient call rate thresholds, we repeat the analysis, and/or request a new sample. We do not return data to customers that does not meet our quality thresholds.

The 99.9% figure sounds like a lot, but considering there are 960,545 SNPs being called, it’s not that high.

To retrieve raw data, simply click the “Account” link at the top of the page (after you’ve signed in) and click “Browse Raw Data”. There should be a download link. If you’ve never used GPG to encrypt a file, now is the time to learn; keep your SNP data encrypted.

The file 23andme provides has four columns: rs ID, chromosome, position, and genotype.

Loading Raw Data into R

Use read.table to load this data in R. It’s a lot of data, so providing this function with information about the type of data can speed this up quite a bit. Here is the code I used:

library(gwascat)
d <- read.table("data/genome_Vince_Buffalo_Full_20120313162059.txt",
               sep="\t", header=FALSE,
               colClasses=c("character", "character", "numeric", "character"),
               col.names=c("rsid", "chrom", "position", "genotype"))

You may notice that chromosome has the class “character” - this is because there are chromosomes X, Y, and MT (for mitochondrial). For later plotting purposes, it’s good to make this an ordered factor:

tmp <- d$chrom
d$chrom = ordered(d$chrom, levels=c(seq(1, 22), "X", "Y", "MT"))
## It's never a bad idea to check your work
stopifnot(all(as.character(tmp) == as.character(d$chrom)))

Where are the SNPs 23andme Genotypes?

Using Hadley Wickham’s excellent ggplot2 package, we can look at the distribution of SNPs by chromosome:

ggplot(d) + geom_bar(aes(chrom))

This isn’t providing information on SNP density as much as it is chromosome length (except X). We’ll take a more detailed look a bit later.

Another really wonderful aspect of Bioconductor is that the project isn’t just a repository of code: it also stores annotation, full genomes, and experimental data. Such packaged data is the foundating of reproducible bioinformatics, as you no longer have to worry about keeping track of data versions and storing downloaded data yourself. If you need to work with cutting edge data from Ensembl or UCSC tracks, the packages biomaRt and rtracklayer work well.

A Quick Demonstration of GenomicRanges and Bioconductor Annotation Packages

Suppose I want to see if any of my SNPs fall in the APOE gene region. For this, I’ll need transcript annotation data. If I wished to create a fresh database of exon, gene, transcript, and splicing data, I could with the GenomicFeature package. This package has methods for building transcriptDb objects from the Known Gene track from UCSC, as well as Ensembl databases. However, I’ll just use a pre-packaged version, TxDb.Hsapiens.UCSC.hg18.knownGene. I use hg18 rather than hg19 because this is the build that 23andme’s coordinates reference.

library(TxDb.Hsapiens.UCSC.hg18.knownGene)
txdb <- TxDb.Hsapiens.UCSC.hg18.knownGene
class(txdb) ## do some digging around!

transcriptDb objects have nice accessor functions for accessing their components. Behind the scenes, everything is in SQLite and very efficient (are you seeing why I love Bioconductor?).

If we look at the transcripts with the transcripts accessor function, we see it’s a GenomicRanges object:

> transcripts(txdb)

GRanges with 66803 ranges and 2 elementMetadata values:
          seqnames               ranges strand   |     tx_id     tx_name
             <Rle>            <IRanges>  <Rle>   | <integer> <character>
      [1]     chr1     [  1116,   4121]      +   |         1  uc001aaa.2
      [2]     chr1     [  1116,   4272]      +   |         2  uc009vip.1
      [3]     chr1     [ 19418,  20957]      +   |        26  uc009vjg.1
      [4]     chr1     [ 55425,  59692]      +   |        28  uc009vjh.1
      [5]     chr1     [ 58954,  59871]      +   |        29  uc001aal.1
      [6]     chr1     [310947, 310977]      +   |        33  uc001aaq.1
      [7]     chr1     [311009, 311086]      +   |        34  uc001aar.1
      [8]     chr1     [314323, 314353]      +   |        35  uc001aas.1
      [9]     chr1     [314354, 314385]      +   |        36  uc001aat.1
      ...      ...                  ...    ... ...       ...         ...
  [66795]     chrY [25318610, 25368905]      -   |     33721  uc004fwl.1
  [66796]     chrY [25318610, 25368905]      -   |     33722  uc010nxm.1
  [66797]     chrY [25586438, 25607639]      -   |     33731  uc004fws.1
  [66798]     chrY [25739178, 25740308]      -   |     33732  uc004fwt.1
  [66799]     chrY [25949151, 25949179]      -   |     33733  uc004fwu.1
  [66800]     chrY [26012854, 26012887]      -   |     33734  uc004fww.1
  [66801]     chrY [26015033, 26015066]      -   |     33735  uc004fwx.1
  [66802]     chrY [26015782, 26015809]      -   |     33737  uc004fwy.1
  [66803]     chrY [26016792, 26016820]      -   |     33738  uc004fwz.1

To interact with the wealth of data behind a transcriptDb object, we often group individual ranges into groups, leaving us with a GRangesList.

> tx.by.gene <- transcriptsBy(txdb, "gene")
> tx.by.gene
 GRangesList of length 20121:
 $1 
 GRanges with 2 ranges and 2 elementMetadata values:
       seqnames               ranges strand |     tx_id     tx_name
          <Rle>            <IRanges>  <Rle> | <integer> <character>
   [1]    chr19 [63549984, 63556677]      - |     61027  uc002qsd.2
   [2]    chr19 [63551644, 63565932]      - |     61033  uc002qsf.1

 $10 
 GRanges with 2 ranges and 2 elementMetadata values:
       seqnames               ranges strand | tx_id    tx_name
   [1]     chr8 [18293035, 18303003]      + | 26503 uc003wyw.1
   [2]     chr8 [18301794, 18302666]      + | 26504 uc010lte.1

 $100 
 GRanges with 2 ranges and 2 elementMetadata values:
       seqnames               ranges strand | tx_id    tx_name
   [1]    chr20 [42681577, 42713790]      - | 62142 uc002xmj.1
   [2]    chr20 [42681577, 42713790]      - | 62143 uc010ggt.1

 ...
 <20118 more elements>

Holy GRangeList batman! These are the transcripts grouped by gene. There are other methods for grouping by CDS and exons (cdsBy and exonsBy).

The names of the list elements are Entrez gene IDs. We can look up specific genes with another Bioconductor annotation package, org.Hs.eg.db. There are org.* annotation packages for many organisms. You can forge your own and interact with them with the AnnotationDbi package. I’m using a development version of this package that has a new slick SQL-like interface; it will be widely available with the upcoming 2.10 release.

Suppose I want to convert the Entrez Gene IDs to gene names. The “eg” in org.Hs.eg.db refers to Entrez Gene IDs. Printing the org.Hs.eg.db object gives a nice list of information. Let’s look for the APOE gene’s Entrez Gene ID.

> library(org.Hs.eg.db)
>  cols(org.Hs.eg.db)

    [1] "ENTREZID"     "ACCNUM"       "ALIAS"        "CHR"          "ENZYME"      
    [6] "GENENAME"     "MAP"          "OMIM"         "PATH"         "PMID"        
   [11] "REFSEQ"       "SYMBOL"       "UNIGENE"      "CHRLOC"       "CHRLOCEND"   
   [16] "PFAM"         "PROSITE"      "ENSEMBL"      "ENSEMBLPROT"  "ENSEMBLTRANS"
   [21] "UNIPROT"      "UCSCKG"       "GO"

These are the columns we can query out. Certain keys exist: we can access these using keytypes(). Using it all together, we can extract the Entrez Gene ID:

> select(org.Hs.eg.db, keys="APOE", cols=c("ENTREZID", "SYMBOL", "GENENAME"), keytype="SYMBOL")
    SYMBOL ENTREZID         GENENAME
    23200   APOE      348 apolipoprotein E

Now, we can look for this in our tx.by.gene GRangesList. A word of caution: Entrez Gene IDs are names and thus they need to be quoted when working with GRangesList objects from transcript databases.

> tx.by.gene["348"]
  GRangesList of length 1:
  $348 
  GRanges with 1 range and 2 elementMetadata values:
        seqnames               ranges strand |     tx_id     tx_name
           <Rle>            <IRanges>  <Rle> | <integer> <character>
    [1]    chr19 [50100879, 50104490]      + |     59642  uc002pab.1

If I had used tx.by.gene[348] the 348th element of the list would have been returned, not the transcript data for the APOE gene (which has Entrez Gene ID “348”).

Now, do any SNPs fall in this region? Let’s build a GRanges object from my genotyping data, and look for overlaps. Before I do, it’s worth mentioning another gotcha about working with bioinformatics data: chromosome naming schemes. Different databases use all sorts of schemes, and you should always check them. 23andme returns just numbers, X, Y, and MT. Let’s change it to use the same as the Bioconductor annotation.

# CAREFUL: use levels() to check that you're making new factor names
# that correspond to the old ones!
levels(d$chrom) <- paste("chr", c(1:22, "X", "Y", "M"), sep="")
my.snps <- with(d, GRanges(seqnames=chrom, 
                   IRanges(start=position, width=1), 
                   rsid=rsid, genotype=genotype)) # this goes into metadata

Now, let’s find overlaps using, well, findOverlaps:

apoe.i <- findOverlaps(tx.by.gene["348"], my.snps)

apoe.i is an object of class RangesMatching. Note that had we not matched chromosome names, Bioconductor gives us a nice warning that sequence names don’t match. We could look at the slots of apoe.i but output can be seen with matchMatrix:

> hits <- matchMatrix(apoe.i)[, "subject"]
>  hits
     [1] 873650 873651 873652 873653 873654 873655 873656 873657 873658 873659
    [11] 873660 873661 873662 873663 873664 873665 873666 873667 873668 873669
    [21] 873670 873671 873672 873673 873674 873675 873676

So in our subject, we have two hits. Let’s dig them up in our SNP GRanges object:

> my.snps[hits]

  GRanges with 27 ranges and 2 elementMetadata values:
         seqnames               ranges strand   |        rsid    genotype
            <Rle>            <IRanges>  <Rle>   | <character> <character>
     [1]    chr19 [50101007, 50101007]      *   |    rs440446          CG
     [2]    chr19 [50101842, 50101842]      *   |    rs769449          GG
     [3]    chr19 [50102284, 50102284]      *   |    rs769450          AG
     [4]    chr19 [50102751, 50102751]      *   |    rs769451          TT
     [5]    chr19 [50102874, 50102874]      *   |    i5000209          GG
     [6]    chr19 [50102904, 50102904]      *   |    i5000208          GG
     [7]    chr19 [50102940, 50102940]      *   |    i5000201          CC
     [8]    chr19 [50102991, 50102991]      *   |  rs28931576          AA
     [9]    chr19 [50103697, 50103697]      *   |  rs11542040          CC
     ...      ...                  ...    ... ...         ...         ...
    [19]    chr19 [50104077, 50104077]      *   |    i5000212          GG
    [20]    chr19 [50104118, 50104118]      *   |    i5000210          GG
    [21]    chr19 [50104129, 50104129]      *   |    i5000213          CC
    [22]    chr19 [50104154, 50104154]      *   |    i5000207          TT
    [23]    chr19 [50104177, 50104177]      *   |    i5000219          GG
    [24]    chr19 [50104180, 50104180]      *   |    i5000218          GG
    [25]    chr19 [50104198, 50104198]      *   |    i5000206          CC
    [26]    chr19 [50104268, 50104268]      *   |    i5000204          GG
    [27]    chr19 [50104333, 50104333]      *   |  rs28931579          AA

Now, we can verify that these SNPs are in the APOE gene using the UCSC Genome Browser (and actually pull open a browser to this spot from R using rtracklayer, but I’ll save that for another time). Be sure to use hg18/build 36! Note that my genotype information is there.

The ApoE4 allele is rs429358(C) + rs7412(C). The most common allele (ApoE3, or e3/e3) is rs429358(T) + rs7412(C) which is what I have (that’s a relief). There’s a lot of established research that shows homozygous ApoE4 (that is rs429358(C/C) + rs7412(C/C)) leads to substantially higher risk of Alzeheimer’s. According to SNPedia , James Watson requested he not learn his genotype at this locus, and Steven Pinker requested his ApoE data be removed from his PGP10 data.

Looking for Risk Variants using `gwascat`

We can use the metadata provided by gwascat to further look for interesting variants in our 23andme data. I would recommend interpreting this data with caution, as summarizing these findings in a single element metadata data frame is hard: there’s definitely lost information.

The gwrngs GRanges object has lots of metadata you should scan through with elementMetadata(gwrngs). The Strongest.SNP.Risk.Allele is useful for seeing what you’re at risk for. First, using the rs ID as a key, let’s join our SNP data with the gwrngs metadata:

gwrngs.emd <- as.data.frame(elementMetadata(gwrngs))
dm <- merge(d, gwrngs.emd, by.x="rsid", by.y="SNPs")

We can search for the risk allele in the 23andme genotype data with R and attach a vector of i.have.risk to the dm data frame:

risk.alleles <- gsub("[^\\-]*-([ATCG?])", "\\1", dm$Strongest.SNP.Risk.Allele)
i.have.risk <- mapply(function(risk, mine) {
  risk %in% unlist(strsplit(mine, ""))
}, risk.alleles, dm$genotype)
dm$i.have.risk <- i.have.risk

Now that you have this data frame, you can mine it endlessly. You may want to sort by Risk.Allele.Frequency and whether you have the risk. Because there are quite a few columns in the element metadata, it’s nice to define a quick-summary subset:

> my.risk <- dm[dm$i.have.risk, ]
> rel.cols <- c(colnames(d), "Disease.Trait", "Risk.Allele.Frequency",
                "p.Value", "i.have.risk", "X95..CI..text.")

> head(my.risk[order(my.risk$Risk.Allele.Frequency), rel.cols], 1)
            rsid chrom position genotype Disease.Trait Risk.Allele.Frequency
  2553 rs2315504 chr17 36300407       AC        Height                  0.01
       p.Value i.have.risk   X95..CI..text.
  2553   8e-06        TRUE [NR] cm increase

This is a rare variant, but the most important next question is, rare in who?

> dm[which(dm$rsid == "rs2315504"), "Initial.Sample.Size"]
[1] 8,842 Korean individuals

So this clearly doesn’t mean much to me. We can use grep to find studies that mention “European”:

> head(my.risk[grep("European", my.risk$Initial.Sample.Size), rel.cols], 30)

One interesting rs ID that popped up in this list of my data is rs10166942, which is lightly linked to migraines (from which I suffer).

Making Graphics with `ggbio`

ggbio is a new-ish (Bioconductor 2.9) package that produces really nice graphics. Let’s plot the location of all SNPs that gwascat tells me my allele is the “risk” allele (again, strange word choice as some “Disease.Traits” are height). gwascat uses hg19, and ggbio doesn’t have ideogram cytobanding and chromosome position information for hg18 bundled with it (yet?) so we’ll need to work with that.

> library(ggbio)
> p <- plotOverview(hg19IdeogramCyto, cytoband=FALSE)

Now, let’s take the gwrngs object and subset by my risk alleles. Notice how these assignment function elementMetadata<- is overloaded here:

(elementMetadata(gwrngs)$my.genotype <- 
   d$genotype[(match(elementMetadata(gwrngs)$SNPs, d$rsid))])

elementMetadata(gwrngs)$my.risk <- with(elementMetadata(gwrngs), 
    mapply(function(risk, mine) {
      risk %in% unlist(strsplit(mine, ""))
    }, gsub("[^\\-]*-([ATCG?])", "\\1", Strongest.SNP.Risk.Allele), my.genotype))

Now to plot these regions:

p + geom_hotregion(gwrngs, aes(color=my.risk))

Git Notes

Sun, 11 Mar 2012 00:00:00 +0000

Git Notes

These are updated by me periodically. I have tried my best to illustrate common use cases, and the motivation for doing things the “Git” way.

Example Set Up

I’ll use this setup scenario frequently. In a suitable scatch repository (i.e. git-sandbox), make a fake remote:

mkdir fake-remote
cd fake-remote
git init --bare
cd ..

Now, clone it, pretending you are two developers:

git clone fake-remote jerry-repo
git clone fake-remote kramer-repo

Let’s assume you’re Jerry and Kramer is another programmer in your group. As Jerry, let’s make some changes:

cd jerry-repo
echo "an example file" > file.txt
git add file.txt
git commit -am "initial import"
git push origin master
cd ..

Now, let’s pretend we’re Kramer and grab that recent commit:

cd kramer-repo
git pull origin master
cd ..

Git Remote Tracking Branches

Git remote tracking branches are similar to local branches (i.e. the kind you interact with git checkout -b branch-name and see with git branch). However, you don’t work on the remote branch directly, you work on a local branch that’s tracking this remote branch. For example, the most common workflow is to track a remote branch, then push your commits to it or pull commits down from it. Even though it’s a “remote” tracking branch, the branch is stored locally (this branch doesn’t disappear if you can’t connected to the remote).

Git remote tracking branches always have the format remote-repo/remote-branch. After cloning a repository, you can set it to track a remote tracking branch with the -u option of git push, e.g. git push -u origin master. From now on, you can just use git push when on this branch; this branch is tracking origin’s master branch.

git branch shows local branches; to see remote branches use git branch -r, and to see all branches, use git branch -a.

Remote tracking branches are also what determine what is pulled/pushed when using git pull and git push without a remote repository and refspec (i.e. git push origin master).

If the current branch is new-feature, which tracks origin/new-feature, then any branches checked out from new-feature will also track the remote too, unless --no-track is added.

Git Fetch and Merge vs. Git Pull

Recall the above point that remote tracking branches are local, so (unlike Subversion) they function even when you’re not able to connect to the remote. This gives an example of how elegant Git is: being so similar to regular branches, Git remote tracking branches can be merged into local branches. This is precisely what goes on behind the scenes with git pull. Here’s an example. First, let’s set it up such that a developer in your group, Kramer, made some changes, committed them, then pushed them to the remote.

# assuming you're in the right directory
cd kramer-repo
echo "kramer adding gibberish" >> file.txt
git commit -am "I added some gibberish"
git push
cd ..

Now, imagine you (Jerry) have made some commits but want to see what the status of the remote looks like. git remote show can be used to see if the local remote tracking branch is out of date.

vinceb@poisson$ git remote show origin
* remote origin
  Fetch URL: /Users/vinceb/Desktop/git-sandbox/fake-remote
  Push  URL: /Users/vinceb/Desktop/git-sandbox/fake-remote
  HEAD branch: master
  Remote branch:
    master tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (local out of date)

So we are out of date! We can see these commits before merging them in with git fetch. git fetch updates your remote tracking branch with the new changes, allowing you to diff branches just as you would with two regular branches.

vinceb@poisson$ git diff master origin/master
diff --git a/file.txt b/file.txt
index 4e850ce..34ceb34 100644
--- a/file.txt
+++ b/file.txt
@@ -1 +1,2 @@
 an example file

git log origin/master master also works. git log origin/master ^master shows us just the new commits. We could really explore these commits by checkout out the remote tracking branch (but consider this to be “taking a visit”; don’t commit anything). Suppose we did, and we decide we want to merge them with our current branch. For this, we just use git merge: remember remote tracking branches are just branches!

vinceb@poisson$ git branch ## always check what branch you're on!
* master
vinceb@poisson$ git merge origin/master
Updating ee922a9..8c8c240
Fast-forward
 file.txt |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

Note that git pull basically is git fetch && git merge.

Great resources

The Beauty of Bioconductor

Thu, 08 Mar 2012 00:00:00 +0000

In talking with bioinformaticians, biologists, and other researchers, I’ve seen some worrying trends in computation in the sciences. I plan on writing about these extensively in the future, as I believe computation in the sciences will not scale well to face the huge wealth of data coming experiments will provide. This is not due to algorithmic or hardware limitations, but rather to the fact that scientific programmers simply do not have the same standards and practices that the software industry does.

Three events are simultaneously occurring that could endanger the validity of scientific conclusions in the future. First, new technology is providing the average scientist with more data than ever before. Genomics is the prime example of this: the average biologist can now sequence multiple samples simultaneously whereas this would be prohibitively expensive just a few years earlier. As metabolomic and proteomic data are increasingly incorporated into research alongside genomic data, the work done by bioinformaticians will increase significantly. More lines of code to write and more data to process under deadlines will doubtlessly lead to mistakes.

The second contributing factor is that more researchers are writing code and analyzing their own data rather hiring a bioinformatician or statistician. It’s an awesome and commendable occurrence, but sadly academic institutions don’t adequately prepare researchers to code to high standards. Also, in many cases these researchers learn to program by analyzing their own experimental data, rather than example or “toy” data. This makes “silent” mistakes (i.e. those that don’t prompt an error, but lead to incorrect results) impossible to discover as the actual results are not known.

The last contributing factor is that there’s not a strong expectation that coding standards and software engineering practices be upheld in the sciences. There’s a strong cowboy coding culture in scientific programming. In this mindset, the coding is done when the data is processed, not when the data is processed, the code documented, the unit tests passed, the code checked into a repository, etc. The scientific community needs to embrace the idea that proper data analysis takes time: perhaps as long or longer than gathering experimental data.

In future essays I’ll talk more about these issues in depth. This stuff honestly keeps me (and other people I know) awake at night. I worry humanity may face a Thalidomide-like event in the future due to an error in scientific programming.

However, here I want to commend a project that I feel is underutilized in the biology and bioinformatics communities: Bioconductor. It’s worthy of praise as both an example of, and tool to aid in excellency in bioinformatics programming. I’ll focus primarily on its capacity for handling high throughput sequencing data (even though it handles data from other assays very well too).

Where is Bioinformatics?

Currently, many bioinformatics analyses go something like this: experimental data is received, and then a bioinformatician downloads vast amounts of other data relating to the experiment from web resource such as the UCSC Genome Browser, Ensembl, Phytozome, etc. Often this includes genome assemblies, transcript sequences, and annotation data. Then, application code (alignment software, assemblers, SNP finding tools, etc) are downloaded and compiled. These tools are used alongside custom code written that combines downloaded data with the experimental data, and this produces results that are interpreted. Intermediate results may be fed into other online tools and databases like DAVID or Reactome .

However, this is a bad model if one wants the analysis to be reproducible. The common weakness is that web resources can be unstable. It’s then necessary for the researcher to record software and data versions manually. Even if the researcher dutifully complies, outside databases and code repositories may disappear and leave the project unable to be reproduced. Researchers truly invested in conducting reproducible research then have to store data and software versions themselves, which given the scale of genomic data is quite a burden.

Thus, three things currently perplex reproducible research in bioinformatics: the scale of both experimental and other required data prevents easy self-archival, the fast-paced development of bioinformatics tools could lead to differing results across versions, and the overwhelming prevalence of web-based data resources and applications which are not easily reproducible.

The Bioconductor Solution

Bioconductor has, in my opinion, the best solution to these problems. First, Bioconductor stores past versions of its packages back to their earliest releases. Past experiments can be replicated using the exact version of software that was used for the actual analysis.

Second, Bioconductor stores data as packages. Pre-packaged versioned data is a cornerstone of reproducible research. For example, suppose I am working with human RNA-seq data. This requires transcript annotation data, which could be downloaded from an online resource. To be reproducible, this requires that:

The webhost be up indefinitely.
The URL remain stable and point to the exact same resource.
The user provide not only a URL but the version of data/software downloaded.
That the external resource provider (i.e. database or application developer) actually update their versions accordingly.

For absolute best practice, it’s also necessary to MD5 checksum the data and record this value to maintain any data gathered from the same source is the exact same.

In contrast, consider how I would load human transcript data into R with Bioconductor:

library(TxDb.Hsapiens.UCSC.hg19.knownGene)

The versioning here is done explicitly in the package name: hg19. I could also easily record the state of all my Bioconductor packages and my session with sessionInfo():

> sessionInfo()
  
  R version 2.14.1 (2011-12-22)
  Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
  
  locale:
  [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
  
  attached base packages:
  [1] stats     graphics  grDevices utils     datasets  methods   base     
  
  loaded via a namespace (and not attached):
   [1] annotate_1.32.2       AnnotationDbi_1.17.23 Biobase_2.14.0       
   [4] BiocGenerics_0.1.12   DBI_0.2-5             DESeq_1.6.1          
   [7] genefilter_1.36.0     geneplotter_1.32.1    grid_2.14.1          
   [10] IRanges_1.12.6        RColorBrewer_1.0-5    RSQLite_0.11.1       
   [13] splines_2.14.1        survival_2.36-12      xtable_1.7-0

Entire genomes are also packaged via the BSgenome package (BS refers here to Biostrings ). If the data in packages is not sufficiently recent, the GenomicFeatures package provides a programmatic way of downloading, packaging, and using data from BioMart and UCSC Genome Browser tracks, and provides functions for saving and loading transcriptDb objects from such resources. Recently Duncan Temple Lang and I were speaking about reproducible research, and he said “people adopt best practices when they’re right in front of their face”. Bioconductor’s tools do just that. Furthermore, Bioconductor has strict coding and documentation standards (much stricter than CRAN actually), which ensures user-contributed packages are high quality.

Information leakage and statistics at every level

When discussing R and Bioconductor with other researchers, it’s easy to convince them to adopt both for analyzing statistical data - the data that comes in the very final stages of a bioinformatics analysis. It’s usually much more difficult to convince them to consider working with high throughput sequencing data in Bioconductor. Folks complain that it’s (1) not worth it to process sequencing data with Bioconductor tools or (2) it’s not fast enough. I’ll address the second point in a bit; more importantly I want to emphasize that it’s absolutely worth it to process sequencing data in Bioconductor.

In analyzing genomic data, we take very, very, very high dimension data and try to condense it into biologically meaningful conclusions without being misleading or getting something wrong. Every step is about taking dense data and making it understandable: we take sequence reads and try to assemble them into larger contigs and scaffolds, we take cDNA reads and try to map them back to genomes to understand expression, etc. At each step, our tools make heuristic or statistical choices for us. Pipelines woefully ignore these choices because in most cases, after a step is completed, a script jumps to the next step.

When I think about these steps, I try to assess what I think of as “informational leakage” in bioinformatics processing. Each step summarizes something, hopefully in a way without bias or too much noise. Informational leakage is the information that’s lost between steps. Catastrophic information leakage occurs when we lose information that could have indicated whether the data is biased or incorrect. We can hedge the risk of information leakage when we use summary statistics between steps that try to capture this leaked information.

Consider processing RNA-seq reads. The first step is usually quality control, i.e. removing adapter sequences and trimming off poor quality bases. Failing to gather summary statistics before and after each of these steps leads to potentially catastrophic information leakage. Suppose that an experiment with control and treatment groups, sequenced on two different lanes (bad experiment design!). If one lane has systematically lower 3’-end quality than the other, quality trimming software will trim these bases off and lead one experimental group to have much shorter sequences than the other. The mapping rates will differ significantly, as shorter reads may map less uniquely. In the end, our data is completely confounded not only by the lane (and bad experimental design), but by our own tools! If these tools are being run in a pipeline without intermediate summary statistics being gathered, this catastrophic information leakage will go unnoticed.

Loading sequencing data into R and using Bioconductor’s tools earlier allows summary statistics to be gathered earlier and more easily (R is, after all, great for statistics and visualization), which I strongly believe will decrease the risk of catastrophic information leakage in genomics data analysis. This is why I wrote qrqc , which can provide quick summary statistics on sequencing read quality. Used before and after the application of quality tools, qrqc can provide information not only on the state of the data, but also the effect of the tools.

With existing Bioconductor packages, many useful statistics can be gathered on whole reads, BAM mapping results, VCF files, etc.

Massive Power

The complaint that R is slow, and couldn’t possibly be used with sequencing/mapping-level data is unwarranted. In reality, some of Bioconductor’s core packages for working with high throughput sequencing data such as Biostrings and IRanges (the foundation of GenomicRanges) are astoundingly fast because most of their backends are written in C. Biostring actually uses external pointers to C structures and bit patterns to encode biological string data efficiently.

In addition to being fast, they’re also clever. Biostrings implements an abstraction called a “view” on an XString object, which efficiently represents multiple sections of the same string object (such as subsequences of interest). While I wouldn’t write a short read aligner or assembler entirely in R, many bioinformatic tasks are more than sufficiently fast with Bioconductor tools.

Conclusion

In evangelizing Bioconductor, I have two goals. First, I want to spread awareness that it’s the best way to do reproducible bioinformatics that I think exists today. I want more people to use it just because I deeply care about the state of science and reproducibility. Second, I want to build excitement about this project so that more people will contribute. I believe that far too many high quality bioinformatics tools are written outside of Bioconductor. Packaging bioinformatics tools in Bioconductor forces the developer to adopt strict standards, write clear documentation, and open up a program to a large, active user base. Any results from a package’s methods can then easily be evaluated using R, CRAN, or Bioconductor’s existing tools.

I also believe that large programs (like BLAST, and maybe assemblers) should provide better interfaces to R, to prevent information leakage in analysis. I’m willing to bet that a large majority of bioinformatics tools could be outputting more statistics than they currently do that could be valuable in assessing their functionality. R interfaces to these bioinformatics tools will drastically make it easier for biologists and bioinformaticians to prevent information leakage.

Thoughts on Julia and R

Wed, 07 Mar 2012 00:00:00 +0000

Hello, Julia

Julia is an exciting new technical computing language. It’s still in its infancy, but it’s fast (see below), and already does a lot.

There’s been some excitement on Twitter about Julia. Excitement combined with open source often yields development, which then leads to further excitement, until a mature open source project arises. One of Julia’s explicit goals is to challenge other statistical computing environments, including R.

What’s wrong with R?

R is, without a doubt, changing the world. It’s being used by industry giants like Facebook and Google, while also providing academic researchers in statistics, biology, psychology, and countless other fields with not only a free and open source statistical environment, but a huge number of user-contributed package through CRAN. Now methods papers in many fields are often accompanied by CRAN or Bioconductor packages. It’s also a brilliant platform for reproducible, open research, as Bioconductor beautifully illustrates with packaged and version-controlled genomes, microarray probesets, etc.

However, R is suffering from growing pains. For example, there are now 64-bit versions of R, however, vector indexing is still limited by R_len_t (see definition in src/include/Rinternals.h):

/* type for length of vectors etc */
typedef int R_len_t; /* will be long later, LONG64 or ssize_t on Win64 */
#define R_LEN_T_MAX INT_MAX

It appears that one can simply change this to a long and recompile to increase the longest possible addressable vector, but no. Take a look at R_euclidean in library/stats/src/distance.c for an example why: almost all variables for iterating over elements in vectors are defined as integers and don’t use this type. One would have to read through every function, and every line of code to fix this.

R_len_t is just one example. Another issue is that R has been slow to adopt new compiler technologies (i.e. JIT, optional type indications, etc). R almost always gains speed from pushing stuff to C (the recent bytecode compiler is an exception). This isn’t a problem, but it’s a huge limitation to require developers to not only know R, but also C, and also how to interface the two. More modern languages (Java, as well as Python and Julia come to mind) spend more time tracking compiler technology developments and implementing them than R core does (again, Luke Tierney and the bytecode compiler are exceptions). It’s still sometimes efficient to use C with these languages (consider Cython ), but developers in these language aren’t cracking open Kernighan and Ritchie everytime they need to have a for loop do something quickly.

Another gripe I have is that R language development is somewhat closed. Despite a quickly expanding user base, the number of R core contributors is not increasing. I find it hard to believe this is due to lack of interest. It seems much more likely this is due to institutional reasons that need to be changed. The nice thing about language development that it’s really hard, so opening up R to more contributors won’t likely flood the existing core with bad ideas and patches. Personally I would dedicate much more time profiling, reading the source, and working on the R language if it were more open.

The last gripe I have is that R is fragmented. Consider Python:

import re
re.search(r'R-([\d]+).([\d]+)', "R-2.15").groups()

Now, consider R:

gsub("R-([\\d]+)\\.([\\d]+)", "\\1", "R-2.15")

# or

library(stringr)
str_match("R-2.15", "R-([0-9]+)\\.([0-9]+)")

Now, Python also has PyPI’s re2 , but most developers are using re. The motivation behind stringr is that R’s currently family of string processing functions are horribly inconsistent:

# (my ... to avoid writing all parameters)
grep(pattern, x, ...)
regexpr(pattern, text, ...)
gsub(pattern, replacement, x, ...)
strsplit(x, split, ...)

But rather than deprecate these and move forward, we now have two sets of string processing functions. Both are being used . I’m not saying Hadley Wickham is to blame here; quite the contrary, he’s trying to fix a very annoying problem in the language and should be commended. I think the community needs to be more open; for example, before writing a package that processes strings, let’s discuss an implementation plan, deprecating old functions, etc. If not, in the future R will be highly fragmented, and end up with five different object orientation systems… oh, wait.

What would it take to “challenge” R?

Contributors to Julia are optimistic they can challenge R based on a solid foundation of JIT compiling, parallelism, and nice language semantics. I salute this optimism, but I think we need to realistically consider what it would take to “challenge” R.

First, we would need to build an equal statistical computing environment. Consider moving all of stats, MASS, graphics, grid, etc to Julia. Is Julia sufficiently faster than R will be in the time it takes to port these base packages? Remember, R is a moving target; despite my few earlier gripes, R will evolve and get faster. Now, consider adding the extremely popular CRAN packages like ggplot2 and lattice to Julia. In the time it takes to port these, is Julia still sufficiently faster than R will be?

Suppose it is still faster than R. What about after we port the rest of CRAN, and all of Bioconductor to Julia? My point isn’t say that it’s unimaginable that Julia will surpass R. It’s that developers should really dissect what makes a successful language successful before they try to challenge it. I don’t have a horse in this race; I would love to see Julia surpass R. But if all developers want is a fast environment to analyze large data sets using a wealth of methods and libraries, it may be a lot easier to make R faster than to develop a new fast language and hope/wait/beg the community to move over.

Update: Julia core developer Jeff Bezanson sent me a very kind email on March 9th, 20012 about this post. In it, he said the “challenge R” statement was made by a community member and is in no way the mission of the Julia language. He had many kind words to say about the R langauge and its statistical functionality.

The Unbelievable Debate: Some Ramblings on Machine Learning in Science

Sat, 03 Mar 2012 00:00:00 +0000

In between refactoring some qrqc code this morning and looking at RNA-seq data, I grabbed some cold brew coffee and caught up on some missed tweets. Admittedly, my brain glosses over most tweets, but this tweet from Drew Conway had the right mix of keywords to actually make me click and read the link:

The data science debate: domain expertise or machine learning? by @medriscoll http://bit.ly/zr17Z2

I don’t mean for this title to be inflammatory, but I do believe this debate is a bit unbelievable. Machine learning is magical; I imagine that everyone that has studied it goes through a hype cycle -like set of epiphanies. This is my hype cycle story, and why I believe machine learners need to calm down, collaborate with domain experts, and together tackle hard problems.

Social Sciences and Machine Learning Caution

Biologists, it’s true: I’m not one of you. I’m a transplant from the social sciences. Specifically, from political science and economics (with statistics too), where my interests lie in methodology and comparative politics.

In the social sciences, the dimensionality of most problems is small enough that data mining is (at least in my experience) frowned upon. A lot of political data is collected by hand, often by undergraduates toiling away for meager pay as they try to understand some cryptic event coding protocol. There are some very large p data sets: The Political Instability Task Force’s data set is something I’ll keep mentioning. Mining this data with algorithms looking for interesting relationships was exactly how I was taught not to do political science.

I recall one story of a candidate giving a job talk mentioning he used forward step-wise regression to find interesting variables (in a presumably small p data set) and three people immediately stood up and left. I was proud to be knowledgeable of, but avoid statistical learning techniques. Political science had flirted with neural networks to understand massive state failure data sets, but my endless gripe was there these were predictive, not causal models. The latter required some a priori testable theory, often derived from an intimate knowledge of political crisis in a variety of countries. Just as I thought biologists knew c. elegans or s. cerevisiae well enough to form interesting experiment ideas, political scientists knew many political crises well enough to form theories and test them on a larger set of data in a quantitatively rigorous fashion. I also believed that predictive models of state failure may predict recorded (even when out-sample!) state failures well, but a model backed in a good theory that fits existing data slightly less well could predict unseen cases even better (Bruce Bueno de Mesquita has an entire wonderful book about game theory being such a model ).

The Machine Learning Awakening

When I made the jump to analyzing gene expression data, I was initially astounded at how many algorithms were thrown at it. I had this vision of the hard sciences having randomization and experimentation at their disposal to lead to the purest causal findings. Looking for any differences in 30,000 genes’ expression values and then forming hypotheses after seemed backwards. Microarrays shocked biology with what they revealed about cancer and the cell, but they probably shocked the methods of experimental biology more. If your average biologist had a tenuous knowledge of p-values to begin with, now microarrays analysts were throwing around false discovery rates, empirical Bayesian techniques, Storey’s q-value, etc.

However, as I analyzed more and more sets of data, the initial reluctance I had about employing machine learning algorithms disappeared. In hype cycle terms, I was climbing the peak of inflated expectations. A quote from Michael E. Driscoll’s article captures this excitement:

Claudia Perlich, a three-time winner of the KDD Nuggets competition, stood up and shared how she had won contests in domains as varied as “breast cancer, movie prediction, and sales performance - and I can tell you I knew next to nothing about those things when I started.”

This optimism is abundant, and not entirely without justification. Coming from a non-biological background yet thoroughly understanding machine learning provides an immensely satisfying feeling of understanding of the cell. Employing all sorts of machine learning techniques, I could find “biologically interesting” genes in data sets and help biologists understand the cell.

A Few Epiphanies and a Dip of Disillusionment

The Hype Cycle’s lowest stage is the “trough of disillusionment”. Machine learning in biology certainly hasn’t had its trough (and I don’t think it will), but it is priming up to have its “slope of enlightenment” and “plateau of productivity”. There will be future machine learning hype cycles in biology, especially as multiple heterogeneous data sets need to be simultaneously mined to understand the cell with the systems approach.

My personal dip didn’t happen because machine learning left me with a particularly terrible result - it occurred because (1) because of an interaction I had with an experimental biologist and (2) I realized how wonderfully complex the cell is.

Let’s Put That in This

The first interaction I had was with a graduate student friend of mine. We were discussing an interesting finding the Korf Lab made: that some introns lead to increased expression (paper here ). Introns traditionally haven’t had the same attention as promoters of enhancers in regulating gene expression. A member of the Korf lab had previously mentioned intron-mediated expression in passing to me, and I immediately started imagining what ways I could look for such an effect in silico. As I understood it, in silico was how it was first discovered, further increasing my admiration of algorithms applied to biology. When my friend mentioned it again, the first thing she said was, “well, we just need to take that intron and put it in something”.

I immediately agreed, but I realized something: I hadn’t thought of that simple step the first time I thought about intron-mediated expression. Machine learning can bring so much wealth in finding interesting relationships that my mind had glossed over the most important question in science: whether these relationships were spurious or causal. This is why my training in the social sciences was rigidly anti-machine learning: it’s far too easy to let our thought processes about understanding a complex system be biased by some spurious relationships machine learning and predictive models can quickly give us.

The Complexity of the Cell

The epiphany was gradual (and still occurring): the cell is wonderfully complex, or as my mind puts it “fucking awesomely complex”. Machine learning applied to gene expression data gives valuable insights into a complex system, but it’s really a messy snapshot. I think we’ll look at current pristine RNA-seq experiments in twenty years and we’ll realize they’re giving us an image into cellular activity that is akin to a photograph from a cheap Soviet-era camera.

Measuring gene expression from many cells glosses over interesting variation in each cell; this is certainly not a new complaint. However, even a single cell image is messy: mRNAs that make their way into gene expression values may have never been exported from the nucleus, they could have been degraded by the cell, silenced, undergone post translational modification, etc. What’s astounding is that these systems are not just complex, but are amazingly accurate. Cellular data is messy, but the cell certainly isn’t. Development is a prime example of how tightly regulated these processes are. It’s up to us to understand these tightly regulated systems with the messy images scientific data gives us. Machine learning is a necessary, but not sufficient tool to help us understand the cell.

As an example, genes interact in groups, and many algorithms can gloss over this detail. If an algorithm tries to find a sparse set of genes that are biologically interesting to the problem at hand, it may be indifferent to which it includes from a set of co-expressed genes (consider the lasso against the elastic net here). If a biologist reviews these findings, they could easily miss something vastly important based on machine learning’s indifference.

Let’s Use Both.

These epiphanies are now what guides my path through biology and machine learning. I still love and am infatuated with machine learning (although, I much prefer the phrase statistical learning). However, if we wish to understand a complex system, we need to take the approach that modern biology does: leverage machine learning with a priori biological expert knowledge to bootstrap findings. We need to design experiments that also harness the power of machine learning to help us understand, and not just predict the behavior of complex systems. Applied machine learners need to realize the power of experimental data. Chances are if you’re finding everything you think is out there with just machine learning, you’re making a mistake or your problem is too simple.

Elucidating k-mer Contamination with Kullback-Leibler Divergence

Thu, 01 Mar 2012 00:00:00 +0000

Recently a coworker showed me a FASTQ file from an Illumina HiSeq run (which will be packaged in the new release of my Bioconductor package qrqc ) that was severely contaminated. Below is the file in less with a string highlighted:

Holy contamination, Batman! There are a few approaches to handling this level of contamination. The program tagdust will match contaminated reads and remove them. My program Scythe is being changed so that it can match adapter contaminants further in the read using its Bayesian model. Both programs require a priori knowledge of possible contamiant sequences - what if this is a novel sequence contaminant? In this case, AAGCAGTGGTATCAACGCAGAGT appears to be a PCR primer related to DSN normalization that may not have made it into our adapter files.

k-mer Entropy Approaches

k-mer approaches are a nice way of searching for such contaminants. I am currently adding this feature to qrqc; you can follow development on the kmer branch on Github but this branch may merge into master and disappear.

The C functions I’ve written use Heng Li’s khash.h library to quickly hash k-mer sequences with their positions in the read. The end result is a data frame in R of k-mer sequence, position, and counts in that position.

Looking at raw k-mer counts is somewhat useful, but I’ve been exploring some information theoretical approaches to analyzing this data. One useful graphic is entropy of k-mers by position:

These are 6-mers, so there are 4,096 possible k-mers (excluding N). If the k-mer distribution were uniform, 12 bits would be needed to encode each k-mer. This graph illustrates that even at the most random 3’-end of the read, only about 6.5 bits are needed. In the first 20 bases, the distribution of k-mers is so skewed that less the Shannon entropy is less than four bits.

Kullback-Leibler Divergence Approach

It makes sense biologically that the k-mers don’t have uniform frequency. In the case of read contaminants, the enrichment by position against an empirical k-mer distribution may be as interesting as total k-mer enrichment against a random distribution model.

To assess this, some beta qrqc code pools k-mer counts across position to find an empirical k-mer distribution. Then, the k-mer distribution per position is compared to the pooled distribution using the Kullback-Leibler divergence . K-L divergence is only defined when both distributions sum to 1, the sample spaces are the same, and if $q(i) > 0$ for any $i$ such that $p(i)> 0$.

In essence, the K-L divergence is measuring the average number of bits needed to encode data from P with a code based on the distribution of Q. In the k-mer case, Q is the empirical distribution of k-mers irrespective of k-mer position and P is the position-specific distribution of k-mers. Thus, an enrichment of k-mers at a particular position would lead to more divergence.

A nice feature of ggplot2 is the stacking of the “bar” geom. Since K-L divergences are sums, stacking and setting fill color by k-mer (the terms of the sum) gives us a sense of the total divergence and each k-mer’s effect on the total. There are too many k-mers to plot, so I have some procedures that find a nice subset. Because this is a subset, the K-L total (indicated by bar height) is wrong, but the graphical interpretation is easier. Now, the enrichment by position is clear:

This messy dataset has repeat primer contamination. Note that because we’re plotting a subset of k-mers, there is negative total K-L (not mathematically possible) because we’re leaving out terms in the sum, but the meaning still comes through. Also note that there is k-mer nesting: The first wide peak begins with k-mer TATCAA, then ATCAAC, then TCAACG, etc. This indicates that we could adjust k and find the entire repeated k-mer.

Update

I’ve added faceting of multiple SequenceSummary objects’ KL/k-mer diagnostics. Combined with a random data file, this really illustrated contamination:

This is still in development; follow the code on Github and feel free to contact me and make suggestions.

Please developers, don't be dicks.

Tue, 21 Feb 2012 00:00:00 +0000

Please developers, don’t be dicks.

As the author of a few open source tools, I’ve had my fair share of users seeking help. Emails range from the very useful (bug reports, patches, etc) to the annoying (“can you help guide me through this process”). But never once (that I can remember) have I been a dick (and yes, I’ve wanted to be). It will be tricky to write this without sounding self-righteous, but I hope to make the case that open source developers shouldn’t be dicks in all cases.

We’ve All Been There (WABT)

The first reason to never be a dick is that We’ve All Been There (I’m going to give this the acronym WABT). Even the most voracious and diligent manual readers can suffer from the XY problem . A user comes asking how to do Y, which they think is the solution to X. However, it’s a bad solution to X and they don’t know this. These situations will always lead to frustration: users waste time explaining Y and helpers waste time explaining how to do Y to realize the user wanted X. But this is not the user’s fault; it just takes programming practice to realize Y is not the correct way to do X.

We’ve all had these problems in our early stages as developers. Being a dick in these cases will not help the user grok anything. They’re already frustrated - that’s why they’re asking for your help. Being a dick will cause them to get more frustrated and really not grasp anything. They’re not going to have an “ah ha!” moment when they’re too busy trying to come up with a witty response to your burn on IRC.

PCTM has the same number of letters as RTFM

Please Check The Manual (PCTM) has the same number of letters as Read The Fucking Manual (RTFM). I strongly believe it takes more energy for a developer to be a dick than to be nice. We’ve all had dumb questions that disrupt our workflow, make us angry, etc. But being a dick back does not discourage this behavior. Write some boilerplate text for responding to users’ questions. Make this a FAQ. Then respond, PCTM (Please Check the Manual) and send them the link. If they get needy, tell them open source software doesn’t come with a warranty.

People remember dicks

Someone was once quite rude to me via email (I’ll name him Tom). I had voiced some frustrations with software Tom wrote and he attacked me for these public comments. Now, as an aside, there’s a lot of shitty software out there, and signals about software quality (even noisy signals) are very valuable. Tom on one hand attacked me for saying something negative about his software, and on the other hand asked me to help fix it, emphasizing it was open source software. I agree with this sentiment 100%, however the email was clearly very angry.

I told another developer who I’ll call Jerry about the encounter, and he laughed. Apparently, Tom nagged Jerry about portability issues of Jerry’s software years ago. This is evidence of my first point, WABT. It also shows that developers remember interactions with other developers really well. Since then, I’ve also heard other programmers complaining about interactions with Tom. This is all too bad, as Tom is probably very nice in person and certainly a good programmer.

If you’re a dick, you’re hurting OSS

OSS has seen an explosion in recent years. Biologsts, ecologists, and social scientists that never thought they’d write code are using R to analyze data. Folks frustrated by Windows are installing Ubuntu and asking for help. In the early days of the OSS, usenet, and IRC, it was an acceptable norm to be a dick. Now, it’s not.

OSS benefits from a large user base, but it will have growing pains. Being a dick does not alleviate these pains, it makes them worse. Let’s go back to my story about Tom.

In the second half of Tom’s email (after attacking me), he asked me to help him fix his software. Now, collaboration can be difficult; code style clashes, merges fail, frustration is common. In a small project, you’re really in bed with your collaborators. Now that Tom has sent me the signal he’s nasty in correspondences, do you think I’ll work on this project with him? Hell no. I’d rather fork, fix the problem and encourage others to use my software. Of course this is bad for OSS; consider this passage from Eric S. Raymond’s Jargon File :

Forking is considered a Bad Thing - not merely because it implies a lot of wasted effort in the future, but because forks tend to be accompanied by a great deal of strife and acrimony between the successor groups over issues of legitimacy, succession, and design direction. There is serious social pressure against forking.

Tom’s actions guarantee I will avoid working on his projects at all costs. The two other developers, and anyone else we’ve told will too. In the end, the software loses.

Idolize programmers, not their dickishness

Some abrasive programmers are really gifted. Erik Naggum is regarded as the first Usenet flamer . Theo de Raadt forked NetBSD into what became OpenBSD partially because of issues with other developers. Richard Stallman gave an AMA on reddit a year ago and the most popular question (since deleted) was about a young GNU-lover that was nervous about asking RMS a question and accidentally referred to it as “Linux”, not “GNU/Linux” and RMS ripped him in half.

Now, all of these developers have been dickish and are well-known because they are gifted visionaries. I’m not sure why, but other developers admire this dickishness. But don’t idolize their dickishness, idolize their skill. There are also overwhelmingly nice programmers: John McCarthy , Donald Knuth , and Alan Turing to name a few. Admire their skill and their personality.

Being a dick hurts science

There’s been an explosion of open source software utilization in the sciences. My field, bioinformatics, provides an interesting case study. There are bioinformaticians like myself that write software. Users are divided into other programmer types (other bioinformaticians) and biologists (on average, less knowledgeable of programming). All else equal, biologists and bioinformaticians prefer free, open source software to costly proprietary software.

For these reasons, being helpful and nice to scientific users is really important. For biologists, choosing tools is about getting analysis done quickly and easily. Rude bioinformaticians will quickly increase the cost of using OSS tools, which is already high for many biologists who aren’t experienced with Unix tools and programming. Consequently, science could become less open, something neither group wants.

Vince Buffalo – Vince Buffalo

Why Do Species Get a Thin Slice of π? Revisiting Lewontin's Paradox of Variation

Lewontin and Hubby’s Quest to Measure Genetic Variability

The Evolutionary Theories Killed by Measuring Genetic Variation

The Neutral Theory

Lewontin’s Paradox of Variation

Lewontin’s Paradox and the Hitchhiking Model

The Neutralist View on Lewontin’s Paradox

Recent Work on Lewontin’s Paradox

Estimating Census Population Sizes

The Relationship Between Genetic Diversity and Population Size

Is the Diversity–Population-Size relationship meaningful?

Can Selection Explain Lewontin’s Paradox?

So How Do We Solve Lewontin’s Paradox?

The Genome-wide Signal of Linked Selection in Temporal Data

The Quantitative Genetics View of Linked Selection and Temporal Covariance

Detecting Temporal Covariance in Evolve-and-Resequence Studies

Convergent Correlations

Shifts in Temporal Covariance

Conclusions

The Problem of Detecting Polygenic Selection from Temporal Data

Natural Selection is Rapid

Natural Selection versus Drift

We can study natural selection through its effects on linked sites

The Problem of Polygenic Selection

Understanding Snakemake

Make

Snakemake

Using Expand to Build up all Filesnames and Parameter Combinations

Using Wildcards

Future

A Genealogical Look at Shared Ancestry on the X Chromosome

Genealogies

Genetic Ancestry

X Genealogies

Models for X chromosome recent genetic ancestry

What recent ancestry on the X can tell us

Using Rcpp and C++ to Count Genotypes

MD Tags in BAM Files

Example 1: With Insertion

Example 2: More Complex Insertion

Example 3: Deletions

Example 4: Insertions and Deletions

Approaches

Using Named Pipes and Process Substitution in Bioinformatics

Why We Love Pipes and Unix

Where Pipes Don’t Work

Named Pipes

Process Substitution

Speed

Bioinformatics and Interface Design

The Current Interfacing Practice and its Paradigm

File Formats and Interfacing

Foreign Function Interfaces and Shared Libraries

FFI in Practice

The Importance of Good Tools

21st Century Science and the Need for Open Data and Open Tools

21st Century Science and the Need for Open Data and Open Tools

Capital-Intensive Science and the Openness of Data

The Importance of Open Source Software

Adjusting Our Thinking

My First Recommendation to New Scientific Coders: Learn Visualization

Problems Look Differently When You Can Visualize Quickly

Smart Visualization Over Stupid Hypothesis Testing

Build Tools that Support Visual Output

Ok, How do I Learn Visualization?

Conclusion

Simple Parallel Bioinformatics Pipelines with find, basename, and xargs

Simple Parallel Bioinformatics Pipelines with find, basename, and xargs

Big-Ass Servers and Data Parallelism

Find-Basename-Xargs

Redirecting Output in Parallel and XBF Chaining

Using Bioconductor to Analyze your 23andme Data

23andme Raw Data

Loading Raw Data into R

Where are the SNPs 23andme Genotypes?

A Quick Demonstration of GenomicRanges and Bioconductor Annotation Packages

Looking for Risk Variants using gwascat

Making Graphics with ggbio

Git Notes

Looking for Risk Variants using `gwascat`

Making Graphics with `ggbio`