This is part 2 of our series on the complexities of the data behind Delphy, our tool for understanding the evolution of pathogens during an outbreak. In part 1, we describe the type of data Delphy generates. If you haven’t already, it’s probably helpful to read that first. As a reminder, the main takeaways are:

• Delphy takes the genetic sequences of a pathogen sampled from infected people during an outbreak, and makes a phylogenetic tree (basically, a family tree of the pathogen).
• The process for this is to generate a lot of equally plausible phylogenetic trees, and then summarize all these individual samples.
• If a feature shows up consistently across all the plausible trees, then it’s likely to be a real thing.
• Since all the trees are equally plausible, it doesn’t make sense to just show one result. Instead, Delphy shows a distribution of results from all the plausible trees.

In this post, we’ll describe how to launch a Delphy run and what that looks like. Which, by the way, is very different from what you get with other tools for Bayesian phylogenetics. They run from the command line, so you might get some readouts on the screen or there might be a log file you can parse to evaluate progress. But since there’s no insight into how it’s progressing, people tend to set the job for a predetermined number of trees, wait hours or days until it’s done, and then take a look at the results. If you happened to have some bad data, that’s a lot of time wasted. In Delphy, the same screen that launches the run has numerous indicators of how the job is progressing. And you can pause the job at any moment to check the data, and restart it if all looks good.

Watching the evolution of the tree

After sequences are uploaded and parsed in Delphy, you are automatically taken to the “Trees” tab.

The Trees tab in Delphy before the run starts. Here you can set parameters for a run by clicking the “Advanced options” in the upper left, start and pause the run, and monitor its progress via trace charts and other indicators on the page.

On the left side, you get to see a first guess of what the evolution of the pathogen looks like. Delphy shows an initial tree, with the root in the upper left, branching towards the right. Each tip where the branches come to an end represents one of the uploaded sequences. Until the user presses the run button (it looks like a play button), we only have this one tree¹.

The initial tree for a Delphy run. The date on the right marks when the last sequence for this run was collected. The layout of the tree is pretty good already, but it will only get better as once the run starts and we begin iterating on it. Those iterations are behind the depth of the data that Delphy generates.

In the app, to the right of the tree, there are charts for various “traces” generated during the run. Each trace is a particular measure of a tree, and the charts track that measure across all the base trees that Delphy generates. For example, since each base tree is a take on the evolution of the pathogen, we get a mutation rate for each one of those trees. The “Mutation Rate μ” trace chart allows us to see all those rates together and compare them across the entire Delphy run. Most of the traces have two charts: a seismograph-like chart showing the values for each base tree (there’s only one value at the start), and below that, a distribution that summarizes the values. They start with only one value each, but they fill up as the run progresses. All together, these traces help evaluate the progress and quality of the run.

Trace charts filling up with data as a Delphy run starts. Offscreen, the user pauses the run and then the mouse comes back to inspect the mutation rate values so far.

There’s another chart besides the trace charts, and it starts off full of useful data. This is the scatter plot in the lower right, labelled “Mutation Count vs. Date”. It compares the date of each uploaded sequence with the number of mutations in that sequence, along with a linear regression. A quick glance at this will help determine whether you have outliers that you may want to remove from your data. For example, in the example below, it’s clear one of these dots is not like the others:

In this scatter plot, most of the data hugs the bottom of the chart, along with the linear regression. One sequence stands apart from all the others, with roughly five times the mutations of the rest. Hovering the mouse over that dot will reveal the id of the sequence, so that you can find it in the input data.

Once the run actually starts, things get much more interesting. Note that it’s way more fun to do it yourself: just go to delphy.bio, select the “Ebola: Gire et al 2014” demo, click the “Run this demo” button, and then click the run button, like this:

A video loop of starting a Delphy run: once the user opens the demo, you see a phylogenetic tree on the left, and the trace charts to the right of it. Once the user clicks the run button, the trace charts fill up with data, and the phylogenetic tree changes shape as new samples are added.

Behind the scenes, Delphy starts generating base trees. Every time a new base tree is sampled, the phylogenetic tree changes shape and each trace chart updates with a new datapoint. Beneath each of seismograph-style charts, the distribution updates to include the new value. In an older version of Delphy, we showed each new tree as it was generated. It helps express how the process works, but it’s not actually that useful, so we took it out to make space for better trace charts.

Looping view from an earlier version of Delphy, displaying each new sampled tree as soon as it is generated. Although each tree is different, you can see that they all have roughly similar shapes.

There’s a general pattern to how the phylogenetic tree on the left updates during a run: it changes dramatically for a bit before it settles down. Recall that the goal of the algorithm is to create a bunch of equally plausible sample trees. The first tree you get is pretty good, but it’s still somewhat random. It can take a bit of the run to reach an equilibrium. This is reflected in the how much the phylogenetic tree changes with each sample: as the run approaches an equilibrium, the phylogenetic tree becomes more stable ².

As we get more and more of the sample trees, we want to see what parts of the tree show up consistently. We use a “Maximum Clade Credibility” tree ^3,4, one of the conventional ways to show those common features for data like this. Basically, every branching point is evaluated to see how many of the base trees it appears in. The ones that appear in 90% or more⁵ of the base trees are colored darker.

The MCC, with the high confidence branches drawn in black. Those are the branching points that we can be pretty sure correspond to the evolution of the pathogen in the real world.

With Delphy, our goal was to create transparency into all the data behind the phylogenetic tree. During the run phase of Delphy, having visibility into all the traces can help experts get fine grained detail on the run’s progress. For non-experts, we also have a simpler way to tell how the run is progressing. In the top right of the tab, there’s a progress meter indicating how reliable or robust the run is so far. This summarizes the trace data and tells you whether it’s too soon to start exploring, good enough to vet your data, robust for exploration, or so stable that it’s really unlikely to change any more.

The quick read of a run's progress. The Minimum effective sample size is a statistical measure of whether there's enough variation in the base trees (which is not the same as the number of trees). That’s linked to the status on the right: under 15, there's not enough data to draw any conclusions. Other thresholds are at 15 and 200, and once it reaches 200, the run has gone long enough that it's unlikely to change any more.

Our recommendation is to pause the run once the results are stable. Then open up the other tabs to take an initial look at the results. This can highlight bad data in the sequences that were uploaded to Delphy. If you find some, it could be worthwhile to remove that sequence and start another run–and because Delphy is so quick, that’s not a costly thing to do. But if everything looks good, then you can go back to the trees tab, unpause the run, and wait a few more minutes for the results to become robust. We’re hoping that having such a quick and easy tool will encourage more people to explore phylogenetic data on their own. In the next part of the series, we’ll walk through the sorts of analysis you can do in Delphy.

Notes

1. We don’t start generating new trees right away, as the user may want to set some different parameters for the run.
2. Note that the tree will continue to update sporadically as the run continues.
3. “Clade” is the technical term for a tree or subtree, and when we look for common features across trees here, we are looking for clades. For the purposes of this algorithm, Delphy identifies clades by all the tips / sequences that are descendants of a branching point–the details of the branching structure don’t matter, just the tips. To build the Maximum Clade Credibility tree, Delphy 1) identifies all the clades in all the base trees, 2) scores each clade according to how many base trees it appears in 3) scores each base tree according to the clades it has 4) takes the structure of the highest scoring tree (the one with the highest clade credibility) 5) adjusts the branch length for each clade in the tree according to the average time of the clade across all the base trees it shows up in.
4. There are other kinds of summary trees, and we’re looking forward to incorporating them into Delphy in the future.
5. You can adjust that percentage by opening up the Tree Settings below the tree.