Use Case:

Union Lightens the Load for Delve Bio

The company

Less than a year after Delve Bio spun out from UC San Francisco, it’s already creating a buzz with an innovative platform to speed diagnosis of infectious diseases. Delve’s test applies metagenomic next-generation sequencing to analyze all the nucleic acids in a single sample of cerebrospinal fluid to sniff out bacteria, fungi, parasites and viruses simultaneously.

Brian O'Donovan, head of bioinformatics and computational biology at Delve, said scaling the technology will speed diagnoses dramatically from the current paradigm, in which clinicians order several tests to screen for different types of infection — often targeting only a single organism or a small panel of pathogens. 

The challenge 

O’Donovan said robust, auditable, workflow management is essential to achieve the precision the process requires. “There's very little nucleic acid in a milliliter of someone's spinal fluid,” he explained. “So the assay itself is very sensitive. … We need to be able to confidently and accurately adjudicate which sequences are derived from the sample and which are potential artifacts from the assay or environment. It’s a typical signal and noise problem.”

When O’Donovan joined Delve as its “first and only bioinformatics scientist and engineer,”  had used Flyte (the workflow orchestration engine behind Union) at his previous company, Freenome. 

“I think bioinformaticians are gravitating towards systems like Flyte,” he said. “If you can already write Python code, implementation is easy! It's very satisfying to just decorate a function and then see it spin up a node.” O’Donovan said the stability of Flyte’s handling of ETL and EDA  simplified working with data, “especially for clinical applications, like we're doing, where traceability and reproducibility are paramount.”

The solution

O’Donovan had “lobbied pretty hard” for orchestration: A formal workflow, he reasoned, would speed onboarding of researchers and reduce technical overhead. 

“We explored launching the workflow as an AWS batch and [tried] other options on GCP,” he said. “But tweaking things like memory requirements in AMI or VM specs creates overhead.”
“After reviewing the available options with our engineering team, including standing up our own Flyte cluster using purely the open source project, we decided that the Union offering was most aligned with our immediate needs and expectations. We met with the Union team several times leading up to the decision and were excited to enter a partnership that included dedicated support and integration of their platform into our existing AWS infrastructure. Throughout the entire process they have been incredibly responsive and insightful, allowing our engineers and scientists to focus more on implementing our core business and clinical logic and less on technical details or overhead relating to our compute cluster.”

Adopting Union relieved Delve of the task of managing infrastructure and Kubernetes. “I'm not gonna lie: Kubernetes is all nuance. If you did a quick assessment of my Slack exchanges with Union dev engineers that help out, they're mostly helping us with Kubernetes questions. That takes a huge load off of our engineering department building systems that integrate with medical systems and billing. They don't need to be managing our ephemeral storage on a node here and there.” 

The results

“One of the things that attracted me the most to Union: Its learning curve’s not terribly steep, but the yield curve is incredible,” O’Donovan said. “Once people get the underlying concept, it's incredibly easy and rewarding 

That ease of use is essential, O’Donovan said, with a team that spans engineers as well as academics with specialized topical knowledge. “I can get a kind of research-to-dev pipeline going where you can show me something in a notebook, if it looks like it's yielding something interesting, we can spend a few hours to turn that into a Flyte task. And if you do that a few times with a green hire, within a month, they'll start producing code that they can register on their own. Eventually, dev habits evolve so even early code is almost immediately amenable to execution on our Union cluster.”