Prima Mente scales DNA foundation-model pipelines past 250,000 tasks with pure Python orchestration

Prima Mente is an AI biology company building general-purpose biological foundation models, starting with the human brain. The company generates its own data, trains multi-modal foundation models on enormous volumes of molecular biology data, and translates the results into downstream research and clinical outcomes — from gene- and cell-level prediction tasks to diagnostics for neurological diseases like Alzheimer’s and Parkinson’s. Its team of AI researchers, experimentalists, clinicians, and operators works across London, San Francisco, and UAE.

Every model the company trains depends on a preprocessing layer that reprocesses raw sequencing data in-house. Each model needs its own pipeline and its own scaling profile, and the volumes are substantial: Prima Mente has roughly 5 petabytes of data to move through these pipelines before the resulting datasets are handed off to a SLURM cluster on Nebius for foundation-model pre-training.

As Prima Mente added more models and more data, these pipelines grew quickly in number and size. The team needed an orchestration layer built for that scale, which led them to Union.ai.

As the team pushed for scale, a few requirements became clear:

• Room to scale. The team regularly needed to fan out to hundreds of thousands of tasks in a single run, and wanted an orchestrator that could handle that without workarounds.
• A central record of every run. With pipelines submitted by several people across two offices, they needed one place to see what had run, with which inputs, and who had launched it.
• Resourcing that takes care of itself. Rather than sizing a dedicated head node for every run, the team wanted resources to scale up automatically when a task needed more.
• Python from end to end. The bioinformatics team works in Python, and so do their coding agents. Keeping workflows in the same language meant better IDE support and faster development.
• Tasks that run on their own. Being able to rerun a single step in isolation, without touching the rest of the workflow, makes debugging and iteration much faster.
• Minimal infrastructure overhead. Orchestration that stays in the background, so the team's time goes into the pipelines themselves.
  

Union offered an orchestrator that could scale to hundreds of thousands of tasks, provide operational visibility and role-based access control, and let the team author everything in plain Python.

“Union has become the backbone of our data processing. Almost all of our bioinformatics pipelines run on it now”
— Jay Ganbat, Principal Bioinformatics Engineer, Prima Mente

After hearing great things about Flyte from other teams, Prima Mente was excited to move its preprocessing onto Union.ai.

A few capabilities did the most work:

Pure Python authoring

Union lets the bioinformatics team write workflows in the language they already use, which also makes the team's coding agents far more effective.

Flyte 2 scale with escalation logic built in

The single most impactful change was Flyte 2. Large workloads run reliably, and the team can scale to 200,000–300,000 pods with automatic resource escalation baked in.

A remote image builder, so no one touches a Dockerfile

Every Prima Mente workflow now uses Union’s remote image builder. The team maintains a base Docker image, and Union layers additional requirements on top at runtime — no rebuilding from scratch, no local image builds. Even for an engineer fully capable of writing Dockerfiles, it removes a recurring chore from daily work.

Operational visibility, RBAC, and true modularity

Union gave the team a centralized control plane it previously didn’t have: a clear record of inputs, outputs, and who ran what, with role-based access scoped to projects and domains. And any single task can be re-run straight from the UI by entering its inputs — no endpoint exports or throwaway files required.

With Flyte 2, the team's largest single run reached 278,000 actions executed across 150,000 reused Pods.

That scale is already doing real work. In the span of a week or two, roughly 500 terabytes of sequencing data moved through the Flyte 2 pipeline, part of the 5 petabytes the team reprocesses in-house to feed foundation-model pre-training.

The change shows up in the day-to-day, too. Prima Mente’s bioinformatics team — two engineers in London and four in San Francisco — works in Union daily, authoring workflows entirely in Python with no infrastructure translation layer in between. And when issues arise, support has kept pace:

“Support-wise it’s been nothing but excellent. Every request from our team gets addressed within a few hours.”
— Jay Ganbat, Principal Bioinformatics Engineer, Prima Mente

Looking ahead

For Prima Mente, the orchestration layer has gone from a recurring source of pain to something the team simply doesn't have to think about. As the team scales toward its next model, it will look to expand Union’s impact beyond preprocessing and data generation: configuring the Nebius data plane, exploring reusable containers and Spark-backed SQL for large-scale data access, and giving Prima Mente’s infrastructure engineer direct visibility into node pools. For a company building foundation models to understand and protect the brain, every cycle reclaimed from infrastructure is a cycle returned to science. With their infrastructure finally matching their ambition, Prima Mente is accelerating what’s possible.