Infrastructure for AI, ML & Data
For developers managing AI, ML, and data workflows in production, the challenges extend well beyond scheduling and orchestrating DAGs. Union.ai addresses these complexities by offering a comprehensive infrastructure management platform designed for the nuances of such environments.
Union optimizes resources across teams and implements cost-effective strategies that can reduce expenses by up to 66%. Moreover, it’s engineered to fit within your own cloud ecosystem, ensuring a robust and tailored infrastructure that scales with your technical demands.
Powerful DAGs, observability & cost-efficient engineering
Union is a fully-managed Flyte platform deployed in your VPC that provides a single-endpoint workflow orchestration and compute service to engineers building data and ML products.
Get built-in dashboards, live-logging, and task-level resource monitoring, enabling users to identify resource bottlenecks and simplifying the debugging process, resulting in optimized infrastructure and faster experimentation.
AI engineering for engineers
Union is an open AI orchestration platform that simplifies AI infrastructure so you can develop, deploy, and innovate faster. Unlike popular—but simple—AI engineering orchestrators, Union wrangles the infrastructure setup and management as well.
Write your code in Python, collaborate across departments, and enjoy full reproducibility and auditability. Union lets you focus on what matters.
Purpose-built for lineage-aware pipeline orchestration
Bring your own Airflow code (BYOAC) and take advantage of modern AI orchestration features—out of the box! Get full reproducibility, audibility, experiment tracking, cross-team task sharing, compile-time error checking, and automatic artifact capture.
Easily experiment and iterate in isolation with versioned tasks and workflows.
A centralized infrastructure for your team and organization, enables multiple users to share the same platform while maintaining their own distinct data and configurations.
Strongly typed inputs and outputs can simplify data validation and highlight incompatibilities between tasks making it easier to identify and troubleshoot errors before launching the workflow.
Caching the output of task executions can accelerate subsequent executions and prevent wasted resources.
As a data-aware platform, it can simplify rollbacks and error tracking.
Immutable executions help ensure reproducibility by preventing any changes to the state of an execution.
Rerun only failed tasks in a workflow to save time, resources, and more easily debug.
Enable human intervention to supervise, tune and test workflows - resulting in improved accuracy and safety.
Globally trusted & tested
Join our developer community
“We got over 66% reduction in orchestration code when we moved to Flyte™ — a huge win!”
“When you write Python scripts, everything runs and takes a certain amount of time, whereas now for free we get parallelism across tasks. Our data scientists think that's really cool.”
“Our contribution velocity and the rate at which we're contributing is a reflection of our confidence in Flyte™ long term as the de facto workflow orchestration engine. I really think Flyte™ has got the model absolutely correct.”
“We’ve migrated about 50% of all training pipelines over to Flyte™ from Kubeflow. In several cases, we saw an 80% reduction in boilerplate between workflows and tasks vs. the Kubeflow pipeline and components. Overall, Flyte™ is a far simpler system to reason about with respect to how the code actually executes, and it’s more self-serve for our research team to handle.”
“We're going to have 10,000-plus CPUs that we plan to use every day to process the raw data. There'll be 30 different targets approximately that we're collecting data on every day. That's about 200 GB of raw data and probably 2 TB or so on the output — a lot of data process. We're leaning heavily on Flyte™ to make that happen.”
“Union Cloud solves our operational complexity problems across diverse workloads, whether it is running data cleaning & pre-processing workflows or protein structure ML predictions for low-volume, high-complexity scientific workloads to large-scale scientific simulations. Additionally, the platform can drive down the relative cost of protein production by orders of magnitude. With Union Cloud as our standardized workflow orchestration platform, we can stop managing our own systems and infrastructure, and instead focus on antibody discovery and development.”
“The multi-tenancy that Flyte™ provides is obviously important in regulated spaces where you need to separate users and resources and things like amongst each other within the same organization.”
“As engineers, a lot of this might be table stakes for us. But for data scientists, being able to get [financial analytics] up and running on Flyte™ and getting all of this stuff for free has been a really big win for them.”
“During our evaluation stage, we did some stress tests to understand whether Flyte™ can satisfy our requirements, and it provided us with a good result.”