From Flyte to Airflow: Alectio’s ML Workflow Evolution
Teams that choose Airflow for ML workflows share many of the same pain points when they scale or introduce more complex workflows. Issues with local debugging, workflow versioning, task reusability are just a few of them.
Jennifer Prendki, Founder and CEO of Alectio, joined Flyte’s community sync to share some of the challenges they encountered in using Airflow for their MLOps pipeline for data-centric AI — leading them to evaluate other workflow orchestration platforms and ultimately choosing Flyte.
Revolutionize data & ML workflow development with Flyte
Learn how organizations are using Flyte to bring efficiency and innovation to their workflows.
How Pachama collaborates on a unified fabric
Scalable pipelines for Gojek’s ML Platform
“Gojek is experiencing rapid growth and incorporating machine learning into various products. To sustain this growth and guarantee success, a reliable and scalable pipeline solution is critical. Flyte plays a vital role as a key component of Gojek’s ML Platform by providing exactly that.”
Pradithya Aria Pura
Principal Software Engineer at Gojek
Declarative infrastructure powers Embarks compute resources
MethaneSAT leverages abstracted data flow for massive data processing
“We’re going to have 10,000-plus CPUs that we plan to use every day processing the raw data. There’ll be 30 different targets that we’re collecting data on every day, and it’s that’s about 200 gigs of raw data, and then our estimate is for final data, we’re probably like two terabytes or so on the output. So a lot of data to process, and we’re leaning heavily on Flyte™ to make that happen.”
A Senior Platform Software Engineer at MethaneSAT
Freenome uses an extensible plugin system for data and ML workloads
“Flyte is really execution engine agnostic and so like if you wanted to run Spark workloads if you wanted to run pytorch workloads uh or whatever else like you know AWS batch or Google AI platform or something like that like we can basically write a plug-in uh that would work and you know some most of these plugins already exist.”
Staff Software Engineer at Freenome
Infiniome experiences ease of use and reduced friction
“One thing that I really like compared to my previous experience with some of these tools was the local dev experience with pyflyte and the sandbox reduce that friction between production and dev environments so that was really nice for me to see it.”
Principal Data Scientist at Infinome
Discover the blueprint to future-proof your workflows
Harness the power of modern AI orchestration with our comprehensive checklist, which breaks down the advanced features and unprecedented functionality of the Flyte platform.
Move beyond Airflow and Kubeflow with
modern AI orchestration
Flyte vs. Kubeflow Comparison
Airflow vs. Flyte Cheat Sheet
Orchestrating Data Pipelines at Lyft: Comparing Flyte and Airflow
AI orchestration news and resources
Unlock critical ML performance insights and optimize AI workflows with Union Cloud. Identify bottlenecks, accelerate execution times, and streamline resource utilization for smoother data and ml workflow execution.
Fine-Tuning Insights: Lessons from Experimenting with RedPajama Large Language Model on Flyte Slack Data
Declarative infrastructure: The power of unmatched reliability
Building FlyteGPT on Flyte with LangChain
Get started with Union Cloud
Union Cloud is immediately available for AWS and GCP and starts at $1,000 per month. Union offers free trials for qualified users.
To get started with Union Cloud check out the documentation or sign up to be an early user below.