How Stash & Spotify Run Their Data & ML Pipelines

Learn How Stash Cut Workflow Compute Costs by 67%

The machine learning team at Stash manages the personal finance app’s activity to protect the system from fraud, optimize model execution time and lower compute costs and time to market. They needed to replace their “homegrown” workflow management process with a more efficient, less costly solution.

Trusted By

Graduate Project
Trusted byTrusted by

A Proven Solution for Data and Machine Learning Workflows at Scale

Flyte

When Lyft open-sourced Flyte™, it provided the market with a structured programming and distributed processing platform that it had already used to create highly concurrent, scalable and maintainable workflows. Flyte™ has been serving production-model training and data processing at Lyft for four years.

It has become the company’s preferred platform among teams running high-demand workflows including Pricing, Locations, Estimated Time of Arrivals (ETA), Mapping and Self-Driving. In fact, Flyte™ manages more than 10,000 unique workflows at Lyft, comprising monthly totals of more than 1 million executions, 20 million tasks and 40 million containers.

Kubernetes
Kubernetes-native workflow-automation platform
SDK
Ergonomic SDKs in Python, Java and Scala
Pipelines
Versioned, auditable and reproducible pipelines
Data-aware
Data-aware and strongly typed
Resource-aware
Resource-aware for deployments at scale
Flyte Icon
Why Lyft Created Flyte™
Learn More
Flyte™ is Multilingual

Python, Java and Scala

Flyte™’s primary programming interface is based in plain and simple Python. Configuration is done inline, no YAML is required, and workflows can be tested locally before shipping. Java and Scala SDKs are also available, and powerful programmatic access ships standard.

Flyte™ is Multilingual Code

Monitor your Data and ML Workflows

The Flyte™ Console gives you visibility into your workflows, tasks, schedules, executions, failures, timeouts, retries, and more. Track performance, view logs and DAG failures as you iterate on your model development.

Monitor your data and ML workflows
Flyte™ vs. Kubeflow
Featured Blog

Flyte™ vs. Kubeflow

Flyte™’s an ML orchestrator, and so is Kubeflow; Flyte™ has a Python SDK, and so does Kubeflow Pipelines; Flyte™’s built on Kubernetes, and so is Kubeflow. Look beyond these basics, however, and you’ll find Flyte™ and Kubeflow offer markedly different developer experiences and levels of effort to scale and deploy them. Get to know the differences in this compare-and-contrast article.

Learn More
Finally

Kubernetes Made Easy for Data and ML Engineers

Kubernetes-native Flyte™ workflows are typesafe, directed graphs of independent tasks that are readily visualized and analyzed.

Kubernetes Made Easy for Data and ML Engineers
UnionCloud

Harnessing the Power of Flyte™ without the Overhead

Flyte™ is helping organizations like Spotify and Lyft build a new generation of products that make elegant use of complex data and machine learning. Now Union AI, the team behind Flyte™, has created a managed version of the workflow orchestrator, freeing data and ML teams from infrastructure constraints and setup.

Try Union Cloud

Integrations for Flyte™ and Union Cloud

Integrations for Flyte™ and Union CloudIntegrations for Flyte™ and Union Cloud
Flyte CommunityFlyte Community
Pandera Joins Union.ai
UnionPandera

Open-source, zero-configuration data testing and validation for more readable and robust data processing pipelines.

Learn More
Union ML
UnionML

Unifying the vast ecosystem of ML tools into a single interface.

Learn More

Join the Community

We’re building the production-grade Kubernetes-native ML and data orchestration platform.

If you want to join us in our mission:

Github

Spread the love and help us build awareness of this project!

Star Us On Github
Slack

We’d love to connect and hear your feedback, questions, and ideas!

Join Us On Slack
Contribute

Help us to help you build amazing open source software!

Contribute