Flyte School: Developing and Productionizing Data and ML Pipelines
In this session, Samhita will guide attendees on building and executing data and ML pipelines with Flyte. The focus will be on preparing pipelines for both development and production environments. The session won't delve into the specifics of defining Flyte workflows but will instead explore features for smoother local development, guiding attendees through various methods to register and run workflows on a Flyte cluster. Additionally, attendees will learn about projects and domains, and the session will end by Samhita sharing insights on transitioning pipelines from development to production, leveraging features such as custom task resources, scheduling, notifications, access to GPUs, etc.
Agenda
- A brief introduction to Flyte tasks and workflows
- How to simplify local development
- How to make pipelines production-ready
- How to transition pipelines from development to production
Learning Objectives
- Simplifying the pipeline development lifecycle
- Building custom images without using a Dockerfile
- Exploring different methods to register Flyte tasks and workflows
- Making data and ML pipelines production-ready
- Understanding how projects and domains facilitate team collaboration and the transition from development to production
Additional Resources
- Check out and star Flyte on GitHub
- Join the vibrant community on Slack
- Check out the Flyte docs