Airflow connector
Apache Airflow is a widely used open source platform for managing workflows with a robust ecosystem. Flyte provides an Airflow plugin that allows you to run Airflow tasks as Flyte tasks. This allows you to use the Airflow plugin ecosystem in conjunction with Flyte’s powerful task execution and orchestration capabilities.
The Airflow connector does not support all Airflow operators. We have tested many, but if you run into issues, please file a bug report.
Installation
To install the plugin, run the following command:
pip install flytekitplugins-airflow
This plugin has two components:
- Airflow compiler: This component compiles Airflow tasks to Flyte tasks, so Airflow tasks can be directly used inside the Flyte workflow.
- Airflow connector: This component allows you to execute Airflow tasks either locally or on a Flyte cluster.
You don’t need an Airflow cluster to run Airflow tasks, since flytekit will automatically compile Airflow tasks to Flyte tasks and execute them on the Airflow connector.
Example usage
Local testing
Airflow doesn’t support local execution natively. However, Flyte compiles Airflow tasks to Flyte tasks, which enables you to test Airflow tasks locally in flytekit’s local execution mode.
In some cases, you will need to store credentials in your local environment when testing locally.
Flyte deployment configuration
To enable the Airflow connector in your Flyte deployment, see the Airflow connector deployment guide.