Airflow connector
Apache Airflow is a widely used open source platform for managing workflows with a robust ecosystem. Union.ai provides an Airflow plugin that allows you to run Airflow tasks as Union.ai tasks. This allows you to use the Airflow plugin ecosystem in conjunction with Union.ai’s powerful task execution and orchestration capabilities.
The Airflow connector does not support all Airflow operators. We have tested many, but if you run into issues, please file a bug report.
Installation
To install the plugin, run the following command:
pip install flytekitplugins-airflow
This plugin has two components:
- Airflow compiler: This component compiles Airflow tasks to Union.ai tasks, so Airflow tasks can be directly used inside the Union.ai workflow.
- Airflow connector: This component allows you to execute Airflow tasks either locally or on a Union.ai cluster.
You don’t need an Airflow cluster to run Airflow tasks, since flytekit will automatically compile Airflow tasks to Union.ai tasks and execute them on the Airflow connector.
Example usage
Local testing
Airflow doesn’t support local execution natively. However, Union.ai compiles Airflow tasks to Union.ai tasks, which enables you to test Airflow tasks locally in flytekit’s local execution mode.
In some cases, you will need to store credentials in your local environment when testing locally.