Integrations

Flyte is designed to be highly extensible and can be customized in multiple ways.

Connectors

Connectors are long-running, stateless services that receive execution requests via gRPC and initiate jobs with appropriate external or internal services. Each connector service is a Kubernetes deployment that receives gRPC requests from FlytePropeller when users trigger a particular type of task. (For example, the BigQuery connector handles BigQuery tasks.) The connector service then initiates a job with the appropriate service.

Flytekit plugins

Flytekit plugins can be implemented purely in Python, unit tested locally, and allow extending Flytekit functionality. For comparison, these plugins can be thought of like Airflow operators.

Using Flytekit plugins

Data is automatically marshalled and unmarshalled in and out of the plugin. Users should mostly implement the flytekit.core.base-task.PythonTask API defined in Flytekit.

Flytekit plugins are lazily loaded and can be released independently like libraries. The naming convention is flytekitplugins-*, where * indicates the package to be integrated into Flytekit. For example, flytekitplugins-papermill enables users to author Flytekit tasks using Papermill.

You can find the plugins maintained by the core Flyte team here.

Native backend plugins

Native backend plugins can be executed without any external service dependencies because the compute is orchestrated by Flyte itself, within its provisioned Kubernetes clusters.

External service backend plugins

As the term suggests, these plugins rely on external services to handle the workload defined in the Flyte task that uses the plugin.

Enabling backend plugins

To enable a backend plugin, you must add the ID of the plugin to the enabled plugins list. The enabled-plugins is available under the tasks > task-plugins section of FlytePropeller’s configuration. The plugin configuration structure is defined here. An example of the config follows:

tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array

Finding the ID of the backend plugin

To find the ID of the backend plugin, look at the source code of the plugin. For examples, in the case of Spark, the value of ID is used here, defined as spark.

SDKs for writing tasks and workflows

The Flyte community would love to help you build new SDKs. Currently, the available SDKs are:

Flyte operators

Flyte can be integrated with other orchestrators to help you leverage Flyte’s constructs natively within other orchestration tools.