# Integrations
> This bundle contains all pages in the Integrations section.
> Source: https://www.union.ai/docs/v1/flyte/integrations/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations ===

# Integrations

Flyte is designed to be highly extensible and can be customized in multiple ways.

> [!NOTE]
> Want to contribute an integration example? Check out the [contribution guide](https://www.union.ai/docs/v1/flyte/community/contribute/contribute-examples).

## Connectors

Flyte supports [the following connectors out-of-the-box](./connectors/_index).
If you don't see the connector you need below, have a look at **Connectors**.

| Agent | Description |
|-------|-------------|
| [SageMaker connector](./connectors/sagemaker-inference-connector/_index) | Deploy models and create, as well as trigger inference endpoints on AWS SageMaker. |
| [Airflow connector](./connectors/airflow-connector/_index) | Run Airflow jobs in your workflows with the Airflow connector. |
| [BigQuery connector](./connectors/bigquery-connector/_index) | Run BigQuery jobs in your workflows with the BigQuery connector. |
| [ChatGPT connector](./connectors/chatgpt-connector/_index) | Run ChatGPT jobs in your workflows with the ChatGPT connector. |
| [Databricks connector](./connectors/databricks-connector/_index) | Run Databricks jobs in your workflows with the Databricks connector. |
| [Memory Machine Cloud connector](./connectors/mmcloud-connector/_index) | Execute tasks using the MemVerge Memory Machine Cloud connector. |
| [OpenAI Batch connector](./connectors/openai-batch-connector/_index) | Submit requests for asynchronous batch processing on OpenAI. |
| [Perian connector](./connectors/perian-connector/_index) | Execute tasks on Perian Job Platform. |
| [Sensor connector](./connectors/sensor/_index) | Run sensor jobs in your workflows with the sensor connector. |
| [Slurm connector](./connectors/slurm-connector/_index) | Run Slurm jobs in your workflows with the Slurm connector. |
| [Snowflake connector](./connectors//snowflake-connector/_index) | Run Snowflake jobs in your workflows with the Snowflake connector. |

## Flytekit plugins

Flytekit plugins can be implemented purely in Python, unit tested locally, and allow extending
Flytekit functionality. For comparison, these plugins can be thought of like
[Airflow operators](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/index.html).

| Plugin | Description |
|--------|-------------|
| [Comet](./flytekit-plugins/comet-ml-plugin/_index) | `comet-ml`: Comet’s machine learning platform. |
| [DBT](./flytekit-plugins/dbt-plugin/_index) | Run and test your `dbt` pipelines in Flyte. |
| [Dolt](./flytekit-plugins/dolt-plugin/_index) | Version your SQL database with `dolt`. |
| [DuckDB](./flytekit-plugins/duckdb-plugin/_index) | Run analytical queries using DuckDB. |
| [Great Expectations](./flytekit-plugins/greatexpectations-plugin/_index) | Validate data with `great_expectations`. |
| [Memray](./flytekit-plugins/memray-plugin/_index) | `memray`: Memory profiling with memray. |
| [MLFlow](./flytekit-plugins/mlflow-plugin/_index) | `mlflow`: the open standard for model tracking. |
| [Modin](./flytekit-plugins/modin-plugin/_index) | Scale pandas workflows with `modin`. |
| [Neptune](./flytekit-plugins/neptune-plugin/_index) | `neptune`: Neptune is the MLOps stack component for experiment tracking. |
| [NIM](./flytekit-plugins/nim-plugin/_index) | Serve optimized model containers with NIM. |
| [Ollama](./flytekit-plugins/ollama-plugin/_index) | Serve fine-tuned LLMs with Ollama in a Flyte workflow. |
| [ONNX](./flytekit-plugins/onnx-plugin/_index) | Convert ML models to ONNX models seamlessly. |
| [Pandera](./flytekit-plugins/pandera-plugin/_index) | Validate pandas dataframes with `pandera`. |
| [Papermill](./flytekit-plugins/papermill-plugin/_index) | Execute Jupyter Notebooks with `papermill`. |
| [SQL](./flytekit-plugins/sql-plugin/_index) | Execute SQL queries as tasks. |
| [Weights and Biases](./flytekit-plugins/wandb-plugin/_index) | `wandb`: Machine learning platform to build better models faster. |
| [WhyLogs](./flytekit-plugins/whylogs-plugin/_index) | `whylogs`: the open standard for data logging. |

### Using Flytekit plugins

Data is automatically marshalled and unmarshalled in and out of the plugin. Users should mostly implement the `flytekit.core.base-task.PythonTask` API defined in Flytekit.

Flytekit plugins are lazily loaded and can be released independently like libraries. The naming convention is `flytekitplugins-*`, where `*` indicates the package to be integrated into Flytekit. For example, `flytekitplugins-papermill` enables users to author Flytekit tasks using [Papermill](https://papermill.readthedocs.io/en/latest/).

You can find the plugins maintained by the core Flyte team [here](https://github.com/flyteorg/flytekit/tree/master/plugins).

## Native backend plugins

Native backend plugins can be executed without any external service dependencies because the compute is orchestrated by Flyte itself, within its provisioned Kubernetes clusters.

| Plugin | Description |
|--------|-------------|
| [Kubeflow PyTorch](./native-backend-plugins/kfpytorch-plugin/_index) | Run distributed PyTorch training jobs using `Kubeflow`. |
| [Kubeflow TensorFlow](./native-backend-plugins/kftensorflow-plugin/_index) | Run distributed TensorFlow training jobs using `Kubeflow`. |
| [Kubernetes cluster Dask jobs](./native-backend-plugins/k8s-dask-plugin/_index) | Run Dask jobs on a Kubernetes Cluster. |
| [Kubernetes cluster Spark jobs](./native-backend-plugins/k8s-spark-plugin/_index) | Run Spark jobs on a Kubernetes Cluster. |
| [MPI Operator](./native-backend-plugins/kfmpi-plugin/_index) | Run distributed deep learning training jobs using Horovod and MPI. |
| [Ray](./native-backend-plugins/ray-plugin/_index) | Run Ray jobs on a K8s Cluster. |

<!-- TODO: Include this above? | [Kubernetes pods](./native-backend-plugins/k8s-pod-plugin/_index) | Execute Kubernetes pods for arbitrary workloads. | -->

## External service backend plugins

As the term suggests, these plugins rely on external services to handle the workload defined in the Flyte task that uses the plugin.

| Plugin | Description |
|--------|-------------|
| [AWS Athena](./external-service-backend-plugins/athena-plugin/_index) | Execute queries using AWS Athena |
| [AWS Batch](./external-service-backend-plugins/aws-batch-plugin/_index) | Running tasks and workflows on AWS batch service |
| [Flyte Interactive](./external-service-backend-plugins/flyteinteractive-plugin/_index) | Execute tasks using Flyte Interactive to debug. |
| [Hive](./external-service-backend-plugins/hive-plugin/_index) | Run Hive jobs in your workflows. |

## Enabling backend plugins

To enable a backend plugin, you must add the `ID` of the plugin to the enabled plugins list. The `enabled-plugins` is available under the `tasks > task-plugins` section of FlytePropeller's configuration.
The plugin configuration structure is defined [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller@v0.6.1/pkg/controller/nodes/task/config#TaskPluginConfig). An example of the config follows:

```yaml
tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array
```

**Finding the `ID` of the backend plugin**

To find the `ID` of the backend plugin, look at the source code of the plugin. For examples, in the case of Spark, the value of `ID` is used [here](https://github.com/flyteorg/flyteplugins/blob/v0.5.25/go/tasks/plugins/k8s/spark/spark.go#L424), defined as [spark](https://github.com/flyteorg/flyteplugins/blob/v0.5.25/go/tasks/plugins/k8s/spark/spark.go#L41).

## Flyte operators

Flyte can be integrated with other orchestrators to help you leverage Flyte's
constructs natively within other orchestration tools.

| Operator | Description |
|----------|-------------|
| [Airflow](./flyte-operators/airflow-plugin/_index) | Trigger Flyte executions from Airflow. |

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors ===

# Connectors

Connectors are long-running, stateless services that receive execution requests via gRPC and initiate jobs with appropriate external or internal services. Each connector service is a Kubernetes deployment that receives gRPC requests when users trigger a particular type of task. (For example, the BigQuery connector is tiggered by the invocation of a BigQuery tasks.) The connector service then initiates a job with the appropriate service.

Connectors can be run locally as long as the appropriate connection secrets are locally available, since they are spawned in-process.

Connectors are designed to be scalable and can handle large workloads efficiently, and decrease load on the core system, since they run outside it.
You can also test connectors locally without having to change the backend configuration, streamlining workflow development.

Connectors enable two key use cases:

* **Asynchronously** launching jobs on hosted platforms (e.g. Databricks or Snowflake).
* Calling external **synchronous** services, such as access control, data retrieval, or model inferencing.

This section covers all currently available connectors:

* [Airflow connector](./airflow-connector/_index)
* [BigQuery connector](./bigquery-connector/_index)
* [OpenAI ChatGPT connector](./chatgpt-connector/_index)
* [OpenAI Batch connector](./openai-batch-connector/_index)
* [Databricks connector](./databricks-connector/_index)
* [Memory Machine Cloud connector](./mmcloud-connector/_index)
* [Perian connector](./perian-connector/_index)
* [Sagemaker connector](./sagemaker-inference-connector/_index)
* [Sensor connector](./sensor/_index)
* [Slurm connector](./slurm-connector/_index)
* [Snowflake connector](./snowflake-connector/_index)

## Creating a new connector

If none of the existing connectors meet your needs, you can implement your own connector.

There are two types of connectors: **async** and **sync**.
* **Async connectors** enable long-running jobs that execute on an external platform over time.
  They communicate with external services that have asynchronous APIs that support `create`, `get`, and `delete` operations.
  The vast majority of connectors are async connectors.
* **Sync connectors** enable request/response services that return immediate outputs (e.g. calling an internal API to fetch data or communicating with the OpenAI API).

> [!NOTE]
> While connectors can be written in any programming language since they use a protobuf interface,
> we currently only support Python connectors.
> We may support other languages in the future.

### Async connector interface specification

To create a new async connector, extend the `AsyncConnectorBase` and implement `create`, `get`, and `delete` methods. These methods must be idempotent.

- `create`: This method is used to initiate a new job. Users have the flexibility to use gRPC, REST, or an SDK to create a job.
- `get`: This method retrieves the job resource (job ID or output literal) associated with the task, such as a BigQuery job ID or Databricks task ID.
- `delete`: Invoking this method will send a request to delete the corresponding job.

For an example implementation, see the [BigQuery connector code](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-bigquery/flytekitplugins/bigquery/connector.py).

### Sync connector interface specification

To create a new sync connector, extend the `SyncConnectorBase` class and implement a `do` method. This method must be idempotent.

- `do`: This method is used to execute the synchronous task, and the worker in Flyte will be blocked until the method returns.

For an example implementation, see the [ChatGPT connector code](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-openai/flytekitplugins/openai/chatgpt/connector.py).

### Testing your connector locally

To test your connector locally, create a class for the connector task that inherits from [`AsyncConnectorExecutorMixin`](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L354). This mixin can handle both asynchronous tasks and synchronous tasks and allows Flytekit to mimic the system's behavior in calling the connector.

For testing examples, see the **Connectors > BigQuery connector** and **Connectors > Databricks connector** documentation.

## Enabling a connector in your Flyte deployment

For information on setting up a connector in your Flyte deployment, see [Deployment > Connector setup](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/_index)

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/airflow-connector ===

# Airflow connector

[Apache Airflow](https://airflow.apache.org) is a widely used open source platform for managing workflows with a robust ecosystem. Flyte provides an Airflow plugin that allows you to run Airflow tasks as Flyte tasks.
This allows you to use the Airflow plugin ecosystem in conjunction with Flyte's powerful task execution and orchestration capabilities.

> [!NOTE]
> The Airflow connector does not support all [Airflow operators](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html).
> We have tested many, but if you run into issues,
> please [file a bug report](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&projects=&template=bug_report.yaml&title=%5BBUG%5D+).

## Installation

To install the plugin, run the following command:

`pip install flytekitplugins-airflow`

This plugin has two components:
* **Airflow compiler:** This component compiles Airflow tasks to Flyte tasks, so Airflow tasks can be directly used inside the Flyte workflow.
* **Airflow connector:** This component allows you to execute Airflow tasks either locally or on a Flyte cluster.

> [!NOTE]
> You don't need an Airflow cluster to run Airflow tasks, since flytekit will
> automatically compile Airflow tasks to Flyte tasks and execute them on the Airflow connector.

## Example usage

For an example query, see **Connectors > Airflow connector > Page**

## Local testing

Airflow doesn't support local execution natively. However, Flyte compiles Airflow tasks to Flyte tasks,
which enables you to test Airflow tasks locally in flytekit's local execution mode.

> [!NOTE]
> In some cases, you will need to store credentials in your local environment when testing locally.

## Flyte deployment configuration

To enable the Airflow connector in your Flyte deployment, see the [Airflow connector deployment guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/airflow/page.md).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/airflow-connector/airflow-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/airflow-connector/airflow-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/airflow-connector/airflow-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/bigquery-connector ===

# BigQuery connector

## Installation

To install the BigQuery connector, run the following command:

This connector is purely a spec. Since SQL is completely portable, there is no need to build a Docker container.

## Example usage

For an example query, see **Connectors > BigQuery connector > Page**

## Local testing

To test the BigQuery connector locally, create a class for the connector task that inherits from
[AsyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L354).
This mixin can handle asynchronous tasks and allows the SDK to mimic the system's behavior in calling the connector.

For more information, see **Connectors**.

> [!NOTE]
> In some cases, you will need to store credentials in your local environment when testing locally.

## Flyte deployment configuration

To enable the BigQuery connector in your Flyte deployment, see the [BigQuery connector deployment guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/bigquery/page.md)

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/bigquery-connector/bigquery-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/bigquery-connector/bigquery-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/bigquery-connector/bigquery-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/chatgpt-connector ===

# ChatGPT connector

## Installation

To install the ChatGPT connector, run the following command:

```shell
$ pip install flytekitplugins-openai
```

## Example usage

For an example query, see **Connectors > ChatGPT connector > Page**

## Local testing

To test the ChatGPT connector locally, create a class for the connector task that inherits from
[SyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L304).
This mixin can handle synchronous tasks and allows the SDK to mimic the system's behavior in calling the connector.

For more information, see **Connectors**.

> [!NOTE]
> In some cases, you will need to store credentials in your local environment when testing locally.

## Flyte deployment configuration

To enable the ChatGPT connector in your Flyte deployment, see the [ChatGPT connector deployment guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/chatgpt/page.md).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/chatgpt-connector/chatgpt-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/chatgpt-connector/chatgpt-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/chatgpt-connector/chatgpt-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/databricks-connector ===

# Databricks connector

Flyte can be integrated with the [Databricks](https://www.databricks.com/) service,
enabling you to submit Spark jobs to the Databricks platform.

## Installation

The Databricks connector comes bundled with the Spark plugin. To install the Spark plugin, run the following command:

```shell
$ pip install flytekitplugins-spark
```

## Example usage

For an example query, see **Connectors > Databricks connector > Page**

## Local testing

To test the Databricks connector locally, create a class for the connector task that inherits from
[AsyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L354).
This mixin can handle asynchronous tasks and allows the SDK to mimic the system's behavior in calling the connector.

For more information, see **Connectors**.

> [!NOTE]
> In some cases, you will need to store credentials in your local environment when testing locally.

## Flyte deployment configuration

To enable the Databricks connector in your Flyte deployment, see the [Databricks connector setup guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/databricks/page.md).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/databricks-connector/databricks-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/databricks-connector/databricks-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/databricks-connector/databricks-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/mmcloud-connector ===

# Memory Machine Cloud connector

[MemVerge](https://memverge.com/) [Memory Machine Cloud](https://www.mmcloud.io/) (MMCloud)—available on AWS, GCP, and AliCloud—empowers users to continuously optimize cloud resources during runtime, safely execute stateful tasks on spot instances, and monitor resource usage in real time. These capabilities make it an excellent fit for long-running batch workloads. Flyte can be integrated with MMCloud, allowing you to execute Flyte tasks using MMCloud.

## Installation

To install the connector, run the following command:

```shell
$ pip install flytekitplugins-mmcloud
```

To get started with Memory Machine Cloud, see the [Memory Machine Cloud user guide](https://docs.memverge.com/MMCloud/latest/User%20Guide/about).

## Example usage

For an example query, see **Connectors > Memory Machine Cloud connector > Page**

## Local testing

To test the MMCloud connector locally, create a class for the connector task that inherits from
[AsyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L354).
This mixin can handle asynchronous tasks and allows the SDK to mimic the system's behavior in calling the connector.

For more information, see **Connectors**.

> [!NOTE]
> In some cases, you will need to store credentials in your local environment when testing locally.

## Flyte deployment configuration

To enable the Memory Machine Cloud connector in your Flyte deployment, see the [MMCloud connector setup guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/mmcloud/page.md).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/mmcloud-connector/mmcloud-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/mmcloud-connector/mmcloud-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/mmcloud-connector/mmcloud-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/openai-batch-connector ===

# OpenAI Batch connector

The Batch API connector allows you to submit requests for asynchronous batch processing on OpenAI.
You can provide either a JSONL file or a JSON iterator, and the connector handles the upload to OpenAI,
creation of the batch, and downloading of the output and error files.

## Installation

To use the OpenAI Batch connector, run the following command:

```shell
$ pip install flytekitplugins-openai
```

## Example usage

For an example query, see **Connectors > OpenAI Batch connector > Page**

## Local testing

To test an connector locally, create a class for the connector task that inherits from
[SyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L304)
or [AsyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L354).
These mixins can handle synchronous and synchronous tasks, respectively, and allow the SDK to mimic the system's behavior in calling the connector.

For more information, see **Connectors**.

## Flyte deployment configuration

To enable the OpenAI Batch connector in your Flyte deployment, refer to the [OpenAI Batch connector setup guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/openai-batch/page.md)

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/openai-batch-connector/openai-batch-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/openai-batch-connector/openai-batch-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/openai-batch-connector/openai-batch-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/perian-connector ===

# Perian connector

The Perian connector enables you to execute Flyte tasks on the [Perian Sky Platform](https://perian.io/).
Perian allows the execution of any task on servers aggregated from multiple cloud providers.

To get started with Perian, see the [Perian documentation](https://perian.io/docs/overview) and the [Perian connector documentation](https://perian.io/docs/flyte-getting-started).

## Example usage

For an example, see **Connectors > Perian connector > Page**

## Connector setup

Consult the [PERIAN connector setup guide](https://perian.io/docs/flyte-setup-guide).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/perian-connector/example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/perian-connector/example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/perian-connector/example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/sagemaker-inference-connector ===

# SageMaker connector

The SageMaker connector allows you to deploy models, and create and trigger inference endpoints.
You can also fully remove the SageMaker deployment.

## Installation

To use the SageMaker connector, run the following command:

```shell
$ pip install flytekitplugins-awssagemaker
```

## Example usage

For an example query, see **Connectors > SageMaker connector > Page**

## Local testing

To test an connector locally, create a class for the connector task that inherits from
[SyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L304)
or [AsyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L354).
These mixins can handle synchronous and synchronous tasks, respectively, and allow the SDK to mimic the system's behavior in calling the connector.

For more information, see **Connectors**.

## Flyte deployment configuration

To enable the AWS SageMaker inference connector in your Flyte deployment, refer to the [AWS SageMaker inference connector setup guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/sagemaker-inference/page.md).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/sagemaker-inference-connector/sagemaker-inference-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/sagemaker-inference-connector/sagemaker-inference-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/sagemaker-inference-connector/sagemaker-inference-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/sensor ===

# Sensor connector

## Example usage

For an example query, see **Connectors > Sensor connector > Page**

## Flyte deployment configuration

To enable the Sensor connector in your Flyte deployment, see the [Airflow connector deployment guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/sensor/page.md).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/sensor/file-sensor-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/sensor/file-sensor-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/sensor/file-sensor-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/slurm-connector ===

# Slurm connector

## Installation

To install the Slurm connector, run the following command:

```shell
$ pip install flytekitplugins-slurm
```

## Example usage

For an example query, see **Connectors > Slurm connector > Page**

## Local testing

To test the Slurm connector locally, create a class for the connector task that inherits from
[AsyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L354).
This mixin can handle asynchronous tasks and allows the SDK to mimic the system's behavior in calling the connector.

For more information, see **Connectors**.

> [!NOTE]
> In some cases, you will need to store credentials in your local environment when testing locally.

## Flyte deployment configuration

To enable the Slurm connector in your Flyte deployment, see the [Slurm connector deployment guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/slurm/page.md).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/slurm-connector/slurm-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/slurm-connector/slurm-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/slurm-connector/slurm-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/snowflake-connector ===

# Snowflake connector

Flyte can be seamlessly integrated with the [Snowflake](https://www.snowflake.com) service,
providing you with a straightforward means to query data in Snowflake.

## Installation

To use the Snowflake connector, run the following command:

```shell
$ pip install flytekitplugins-snowflake
```

## Example usage

For an example query, see **Connectors > Snowflake connector > Page**

## Local testing

To test the Snowflake connector locally, create a class for the connector task that inherits from
[AsyncConnectorExecutorMixin](https://github.com/flyteorg/flytekit/blob/1bc8302bb7a6cf4c7048a7f93627ee25fc6b88c4/flytekit/extend/backend/base_connector.py#L354).
This mixin can handle asynchronous tasks and allows the SDK to mimic the system's behavior in calling the connector.

For more information, see **Connectors**.

> [!NOTE]
> In some cases, you will need to store credentials in your local environment when testing locally.

## Flyte deployment configuration

To enable the Snowflake connector in your Flyte deployment, see the [Snowflake connector setup guide](https://www.union.ai/docs/v1/flyte/deployment/flyte-connectors/snowflake/page.md).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/connectors/snowflake-connector/snowflake-connector-example-usage ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/connectors/snowflake-connector/snowflake-connector-example-usage.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/connectors/snowflake-connector/snowflake-connector-example-usage/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins ===

# Flytekit plugins

This section covers Flytekit plugins.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/comet-ml-plugin ===

# Comet ML

Comet’s machine learning platform integrates with your existing infrastructure and tools so you can manage, visualize, and optimize models from training runs to production monitoring. This plugin integrates Flyte with Comet by configuring links between the two platforms.

To install the plugin, run:

```shell
$ pip install flytekitplugins-comet-ml
```

Comet requires an API key to authenticate with their platform. In the above example, a secret is created using
[Flyte's Secrets manager](https://www.union.ai/docs/v1/flyte/deployment/flyte-configuration/secrets/page.md).

To enable linking from the Flyte side panel to Comet.ml, add the following to Flyte's configuration:

```yaml
plugins:
  logs:
    dynamic-log-links:
      - comet-ml-execution-id:
          displayName: Comet
          templateUris: "{{ .taskConfig.host }}/{{ .taskConfig.workspace }}/{{ .taskConfig.project_name }}/{{ .executionName }}{{ .nodeId }}{{ .taskRetryAttempt }}{{ .taskConfig.link_suffix }}"
      - comet-ml-custom-id:
          displayName: Comet
          templateUris: "{{ .taskConfig.host }}/{{ .taskConfig.workspace }}/{{ .taskConfig.project_name }}/{{ .taskConfig.experiment_key }}"
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/comet-ml-plugin/comet-ml-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/comet-ml-plugin/comet-ml-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/comet-ml-plugin/comet-ml-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/dbt-plugin ===

# DBT

[dbt](https://www.getdbt.com/) is one of the widely-used data transformation
tools for working with data directly in a data warehouse. It's optimized for
analytics use cases and can be used for business intelligence, operational
analytics, and even machine learning.

The Flytekit `dbt` plugin is a Python module that provides an easy way to
invoke basic `dbt` CLI commands from within a Flyte task. The plugin supports
the commands [dbt run](https://docs.getdbt.com/reference/commands/run),
[dbt test](https://docs.getdbt.com/reference/commands/test), and
[dbt source freshness](https://docs.getdbt.com/reference/commands/source).

## Prerequisities

To use the `dbt` plugin you'll need to install the `flytekitplugins-dbt`
plugin.

> [!NOTE]
> See [the PyPi page here](https://pypi.org/project/flytekitplugins-dbt/).

```shell
$ pip install flytekitplugins-dbt
```

Then install dbt itself. You will have to install `dbt-core` as well as
the correct adapter for the database that you are accessing.

For example, if you are using a Postgres database you would do:

This will install `dbt-core` and `dbt-postgres`, but not any of the other
adapters, `dbt-redshift`, `dbt-snowflake`, or `dbt-bigquery`. See
[the official installation page](https://docs.getdbt.com/docs/get-started/pip-install)
for details.

## Running the Example

We use a Postgres database installed on the cluster and an example project from
dbt, called [jaffle-shop](https://github.com/dbt-labs/jaffle_shop).
To run the example on your local machine, do the following.

> [!IMPORTANT]
> The example below is not designed to run directly in your local
> python environment. It must be run in a Kubernetes cluster, either locally on
> your machine using the `flytectl demo start` command or on a cloud cluster.

Start up the demo cluster on your local machine:

```shell
$ flytectl demo start
```

Pull the pre-built image for this example:

```shell
$ docker pull ghcr.io/flyteorg/flytecookbook:dbt_example-latest
```

This image is built using the following `Dockerfile` and contains:

- The `flytekitplugins-dbt` and `dbt-postgres` Python dependencies.
- The `jaffle-shop` example.
- A postgres database.

This Dockerfile can be found in the ``flytesnacks/examples`` directory under
the filepath listed in the code block title below.

```dockerfile
FROM python:3.8-slim-buster

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

RUN apt-get update && apt-get install -y build-essential git postgresql-client libpq-dev

# Install the AWS cli separately to prevent issues with boto being written over
RUN pip3 install awscli

ENV VENV /opt/venv
# Virtual environment
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

# Install Python dependencies
COPY requirements.in /root/
RUN pip install -r /root/requirements.in
# psycopg2-binary is a dependency of the dbt-postgres adapter, but that doesn't work on mac M1s.
# As per https://github.com/psycopg/psycopg2/issues/1360, we install psycopg to circumvent this.
RUN pip uninstall -y psycopg2-binary && pip install psycopg2

# Copy the actual code
COPY . /root/

# Copy dbt-specific files
COPY profiles.yml /root/dbt-profiles/
RUN git clone https://github.com/dbt-labs/jaffle_shop.git

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag

ENV FLYTE_SDK_LOGGING_LEVEL 10
```

To run this example, copy the code in the **dbt example** below into a file
called `dbt_example.py`, then run it on your local container using the
provided image:

```shell
$ pyflyte run --remote \
       --image ghcr.io/flyteorg/flytecookbook:dbt_example-latest \
       dbt_plugin/dbt_example.py wf
```

Alternatively, you can clone the `flytesnacks` repo and run the example directly:

```shell
$ git clone https://github.com/flyteorg/flytesnacks
$ cd flytesnacks/examples/dbt_example
$ pyflyte run --remote \
          --image ghcr.io/flyteorg/flytecookbook:dbt_example-latest \
          dbt_plugin/dbt_example.py wf
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/dbt-plugin/dbt-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/dbt-plugin/dbt-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/dbt-plugin/dbt-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/dolt-plugin ===

# Dolt

The `DoltTable` plugin is a wrapper that uses [Dolt](https://github.com/dolthub/dolt) to move data between
`pandas.DataFrame`'s at execution time and database tables at rest.

## Installation

The dolt plugin and dolt command line tool are required to run these examples:

```shell
$ pip install flytekitplugins.dolt
$ sudo bash -c 'curl -L https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash'
```

Dolt requires a user configuration to run `init`:

```shell
$ dolt config --global --add user.email <email>
$ dolt config --global --add user.name <name>
```

These demos assume a `foo` database has been created locally:

```shell
$ mkdir foo
$ cd foo
$ dolt init
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/dolt-plugin/dolt-branch-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/dolt-plugin/dolt-branch-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/dolt-plugin/dolt-branch-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/dolt-plugin/dolt-quickstart-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/dolt-plugin/dolt-quickstart-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/dolt-plugin/dolt-quickstart-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/greatexpectations-plugin ===

# Great Expectations

**Great Expectations** is a Python-based open-source library for validating, documenting, and profiling your data.
It helps maintain data quality and improve communication about data between teams.

The goodness of data validation in Great Expectations can be integrated with Flyte to validate the data moving in and out of
the pipeline entities you may have defined in Flyte. This helps establish stricter boundaries around your data to
ensure that everything is as you expected and hence, data will not crash your pipelines anymore unexpectedly!

## How to Define the Integration

Great Expectations supports native execution of expectations against various [Datasources](https://docs.greatexpectations.io/docs/terms/datasource/),
such as Pandas dataframes, Spark dataframes, and SQL databases via SQLAlchemy.

We're supporting two Flyte types that should suit Great Expectations' `Datasources`:

- `flytekit.types.file.FlyteFile`: `FlyteFile` represents an automatic persistence object in Flyte.
  It can represent files in remote storage and Flyte transparently materializes them in every task execution.
- `flytekit.types.structured.StructuredDataset`: `StructuredDataset` supports pandas dataframes, which the plugin will convert into a parquet file and validate the data using Great Expectations.

> [!NOTE]
> Flyte types are added because, in Great Expectations, we have the privilege to give a non-string (Pandas/Spark DataFrame) when using a`RuntimeDataConnector`
> but not when using an `InferredAssetFilesystemDataConnector` or a `ConfiguredAssetFilesystemDataConnector`.
> For the latter case, with the integration of Flyte types, we can give a Pandas/Spark DataFrame or a remote URI as the dataset.

The datasources can be well-integrated with the plugin using the following two modes:

- **Flyte Task**: A Flyte task defines the task prototype that one could use within a task or a workflow to validate data using Great Expectations.
- **Flyte Type**: A Flyte type helps attach the `GreatExpectationsType` to any dataset.
  Under the hood, `GreatExpectationsType` can be assumed as a combination of Great Expectations and Flyte types where every data is validated against the expectations, much like the OpenAPI Spec or the gRPC validator.

### Data Validation Failure

If the data validation fails, the plugin will raise a `GreatExpectationsValidationError`.

For example, this is how the error message looks on the Flyte UI:

```shell
Traceback (most recent call last):
...
great_expectations.marshmallow__shade.exceptions.ValidationError: Validation failed!
COLUMN          FAILED EXPECTATION
passenger_count -> expect_column_min_to_be_between
passenger_count -> expect_column_mean_to_be_between
passenger_count -> expect_column_quantile_values_to_be_between
passenger_count -> expect_column_values_to_be_in_set
passenger_count -> expect_column_proportion_of_unique_values_to_be_between
trip_distance -> expect_column_max_to_be_between
trip_distance -> expect_column_mean_to_be_between
trip_distance -> expect_column_median_to_be_between
trip_distance -> expect_column_quantile_values_to_be_between
trip_distance -> expect_column_proportion_of_unique_values_to_be_between
rate_code_id -> expect_column_max_to_be_between
rate_code_id -> expect_column_mean_to_be_between
rate_code_id -> expect_column_proportion_of_unique_values_to_be_between
```

## Plugin Parameters

- **datasource_name**: Data source, in general, is the "name" we use in the Great Expectations config file.
  A Datasource brings together a way of interacting with data (like a database or Spark cluster) and some specific data
  (like a CSV file, or a database table). Moreover, data source assists in building batches out of data (for validation).
- **expectation_suite_name**: Defines the data validation.
- **data_connector_name**: Tells how the data batches have to be identified.

### Optional Parameters

- **context_root_dir**: Sets the path of the great expectations config directory.

- **checkpoint_params**: Optional `SimpleCheckpoint` class parameters.

- **batch_request_config**: Additional batch request configuration parameters.

  - data_connector_query: Query to request a data batch
  - runtime_parameters: Parameters to be sent at run-time
  - batch_identifiers: Batch identifiers
  - batch_spec_passthrough: Reader method if your file doesn't have an extension

- **data_asset_name**: Name of the data asset (to be used for `RuntimeBatchRequest`)

- **local_file_path**: Helpful to download the given dataset to the user-given path

> [!NOTE]
> You may always want to mention the **context_root_dir** parameter, as providing a path means no harm!
> Moreover, **local_file_path** is essential when using `FlyteFile` and `FlyteSchema`.

## Plugin Installation

To use the Great Expectations Flyte plugin, run the following command:

```shell
$  pip install flytekitplugins-great_expectations
```

> [!NOTE]
> Make sure to run workflows from the `flytekit_plugins` directory.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/greatexpectations-plugin/task-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/greatexpectations-plugin/task-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/greatexpectations-plugin/task-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/greatexpectations-plugin/type-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/greatexpectations-plugin/type-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/greatexpectations-plugin/type-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/memray-plugin ===

# Memray Profiling

Memray tracks and reports memory allocations, both in python code and in compiled extension modules.
This Memray Profiling plugin enables memory tracking on the Flyte task level and renders a memgraph profiling graph on Flyte Deck.

First, install the Memray plugin:

```bash
$ pip install flytekitplugins-memray
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/memray-plugin/memray-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/memray-plugin/memray-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/memray-plugin/memray-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/mlflow-plugin ===

# MLflow

The MLflow Tracking component is an API and UI for logging parameters,
code versions, metrics, and output files when running your machine learning code and for later visualizing the results

First, install the Flyte MLflow plugin:

```shell
$  pip install flytekitplugins-mlflow
```

To log the metrics and parameters to Flyte deck, add `@mlflow_autolog` to the task. For example

```python
@task(enable_deck=True)
@mlflow_autolog(framework=mlflow.keras)
def train_model(epochs: int):
...
```

To log the metric and parameters to a remote MLflow server, add default environment variable [MLFLOW_TRACKING_URI](https://mlflow.org/docs/latest/tracking.html#logging-to-a-tracking-server) to the flytepropeller config map.

```shell
$  kubectl edit cm flyte-propeller-config
```

```yaml
plugins:
  k8s:
    default-cpus: 100m
    default-env-vars:
    - MLFLOW_TRACKING_URI: postgresql+psycopg2://postgres:@postgres.flyte.svc.cluster.local:5432/flyteadmin
```

![MLflow UI](https://www.union.ai/docs/v1/flyte/_static/images/integrations/flytekit-plugins/mlflow-plugin/mlflow-ui.png)

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/mlflow-plugin/mlflow-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/mlflow-plugin/mlflow-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/mlflow-plugin/mlflow-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/modin-plugin ===

# Modin

Modin is a pandas-accelerator that helps handle large datasets.
Pandas works gracefully with small datasets since it is inherently single-threaded, and designed to work on a single CPU core.
With large datasets, the performance of pandas drops (becomes slow or runs out of memory) due to single core usage.
This is where Modin can be helpful.

Instead of optimizing pandas workflows for a specific setup, we can speed up pandas workflows by utilizing all the resources (cores) available in the system using the concept of `parallelism`, which is possible through modin. [Here](https://modin.readthedocs.io/en/stable/getting_started/why_modin/pandas.html#scalablity-of-implementation) is a visual representation of how the cores are utilized in case of Pandas and Modin.

## Installation

```shell
$ pip install flytekitplugins-modin
```

## How is Modin different?

Modin **scales** the Pandas workflows by changing only a **single line of code**.

The plugin supports the usage of Modin DataFrame as an input to and output of a task/workflow, similar to how a pandas DataFrame can be used.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/modin-plugin/knn-classifier ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/modin-plugin/knn-classifier.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/modin-plugin/knn-classifier/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/neptune-plugin ===

# Neptune

[Neptune](https://neptune.ai/) is an experiment tracker for large-scale model training. It allows AI researchers to monitor their model training in real time, visualize and compare experiments, and collaborate on them with a team. This plugin enables seamless use of Neptune within Flyte by configuring links between the two platforms. You can find more information about how to use Neptune in their [documentation](https://docs.neptune.ai/).

## Installation

To install the Flyte Neptune plugin, run the following command:

```shell
$ pip install flytekitplugins-neptune
```

## Local testing

To run {doc}`Neptune example <neptune_example>` locally:

1. Neptune Scale is available to select customers. You can access it [here](https://neptune.ai/free-trial).
2. Create a project on Neptune.
3. In the example, set `NEPTUNE_PROJECT` to your project name.
4. Add a secret using [Flyte's Secrets manager](https://www.union.ai/docs/v1/flyte/deployment/flyte-configuration/secrets/page.md) with `key="neptune-api-token"` and `group="neptune-api-group"`
5. If you want to see the dynamic log links in the UI, then add the configuration available in the next section.

## Flyte deployment configuration

To enable dynamic log links, add the plugin to Flyte's configuration file as follows:

```yaml
plugins:
  logs:
    dynamic-log-links:
      - neptune-scale-run:
          displayName: Neptune Run
          templateUris:
            - "https://scale.neptune.ai/{{ .taskConfig.project }}/-/run/?customId={{ .podName }}"
      - neptune-scale-custom-id:
          displayName: Neptune Run
          templateUris:
            - "https://scale.neptune.ai/{{ .taskConfig.project }}/-/run/?customId={{ .taskConfig.id }}"
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/neptune-plugin/neptune-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/neptune-plugin/neptune-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/neptune-plugin/neptune-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/nim-plugin ===

# NIM

Serve optimized model containers with NIM in a Flyte task.

[NVIDIA NIM](https://www.nvidia.com/en-in/ai/), part of NVIDIA AI Enterprise, provides a streamlined path
for developing AI-powered enterprise applications and deploying AI models in production.
It includes an out-of-the-box optimization suite, enabling AI model deployment across any cloud,
data center, or workstation. Since NIM can be self-hosted, there is greater control over cost, data privacy,
and more visibility into behind-the-scenes operations.

With NIM, you can invoke the model's endpoint as if it is hosted locally, minimizing network overhead.

## Installation

To use the NIM plugin, run the following command:

```shell
$ pip install flytekitplugins-inference
```

## Example usage

For a usage example, see **Flytekit plugins > NIM > Page**.

> [!NOTE]
> NIM can only be run in a Flyte cluster as it must be deployed as a sidecar service in a Kubernetes pod.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/nim-plugin/serve-nim-container ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/nim-plugin/serve-nim-container.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/nim-plugin/serve-nim-container/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/ollama-plugin ===

# Ollama

Serve large language models (LLMs) in a Flyte task.

[Ollama](https://ollama.com/) simplifies the process of serving fine-tuned LLMs.
Whether you're generating predictions from a customized model or deploying it across different hardware setups,
Ollama enables you to encapsulate the entire workflow in a single pipeline.

## Installation

To use the Ollama plugin, run the following command:

```shell
$ pip install flytekitplugins-inference
```

## Example usage

For a usage example, see **Flytekit plugins > Ollama > Page**

> [!NOTE]
> Ollama can only be run in a Flyte cluster as it must be deployed as a sidecar service in a Kubernetes pod.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/ollama-plugin/serve-llm ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/ollama-plugin/serve-llm.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/ollama-plugin/serve-llm/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/onnx-plugin ===

# ONNX

Open Neural Network Exchange ([ONNX](https://github.com/onnx/onnx)) is an open standard format for representing machine learning
and deep learning models. It enables interoperability between different frameworks and streamlines the path from research to production.

The flytekit onnx type plugin comes in three flavors:

<!-- Empty tabs block -->

> [!NOTE]
> If you'd like to add support for a new framework, please create an issue and submit a pull request to the flytekit repo.
> You can find the ONNX plugin source code [here](https://github.com/flyteorg/flytekit/tree/master/plugins).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/onnx-plugin/pytorch-onnx ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/onnx-plugin/pytorch-onnx.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/onnx-plugin/pytorch-onnx/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/onnx-plugin/scikitlearn-onnx ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/onnx-plugin/scikitlearn-onnx.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/onnx-plugin/scikitlearn-onnx/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/onnx-plugin/tensorflow-onnx ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/onnx-plugin/tensorflow-onnx.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/onnx-plugin/tensorflow-onnx/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/pandera-plugin ===

# Pandera

Flytekit python natively supports {ref}`many data types <python_to_flyte_type_mapping>`,
including a `FlyteSchema` type for
type-annotating pandas dataframes. The flytekit pandera plugin provides an alternative for
defining dataframe schemas by integrating with [pandera](https://pandera.readthedocs.io/en/stable/),
which is a runtime data validation tool for pandas dataframes.

## Installation

```shell
$ pip install flytekitplugins-pandera
```

## Quick Start

Pandera provides a flexible and expressive interface for defining schemas for
tabular data, where you can define the types and other statistical properties
of a column.

```python
import pandas as pd
import pandera as pa
from pandera.typing import DataFrame, Series
class Schema(pa.SchemaModel):
    column_1: Series[int] = pa.Field(ge=0)
    column_2: Series[float] = pa.Field(gt=0, lt=100)
    column_3: Series[str] = pa.Field(str_startswith="prefix")
    @pa.check("column_3")
    def check_str_length(cls, series):
        return series.str.len() > 5
@pa.check_types
def processing_fn(df: DataFrame[Schema]) -> DataFrame[Schema]:
    df["column_1"] = df["column_1"] * 2
    df["column_2"] = df["column_2"] * 0.5
    df["column_3"] = df["column_3"] + "_suffix"
    return df
raw_df = pd.DataFrame({
   "column_1": [1, 2, 3],
   "column_2": [1.5, 2.21, 3.9],
   "column_3": ["prefix_a", "prefix_b", "prefix_c"],
})
processed_df = processing_fn(raw_df)
print(processed_df)
```

```shell
   column_1  column_2  column_3
0         2     0.750  prefix_a_suffix
1         4     1.105  prefix_b_suffix
2         6     1.950  prefix_c_suffix
```

Informative errors are raised if invalid data is passed into `processing_fn`,
indicating the failure case and the index where they were found in the dataframe:

```python
invalid_df = pd.DataFrame({
   "column_1": [-1, 2, -3],
   "column_2": [1.5, 2.21, 3.9],
   "column_3": ["prefix_a", "prefix_b", "prefix_c"],
})
processing_fn(invalid_df)
```

```shell
Traceback (most recent call last):
...
pandera.errors.SchemaError: error in check_types decorator of function 'processing_fn': <Schema Column(name=column_1, type=<class 'int'>)> failed element-wise validator 0:
<Check greater_than_or_equal_to: greater_than_or_equal_to(0)>
failure cases:
   index  failure_case
0      0            -1
1      2            -3
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/pandera-plugin/basic-schema-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/pandera-plugin/basic-schema-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/pandera-plugin/basic-schema-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/pandera-plugin/validating-and-testing-ml-pipelines ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/pandera-plugin/validating-and-testing-ml-pipelines.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/pandera-plugin/validating-and-testing-ml-pipelines/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/papermill-plugin ===

# Papermill

It is possible to run a Jupyter notebook as a Flyte task using [papermill](https://github.com/nteract/papermill).
Papermill executes the notebook as a whole, so before using this plugin, it is essential to construct your notebook as
recommended by papermill. When using this plugin, there are a few important things to keep in mind:

1. This plugin can be used for any task - type.
   : - It can be python code, which can be a tensorflow model, a data transformation, etc - but things that run in a container
       and you would typically write in a `@task`.
     - It can be a `flytekit.dynamic` workflow.
     - It can be a any other plugin like `Spark`, `SageMaker` etc, **ensure that the plugin is installed as well**
2. Flytekit will execute the notebook and capture the output notebook as an *.ipynb* file and an HTML rendered notebook as well
3. Flytekit will pass the inputs into the notebook as long as you have the first cell annotated as `parameters` and inputs are specified
4. Flytekit will read the outputs from the notebook, as long as you use annotate the notebook with `outputs` and outputs are specified

## Installation

To use the flytekit papermill plugin simply run the following:

```shell
$ pip install flytekitplugins-papermill
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/papermill-plugin/simple ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/papermill-plugin/simple.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/papermill-plugin/simple/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/papermill-plugin/nb-simple ===

<!--

   This file was generated by Makefile.jupyter. Do not edit this file directly.

   The only parts of this file that should be edited are the front matter and the
   comment at the top of the file.

-->

# Simple Papermill Notebook

> [!NOTE]
> [View source on GitHub](https://github.com/unionai/unionai-examples/blob/main/v1/flyte-integrations/flytekit-plugins/papermill_plugin/papermill_plugin/nb_simple.ipynb) | [Run in Google Colab](https://colab.research.google.com/github/unionai/unionai-examples/blob/main/v1/flyte-integrations/flytekit-plugins/papermill_plugin/papermill_plugin/nb_simple.ipynb)

This notebook is used in the previous example as a `NotebookTask` to demonstrate
how to use the papermill plugin.

The cell below has the `parameters` tag, defining the inputs to the notebook.

```python
v = 3.14
```

Then, we do some computation:

```python
square = v*v
print(square)
```

Finally, we use the `flytekitplugins.papermill` package to record the outputs
so that flyte understands which state to serialize and pass into a downstream
task.

```python
from flytekitplugins.papermill  import record_outputs

record_outputs(square=square)
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/duckdb-plugin ===

# DuckDB

[DuckDB](https://duckdb.org/) is an in-process SQL OLAP database management system that is explicitly designed to achieve high performance in analytics.

The Flytekit DuckDB plugin facilitates the efficient execution of intricate analytical queries within your workflow.

To install the Flytekit DuckDB plugin, run the following command:

```shell
$ pip install flytekitplugins-duckdb
```

The Flytekit DuckDB plugin includes the `flytekitplugins:flytekitplugins.duckdb.DuckDBQuery` task, which allows you to specify the following parameters:

- `query`: The DuckDB query to execute.
- `inputs`: The query parameters to be used during query execution. This can be a StructuredDataset, a string or a list.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/duckdb-plugin/duckdb-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/duckdb-plugin/duckdb-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/duckdb-plugin/duckdb-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/sql-plugin ===

# SQL

Flyte tasks are not always restricted to running user-supplied containers, nor even containers at all. Indeed, this is
one of the most important design decisions in Flyte. Non-container tasks can have arbitrary targets for execution --
an API that executes SQL queries like SnowFlake, BigQuery, a synchronous WebAPI, etc.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/sql-plugin/sql-alchemy ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/sql-plugin/sql-alchemy.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/sql-plugin/sql-alchemy/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/sql-plugin/sqlite3-integration ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/sql-plugin/sqlite3-integration.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/sql-plugin/sqlite3-integration/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/wandb-plugin ===

# Weights and Biases

The Weights and Biases MLOps platform helps AI developers streamline their ML workflows from end to end. This plugin
enables seamless use of Weights & Biases within Flyte by configuring links between the two platforms.

First, install the Flyte Weights & Biases plugin:

```shell
$ pip install flytekitplugins-wandb
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/wandb-plugin/wandb-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/wandb-plugin/wandb-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/wandb-plugin/wandb-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/whylogs-plugin ===

# WhyLogs

whylogs is an open source software that allows you to log and inspect differents aspects of your data and ML models.
It creates efficient and mergeable statistical summaries of your datasets, called profiles, that have similar properties
to logs produced by regular software applications.

## whylogs with Flyte

The integration we've built consists on a type registration and also renderers.

### whylogs Flyte Type

The first part of the integration consists on the ability to pass a `DatasetProfileView` in and out of
the desired tasks. whylogs' DatasetProfileView is the representation of a snapshot of your dataset.
With this integration under Flyte's type system, users will benefit from profiling their desired dataset once
and be able to use the statistical representation for many validations and drift detection systems downstream.

To be able use it, pass in a `pandas.DataFrame` to a task and call:

```python
@task
def profiling_task(data: pd.DataFrame) -> DatasetProfileView:
    results = why.log(data)
    return results.view()
```

This will grant any downstream task the ability to ingest the profiled dataset and use
basically anything from whylogs' api, such as transforming it back to a pandas DataFrame:

```python
@task
def consume_profile_view(profile_view: DatasetProfileView) -> pd.DataFrame:
    return profile_view.to_pandas()
```

## Renderers

The Summary Drift Report is a neat HTML report containing information on the distribution and drift
detection of a target and a reference profile. It makes it easy for users to compare a new read dataset
against the one that was used to train the model that's in production.

To use it, simply take in the two desired `pandas.DataFrame` objects and call:

```python
renderer = WhylogsSummaryDriftRenderer()
report = renderer.to_html(target_data=new_data, reference_data=reference_data)
flytekit.Deck("summary drift", report)
```

The other report that can be generated with our integration is the Constraints Report. With it, users will
have a neat view on a Flyte Deck that will give intuition on which are the passed and failing constraints, enabling
them to act quicker on potentially wrong results.

```python
from whylogs.core.constraints.factories import greater_than_number

@task
def constraints_report(profile_view: DatasetProfileView) -> bool:
    builder = ConstraintsBuilder(dataset_profile_view=profile_view)
    builder.add_constraint(greater_than_number(column_name="my_column", number=10.0))

    constraints = builder.build()

    renderer = WhylogsConstraintsRenderer()
    flytekit.Deck("constraints", renderer.to_html(constraints=constraints))

    return constraints.validate()
```

Since we need a `Constraints` object to create this report, users can also return a boolean to whether their dataset
passed the validations or not, and take actions depending on this result downstream, as the code snippet above showed.
Other use-case would be to return the constraints report itself and parse it to provide more information to other
systems automatically.

```python
constraints = builder.build()
constraints.report()

>> [('my_column greater than number 10.0', 0, 1)]
```

## Installing the plugin

In order to have the whylogs plugin installed, simply run:

```shell
$ pip install flytekitplugins-whylogs
```

And you should then have it available to use on your environment!

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/whylogs-plugin/whylogs-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flytekit-plugins/whylogs-plugin/whylogs-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flytekit-plugins/whylogs-plugin/whylogs-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins ===

# Native backend plugins

This section covers native backend plugins.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/k8s-dask-plugin ===

# Dask

Flyte can natively execute [Dask](https://www.dask.org/) jobs on a Kubernetes Cluster,
effortlessly managing the lifecycle of a virtual Dask cluster.
This functionality is achieved by leveraging the open-sourced
[Dask Kubernetes Operator](https://kubernetes.dask.org/en/latest/operator.html),
and no additional sign-ups for services are required.
The process is akin to running an ephemeral Dask cluster,
which is created specifically for the Flyte task and subsequently torn down upon completion.

In the Flyte Kubernetes environment, the cost is amortized due to faster pod creation compared to machines.
However, the performance may be affected by the penalty of downloading Docker images.
Additionally, it's essential to keep in mind that starting a pod is not as swift as running a process.

Flytekit enables writing Dask code natively as a task,
with the `Dask()` config automatically configuring the Dask cluster.
The example provided in this section offers a hands-on tutorial for writing Dask Flyte tasks.

## Why use Kubernetes Dask?

Managing Python dependencies can be challenging, but Flyte simplifies the process
by enabling easy versioning and management of dependencies through containers.
The Kubernetes Dask plugin extends the benefits of containerization to Dask without
requiring the management of specialized Dask clusters.

Pros:

1. Simple to get started, providing complete isolation between workloads.
2. Each job runs in isolation with its own virtual cluster, eliminating the complexities of dependency management.
3. Flyte takes care of all the management tasks.

Cons:

1. Short-running, bursty jobs may not be the best fit due to container overhead.
2. Interactive Dask capabilities are not available with Flyte Kubernetes Dask;
   instead, it is better suited for running adhoc and scheduled jobs.

## Install the plugin

Install `flytekitplugins-dask` using `pip` in your environment.

```shell
$ pip install flytekitplugins-dask
```

> [!NOTE]
> To enable Flyte to build the Docker image for you using `ImageSpec`, install `flytekitplugins-envd`.

## Implementation details

### Local execution

When executing the Dask task on your local machine,
it will utilize a local [distributed client](https://distributed.dask.org/en/stable/client.html).
If you intend to link to a remote cluster during local development, simply define the `DASK_SCHEDULER_ADDRESS`
environment variable with the URL of the remote scheduler.
The `Client()` will then automatically connect to the cluster.

### Remote execution

#### Step 1: Deploy Dask plugin in the Flyte backend

Flyte Dask utilizes the [Dask Kubernetes operator](https://kubernetes.dask.org/en/latest/operator.html)
in conjunction with a custom-built
[Flyte Dask plugin](https://pkg.go.dev/github.com/flyteorg/flyteplugins@v1.0.28/go/tasks/plugins/k8s/dask).
To leverage this functionality, you need to enable the backend plugin in your deployment.
You can follow the steps mentioned in the {ref}`deployment-plugin-setup-k8s` section
to enable the Flyte Dask plugin for your deployment.

#### Step 2: Compute setup

Ensure that your Kubernetes cluster has sufficient resources available.
Depending on the resource requirements of your Dask job (including the job runner, scheduler and workers),
you may need to adjust the resource quotas for the namespace accordingly.

> [!NOTE]
> When working with [Dask's custom resources](https://kubernetes.dask.org/en/latest/operator_resources.html#custom-resources),
> your Flyte service account needs explicit > permissions. To that end, you need to create and bind a Cluster role.

##### Sample Cluster Role

```yaml
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  name: dask-dask-kubernetes-operator-role-cluster
  labels:
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  annotations:
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: dask
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: dask
rules:
  - verbs:
      - list
      - watch
    apiGroups:
      - <http://apiextensions.k8s.io|apiextensions.k8s.io>
    resources:
      - customresourcedefinitions
  - verbs:
      - get
      - list
      - watch
      - patch
      - create
      - delete
    apiGroups:
      - <http://kubernetes.dask.org|kubernetes.dask.org>
    resources:
      - daskclusters
      - daskworkergroups
      - daskjobs
      - daskjobs/status
      - daskautoscalers
      - daskworkergroups/scale
```

##### Binding command

```shell
$ kubectl create clusterrolebinding flyte-dask-cluster-role-binding --clusterrole=dask-dask-kubernetes-operator-role-cluster --serviceaccount=<flyte-service-account>
```

### Resource specification

It's recommended to define `limits` as this will establish the
`--nthreads` and `--memory-limit` parameters for the workers,
in line with the suggested practices by Dask
(refer to [best practices](https://kubernetes.dask.org/en/latest/kubecluster.html?highlight=--nthreads#best-practices)).
When configuring resources, the subsequent hierarchy is observed across all components of the Dask job,
which encompasses the job-runner pod, scheduler pod, and worker pods:

1. In the absence of specified resources, the
   [platform resources](https://github.com/flyteorg/flyte/blob/1e3d515550cb338c2edb3919d79c6fa1f0da5a19/charts/flyte-core/values.yaml#L520-L531)
   will be used.

2. When employing task resources, those will be enforced across all segments of the Dask job.
   You can achieve this using the following code snippet:

   ```python
   from flytekit import Resources, task
   from flytekitplugins.dask import Dask

   @task(
     task_config=Dask(),
     limits=Resources(cpu="1", mem="10Gi")  # Applied to all components
   )
   def my_dask_task():
      ...
   ```

3. When resources are designated for individual components, they hold the highest precedence.

   ```python
   from flytekit import Resources, task
   from flytekitplugins.dask import Dask, Scheduler, WorkerGroup

   @task(
     task_config=Dask(
         scheduler=Scheduler(
             limits=Resources(cpu="1", mem="2Gi"),  # Applied to the job pod
         ),
         workers=WorkerGroup(
             limits=Resources(cpu="4", mem="10Gi"), # Applied to the scheduler and worker pods
         ),
     ),
   )
   def my_dask_task():
      ...
   ```

### Images

By default, all components of the deployed `dask` job (job runner pod, scheduler pod and worker pods) will all use the
the image that was used while registering (this image should have `dask[distributed]` installed in its Python
environment). This helps keeping the Python environments of all cluster components in sync.
However, there is the possibility to specify different images for the components. This allows for use cases such as using
different images between tasks of the same workflow. While it is possible to use different images for the different
components of the `dask` job, it is not advised, as this can quickly lead to Python environments getting our of sync.

As the default behavior, all components of the deployed Dask job,
including the job runner pod, scheduler pod and worker pods,
will employ the image that was utilized during registration.
This image must have `dask[distributed]` installed in its Python environment,
ensuring consistency across the Python environments of all cluster components.

However, there exists the option to specify distinct images for these components.
This accommodation caters to scenarios where diverse images are required for tasks within the same workflow.
It is important to note that while it is technically possible to implement varying images
for different components of the dask job, this approach is not recommended.
Doing so can rapidly lead to discrepancies in Python environments.

```python
from flytekit import Resources, task
from flytekitplugins.dask import Dask, Scheduler, WorkerGroup

@task(
  task_config=Dask(
      scheduler=Scheduler(
          image="my_image:0.1.0",  # Will be used by the job pod
      ),
      workers=WorkerGroup(
          image="my_image:0.1.0", # Will be used by the scheduler and worker pods
      ),
  ),
)
def my_dask_task():
   ...
```

### Environment variables

Environment variables configured within the `@task` decorator will be propagated to all components of the Dask job,
encompassing the job runner pod, scheduler pod and worker pods.

```python
from flytekit import Resources, task
from flytekitplugins.dask import Dask

@task(
  task_config=Dask(),
  env={"FOO": "BAR"}  # Will be applied to all components
)
def my_dask_task():
   ...
```

### Labels and annotations

Labels and annotations specified within a {ref}`launch plan <launch_plan>` will be inherited by all components of the dask job,
which include the job runner pod, scheduler pod and worker pods.

```python
from flytekit import Resources, task, workflow, Labels, Annotations
from flytekitplugins.dask import Dask

@task(task_config=Dask())
def my_dask_task():
   ...

@workflow
def my_dask_workflow():
   my_dask_task()

# Labels and annotations will be passed on to all dask cluster components
my_launch_plan = my_dask_workflow.create_launch_plan(
  labels=Labels({"myexecutionlabel": "bar", ...}),
  annotations=Annotations({"region": "SEA", ...}),
)
```

### Interruptible tasks

The Dask backend plugin offers support for execution on interruptible nodes.
When `interruptible==True`, the plugin will incorporate the specified tolerations and node selectors into all worker pods.
It's important to be aware that neither the job runner nor the scheduler will be deployed on interruptible nodes.

```python
from flytekit import Resources, task, workflow, Labels, Annotations
from flytekitplugins.dask import Dask

@task(
  task_config=Dask(),
  interruptible=True,
)
def my_dask_task():
   ...
```

## Run the example on the Flyte cluster

To run the provided example on the Flyte cluster, use the following command:

```shell
$ pyflyte run --remote dask_example.py \
   hello_dask --size 1000
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/k8s-dask-plugin/dask-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/native-backend-plugins/k8s-dask-plugin/dask-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/k8s-dask-plugin/dask-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kfmpi-plugin ===

# MPI

In this section, you'll find a demonstration of running Horovod code with the Kubeflow MPI API.

## Horovod

[Horovod](http://horovod.ai/) stands as a distributed deep learning training framework compatible with
TensorFlow, Keras, PyTorch and Apache MXNet. Its primary objective is to enhance the speed and usability
of distributed deep learning through the implementation of ring-allreduce. This technique necessitates
just a few minimal modifications to the user's code, thereby simplifying the process of distributed deep learning.

## MPI (Message Passing Interface)

The Flyte platform employs the [Kubeflow training operator](https://github.com/kubeflow/training-operator),
to facilitate streamlined execution of all-reduce-style distributed training on Kubernetes.
This integration offers a straightforward interface for conducting distributed training through the utilization of MPI.

The combined power of MPI and Horovod can be harnessed to streamline the complexities of distributed training.
The MPI API serves as a convenient encapsulation to execute Horovod scripts, thereby enhancing the overall efficiency of the process.

## Install the plugin

Install the MPI plugin by running the following command:

```shell
$ pip install flytekitplugins-kfmpi
```

## Build a Docker image

The Dockerfile should include installation commands for various components, including MPI and Horovod.

```dockerfile
FROM ubuntu:focal
LABEL org.opencontainers.image.source https://github.com/flyteorg/flytesnacks

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
ENV DEBIAN_FRONTEND=noninteractive

# Install Python3.10 and other libraries
RUN apt-get update \
    && apt-get install -y software-properties-common \
    && add-apt-repository ppa:ubuntu-toolchain-r/test \
    && add-apt-repository -y ppa:deadsnakes/ppa \
    && apt-get install -y \
    build-essential \
    cmake \
    g++-7 \
    curl \
    git \
    wget \
    python3.10 \
    python3.10-venv \
    python3.10-dev \
    make \
    libssl-dev \
    python3-pip \
    python3-wheel \
    libuv1

# Virtual environment
ENV VENV /opt/venv
RUN python3.10 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

# Install wheel after venv is activated
RUN pip3 install wheel

# Install Open MPI
RUN wget --progress=dot:mega -O /tmp/openmpi-4.1.4-bin.tar.gz https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.4.tar.gz && \
    cd /tmp && tar -zxf /tmp/openmpi-4.1.4-bin.tar.gz && \
    mkdir openmpi-4.1.4/build && cd openmpi-4.1.4/build && ../configure --prefix=/usr/local && \
    make -j all && make install && ldconfig && \
    mpirun --version

# Allow OpenSSH to talk to containers without asking for confirmation
RUN mkdir -p /var/run/sshd
RUN cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new && \
    echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new && \
    mv /etc/ssh/ssh_config.new /etc/ssh/ssh_config

# Install Python dependencies
COPY requirements.in /root
RUN pip install -r /root/requirements.in

# Install TensorFlow
# In case you encounter the "The TensorFlow library was compiled to use AVX instructions, which are not present on your machine" error,
# you can resolve it by installing TensorFlow using the following RUN instruction:
# RUN wget https://tf.novaal.de/westmere/tensorflow-2.8.0-cp310-cp310-linux_x86_64.whl && pip install tensorflow-2.8.0-cp310-cp310-linux_x86_64.whl
# Otherwise:
RUN pip install tensorflow==2.8.0

# Enable GPU
# ENV HOROVOD_GPU_OPERATIONS NCCL
RUN HOROVOD_WITH_MPI=1 pip install --no-cache-dir horovod==0.28.1

# Copy the actual code
COPY . /root/

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
```

## Run the example on the Flyte cluster

To run the provided example on the Flyte cluster, use the following command:

```shell
$ pyflyte run --remote \
  --image ghcr.io/flyteorg/flytecookbook:kfmpi_plugin-latest \
  https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/kfmpi_plugin/kfmpi_plugin/mpi_mnist.py \
  horovod_training_wf
```

## MPI Plugin Troubleshooting Guide

This section covers common issues encountered during the setup of the MPI operator for distributed training jobs on Flyte.

**Worker Pods Failing to Start (Insufficient Resources)**

MPI worker pods may fail to start or exhibit scheduling issues, leading to job timeouts or failures. This often occurs due to resource constraints (CPU, memory, or GPU) in the cluster.

1. Adjust Resource Requests:
Ensure that each worker pod has sufficient resources. You can adjust the resource requests in your task definition:

```python
requests=Resources(cpu="<your_cpu_request>", mem="<your_mem_request>")
```

Modify the CPU and memory values according to your cluster's available resources. This helps prevent pod scheduling failures caused by resource constraints.

2. Check Pod Logs for Errors:
If the worker pods still fail to start, check the logs for any related errors:

```shell
$ kubectl logs <pod-name> -n <namespace>
```

Look for resource allocation or worker communication errors.

**Workflow Registration Method Errors (Timeouts or Deadlocks)**

If your MPI workflow hangs or times out, it may be caused by an incorrect workflow registration method.

1. Verify Registration Method:
    When using a custom image, refer to the Flyte documentation on [Registering workflows](https://www.union.ai/docs/v1/flyte/user-guide/development-cycle/running-your-code/page.md) to ensure you're following the correct registration method.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kfmpi-plugin/mpi-mnist ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/native-backend-plugins/kfmpi-plugin/mpi-mnist.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kfmpi-plugin/mpi-mnist/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kfpytorch-plugin ===

# PyTorch Distributed

The Kubeflow PyTorch plugin leverages the [Kubeflow training operator](https://github.com/kubeflow/training-operator)
to offer a highly streamlined interface for conducting distributed training using different PyTorch backends.

## Install the plugin

To use the PyTorch plugin, run the following command:

```shell
$ pip install flytekitplugins-kfpytorch
```

To enable the plugin in the backend, follow instructions outlined in the {ref}`deployment-plugin-setup-k8s` guide.

## Run the example on the Flyte cluster

To run the provided examples, use the following commands:

Distributed pytorch training:

```shell
$ pyflyte run --remote pytorch_mnist.py pytorch_training_wf
```

Pytorch lightning training:

```shell
$ pyflyte run --remote pytorch_lightning_mnist_autoencoder.py train_workflow
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kfpytorch-plugin/pytorch-mnist ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/native-backend-plugins/kfpytorch-plugin/pytorch-mnist.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kfpytorch-plugin/pytorch-mnist/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kfpytorch-plugin/pytorch-lightning-mnist-autoencoder ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/native-backend-plugins/kfpytorch-plugin/pytorch-lightning-mnist-autoencoder.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kfpytorch-plugin/pytorch-lightning-mnist-autoencoder/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/ray-plugin ===

# Ray

[KubeRay](https://github.com/ray-project/kuberay) is an open-source toolkit designed to facilitate the execution of
Ray applications on Kubernetes. It offers a range of tools that enhance the operational aspects of
running and overseeing Ray on Kubernetes.

Key components include:

- Ray Operator
- Backend services for cluster resource creation and deletion
- Kubectl plugin/CLI for CRD object management
- Seamless integration of Jobs and Serving functionality with Clusters

## Install the plugin

To install the Ray plugin, run the following command:

```shell
$ pip install flytekitplugins-ray
```

To enable the plugin in the backend, refer to the instructions provided in the [Configuyre Kubernetes Pluginsep](https://www.union.ai/docs/v1/flyte/deployment/flyte-plugins/kubernetes-plugins/page.md) guide.

## Implementation details

### Submit a Ray job to existing cluster

```python
import ray
from flytekit import task
from flytekitplugins.ray import RayJobConfig
@ray.remote
def f(x):
    return x * x
@task(
    task_config=RayJobConfig(
        address=<RAY_CLUSTER_ADDRESS>
        runtime_env={"pip": ["numpy", "pandas"]}
    )
)
def ray_task() -> typing.List[int]:
    futures = [f.remote(i) for i in range(5)]
    return ray.get(futures)
```

### Create a Ray cluster managed by Flyte and run a Ray Job on the cluster

```python
import ray
from flytekit import task
from flytekitplugins.ray import RayJobConfig, WorkerNodeConfig, HeadNodeConfig
@task(task_config=RayJobConfig(worker_node_config=[WorkerNodeConfig(group_name="test-group", replicas=10)]))
def ray_task() -> typing.List[int]:
    futures = [f.remote(i) for i in range(5)]
    return ray.get(futures)
```

## Run the example on the Flyte cluster

To run the provided example on the Flyte cluster, use the following command:

```shell
$ pyflyte run --remote ray_example.py \
          ray_workflow --n 10
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/ray-plugin/ray-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/native-backend-plugins/ray-plugin/ray-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/ray-plugin/ray-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/k8s-spark-plugin ===

# Spark

Flyte has the capability to directly execute Spark jobs on a Kubernetes Cluster.
The cluster handles the lifecycle, initiation and termination of virtual clusters.
It harnesses the open-source [Spark on Kubernetes operator](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator)
and can be enabled without requiring any service subscription.
This functionality is akin to operating a transient spark cluster
— a cluster type established specifically for a Spark job and taken down upon completion.

While these clusters are optimal for production workloads, they do come with the additional cost of setup and teardown.
In the Flyte environment, this cost is spread out over time due to the swiftness of creating pods compared to a full machine.
However, keep in mind that the performance might be impacted by the need to download Docker images, and starting a pod is not as immediate as running a process.

With Flytekit, you can compose PySpark code natively as a task.
The Spark cluster will be automatically configured using the specified Spark configuration.
The examples provided in this section offer a hands-on tutorial for writing PySpark tasks.

> [!NOTE]
> This plugin has been rigorously tested at scale, successfully managing more than 100,000 Spark Jobs through Flyte at Lyft.
> However, please bear in mind that this functionality requires a significant Kubernetes capacity and meticulous configuration.

For optimal results, we highly recommend adopting the
[multi-cluster mode](https://www.union.ai/docs/v1/flyte/deployment/flyte-configuration/performance/page.md).
Additionally, consider enabling resource quotas
for Spark Jobs that are both large in scale and executed frequently.

Nonetheless, it is important to note that extremely short-duration jobs might not be the best fit for this setup.
In such cases, utilizing a pre-spawned cluster could be more advantageous.
A job can be considered "short" if its runtime is less than 2 to 3 minutes.
In these situations, the cost of initializing pods might outweigh the actual execution cost.

## Why use Kubernetes Spark?

Managing Python dependencies can be challenging, but Flyte simplifies the process
by enabling easy versioning and management of dependencies through containers.
The Kubernetes Spark plugin extends the benefits of containerization to Spark without
requiring the management of specialized Spark clusters.

Pros:

1. Simple to get started, providing complete isolation between workloads.
2. Each job runs in isolation with its own virtual cluster, eliminating the complexities of dependency management.
3. Flyte takes care of all the management tasks.

Cons:

1. Short-running, bursty jobs may not be the best fit due to container overhead.
2. Interactive Spark capabilities are not available with Flyte Kubernetes Dask;
   instead, it is better suited for running adhoc and scheduled jobs.

## Implementation details

### Step 1: Deploy Spark plugin in the Flyte backend

Flyte Spark employs the Spark on K8s operator in conjunction with a bespoke
[Flyte Spark Plugin](https://pkg.go.dev/github.com/flyteorg/flyteplugins@v0.5.25/go/tasks/plugins/k8s/spark).

This plugin serves as a backend component and necessitates activation within your deployment.
To enable it, follow the instructions outlined in [Kubernetes Plugins](https://www.union.ai/docs/v1/flyte/deployment/flyte-plugins/kubernetes-plugins/page.md).

> [!NOTE]
> Refer to [this guide](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/gcp.md) to use GCP instead of AWS.

### Step 2: Environment Setup

Install `flytekitplugins-spark` using `pip` in your environment.

```shell
$ pip install flytekitplugins-spark
```

> [!NOTE]
> To enable Flyte to build the Docker image for you using `ImageSpec`, install `flytekitplugins-envd`.

Ensure that your Kubernetes cluster has sufficient resources available.
Depending on the resource requirements of your Spark job across the driver and executors,
you may need to adjust the resource quotas for the namespace accordingly.

### Step 3: Optionally, set up visibility

Whenever a Spark job is executed, you have the opportunity to access a Spark application UI link for
real-time job monitoring. Additionally, for past executions, you can leverage the
Spark history server to access the stored history of Spark executions.

Furthermore, Flyte offers the capability to generate direct links to both the Spark driver logs and individual Spark executor logs.

These Spark-related features, including the Spark history server and Spark UI links, are seamlessly displayed on the Flyte Console.
Their availability is contingent upon the following configuration settings:

#### Configure the Spark history link within the UI

To access the Spark history UI link within the Flyte Console,
it's necessary to configure a variable in the Spark section of the Flyteplugins configuration.
Here's an example of how to set it up:

```yaml
plugins:
  spark:
    spark-history-server-url: <root-url-forspark-history server>
```

You can explore various configuration options by referring to
[this link](https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/spark/config.go).

#### Configure the Spark application UI

To obtain a link for the ongoing Spark drivers and the Spark application UI,
you must set up Kubernetes to allow wildcard ingress access using `*.my-domain.net`.
Additionally, you should configure the Spark on Kubernetes operator to
establish a new ingress route for each application.

This can be achieved through the `ingress-url-format` command-line option of the Spark Operator.
You can find more details about this option in the source code
[here](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/d38c904a4dd84e849408153cdf4d7a30a7be5a07/main.go#L62).

#### Configure the Spark driver and executor logs

The logs can be configured by adjusting the `logs` configuration within the Spark plugin settings.
The Spark plugin utilizes the same default log configuration outlined in the section on [Configuring logging links in the UI](https://www.union.ai/docs/v1/flyte/deployment/flyte-configuration/configuring-logging-links-in-the-ui/page.md).

The SparkPlugin offers the capability to segregate user (Spark user code) and system (Spark core logs) logs,
thus enhancing visibility into Spark operations.
This is, however, feasible only if you can route the spark user logs separately from the core logs.
It's important to note that Flyte does not perform automatic log separation. You can review the configuration structure
[here](https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/spark/config.go#L31-L36).

- _Mixed_: Provides unseparated logs from the Spark driver (combining both user and system logs), following the standard structure of all log plugins.
  You can obtain links to the Kubernetes dashboard or a preferred log aggregator as long as it can generate standardized links.
- _User_: Offers logs from the driver with separation (subject to log separation availability).
- _System_: Covers logs from executors, typically without individual links for each executor;
  instead, it provides a prefix where all executor logs are accessible.
- _AllUser_: Encompasses all user logs across spark-submit, driver and executor.

Log configuration example:

```yaml
plugins:
  spark:
    logs:
      user:
        kubernetes-enabled: true
        kubernetes-url: <the existing k8s url you have in the main logs section>
      mixed:
        cloudwatch-enabled: true
        cloudwatch-template-uri: "https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=<LogGroupName>;prefix=var.log.containers.{{.podName}};streamFilter=typeLogStreamPrefix"
      system:
        cloudwatch-enabled: true
        cloudwatch-template-uri: "https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=<LogGroupName>;prefix=system_log.var.log.containers.{{.podName}};streamFilter=typeLogStreamPrefix"
      all-user:
        cloudwatch-enabled: true
        cloudwatch-template-uri: "https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=<LogGroupName>;prefix=var.log.containers.{{.podName}};streamFilter=typeLogStreamPrefix"
```

#### Additional configuration

The Spark plugin provides support for a range of extended configuration options.
For instance, if you wish to enable specific Spark features as defaults for all Spark applications,
you can apply default Spark configurations.

For more comprehensive information, please consult the [configuration structure](https://github.com/flyteorg/flyteplugins/blob/c528bb88937b4732c9cb5537ed8ea6943ff4fb56/go/tasks/plugins/k8s/spark/config.go#L24-L29).

## Run the examples on the Flyte cluster

To run the provided examples on the Flyte cluster, use any of the following commands:

```shell
$ pyflyte run --remote pyspark_pi.py my_spark
```

```shell
$ pyflyte run --remote dataframe_passing.py my_smart_structured_dataset
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/k8s-spark-plugin/dataframe-passing ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/native-backend-plugins/k8s-spark-plugin/dataframe-passing.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/k8s-spark-plugin/dataframe-passing/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/k8s-spark-plugin/pyspark-pi ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/native-backend-plugins/k8s-spark-plugin/pyspark-pi.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/k8s-spark-plugin/pyspark-pi/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kftensorflow-plugin ===

# TensorFlow Distributed

TensorFlow operator is useful to natively run distributed TensorFlow training jobs on Flyte.
It leverages the [Kubeflow training operator](https://github.com/kubeflow/training-operator).

## Install the plugin

To install the Kubeflow TensorFlow plugin, run the following command:

```shell
$ pip install flytekitplugins-kftensorflow
```

To enable the plugin in the backend, follow instructions outlined in the {ref}`deployment-plugin-setup-k8s` guide.

## Run the example on the Flyte cluster

To run the provided example on the Flyte cluster, use the following command:

```shell
$ pyflyte run --remote tf_mnist.py \
          mnist_tensorflow_workflow
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kftensorflow-plugin/tf-mnist ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/native-backend-plugins/kftensorflow-plugin/tf-mnist.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/native-backend-plugins/kftensorflow-plugin/tf-mnist/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins ===

# External service backend plugins

This section covers external service backend plugins.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/athena-plugin ===

# AWS Athena

## Executing Athena Queries

Flyte backend can be connected with Athena. Once enabled, it allows you to query AWS Athena service (Presto + ANSI SQL Support) and retrieve typed schema (optionally).
This plugin is purely a spec and since SQL is completely portable, it has no need to build a container. Thus this plugin example does not have any Dockerfile.

### Installation

To use the flytekit Athena plugin, simply run the following:

```shell
$ pip install flytekitplugins-athena
```

Now let's dive into the code.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/athena-plugin/athena ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/external-service-backend-plugins/athena-plugin/athena.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/athena-plugin/athena/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/aws-batch-plugin ===

# AWS Batch

## Executing Batch Job

Flyte backend can be connected with batch. Once enabled, it allows you to run regular task on AWS batch.

This section provides a guide on how to use the AWS Batch Plugin using flytekit python.

### Installation

To use the flytekit batch plugin simply run the following:

]
### Configuring the backend to get AWS Batch working

[Follow this guide to setting up the AWS Batch Plugin](https://www.union.ai/docs/v1/flyte/deployment/flyte-plugins/batch/page.md)

### Quick Start

This plugin allows you to run batch tasks on AWS and only requires you to change a few lines of code.
We can then move workflow execution from Kubernetes to AWS.

```python
from flytekitplugins.awsbatch import AWSBatchConfig

config = AWSBatch(
    parameters={"codec": "mp4"},
    platformCapabilities="EC2",
    propagateTags=True,
    retryStrategy={"attempts": 10},
    tags={"hello": "world"},
    timeout={"attemptDurationSeconds": 60},
)

@task(task_config=config)
def t1(a: int) -> str:
    return str(a)
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/aws-batch-plugin/batch ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/external-service-backend-plugins/aws-batch-plugin/batch.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/aws-batch-plugin/batch/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/flyteinteractive-plugin ===

# FlyteInteractive

FlyteInteractive provides interactive task development in a remote environment. This allows developers to leverage remote environment capabilities while accessing features like debugging, code inspection, and Jupyter Notebook, traditionally available in local IDEs.

Flyte tasks, designed as one-off jobs, require users to wait until completion to view results. These tasks are developed locally in a virtual environment before being deployed remotely. However, differences in data access, GPU availability, and dependencies between local and remote environments often lead to discrepancies, making local success an unreliable indicator of remote success. This results in frequent, tedious debugging cycles.

## Installation

To use the Flyte interactive plugin, run the following command:

```shell
$ pip install flytekitplugins-flyteinteractive
```

## Acknowledgement

This feature was created at LinkedIn and later donated to Flyte.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/flyteinteractive-plugin/jupyter ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/external-service-backend-plugins/flyteinteractive-plugin/jupyter.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/flyteinteractive-plugin/jupyter/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/flyteinteractive-plugin/vscode ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/external-service-backend-plugins/flyteinteractive-plugin/vscode.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/flyteinteractive-plugin/vscode/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/hive-plugin ===

# Hive

Flyte backend can be connected with various hive services. Once enabled it can allow you to query a hive service (e.g. Qubole) and retrieve typed schema (optionally).
This section will provide how to use the Hive Query Plugin using flytekit python

## Installation

To use the flytekit hive plugin simply run the following:

```shell
$ pip install flytekitplugins-hive
```

## No Need of a dockerfile

This plugin is purely a spec. Since SQL is completely portable there is no need to build a Docker container.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/hive-plugin/hive ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/external-service-backend-plugins/hive-plugin/hive.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/external-service-backend-plugins/hive-plugin/hive/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flyte-operators ===

# Flyte operators

This section covers Flyte operators.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flyte-operators/airflow-plugin ===

# Airflow Provider

The `airflow-provider-flyte` package provides an operator, a sensor, and a hook that integrates Flyte into Apache Airflow.
`FlyteOperator` is helpful to trigger a task/workflow in Flyte and `FlyteSensor` enables monitoring a Flyte execution status for completion.

The primary use case of this provider is to **scale Airflow for machine learning tasks using Flyte**.
With the Flyte Airflow provider, you can construct your ETL pipelines in Airflow and machine learning pipelines in Flyte
and use the provider to trigger machine learning or Flyte pipelines from within Airflow.

## Installation

```shell
$ pip install airflow-provider-flyte
```

All the configuration options for the provider are available in the provider repo's [README](https://github.com/flyteorg/airflow-provider-flyte#readme).

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/flyte-operators/airflow-plugin/airflow ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/flyte-operators/airflow-plugin/airflow.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/flyte-operators/airflow-plugin/airflow/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations ===

# Deprecated integrations

This section covers deprecated integrations.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/bigquery-plugin ===

# BigQuery plugin

> [!WARNING]
> This example code uses the legacy implementation of the BigQuery integration. We recommend using the [BigQuery connector](../../connectors/bigquery-connector/_index) instead.

This directory contains example code for the deprecated BigQuery plugin.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/bigquery-plugin/bigquery-plugin-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/deprecated-integrations/bigquery-plugin/bigquery-plugin-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/bigquery-plugin/bigquery-plugin-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/databricks-plugin ===

# Databricks plugin

> [!WARNING]
> This example code uses a legacy implementation of the Databricks integration. We recommend using the [Databricks connector](../../connectors/databricks-connector/_index) instead.

This directory contains example code for the deprecated Databricks plugin.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/databricks-plugin/databricks-plugin-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/deprecated-integrations/databricks-plugin/databricks-plugin-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/databricks-plugin/databricks-plugin-example/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/k8s-pod-plugin ===

# Kubernetes Pods

> [!IMPORTANT]
> This plugin is no longer needed and is here only for backwards compatibility. No new versions will be published after v1.13.x Please use the `pod_template` and `pod_template_name` arguments to `@task`.

Flyte tasks, represented by the `@task` decorator, are essentially single functions that run in one container.
However, there may be situations where you need to run a job with more than one container or require additional capabilities, such as:

- Running a hyper-parameter optimizer that stores state in a Redis database
- Simulating a service locally
- Running a sidecar container for logging and monitoring purposes
- Running a pod with additional capabilities, such as mounting volumes

To support these use cases, Flyte provides a Pod configuration that allows you to customize the pod specification used to run the task.
This simplifies the process of implementing the Kubernetes pod abstraction for running multiple containers.

> [!NOTE]
> A Kubernetes pod will not exit if it contains any sidecar containers (containers that do not exit automatically).
> You do not need to write any additional code to handle this, as Flyte automatically manages pod tasks.

## Installation

To use the Flytekit pod plugin, run the following command:

```shell
$ pip install flytekitplugins-pod
```

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/k8s-pod-plugin/pod ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/deprecated-integrations/k8s-pod-plugin/pod.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/k8s-pod-plugin/pod/

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/snowflake-plugin ===

# Snowflake plugin

> [!WARNING]
> This example code uses a legacy implementation of the Snowflake integration. We recommend using the [Snowflake connector](../../connectors/snowflake-connector/_index) instead.

This directory contains example code for the deprecated Snowflake plugin.

=== PAGE: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/snowflake-plugin/snowflake-plugin-example ===

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/deprecated-integrations/snowflake-plugin/snowflake-plugin-example.md
**HTML**: https://www.union.ai/docs/v1/flyte/integrations/deprecated-integrations/snowflake-plugin/snowflake-plugin-example/