* This page is part of the Union.ai v2 docs.
* Full index: https://www.union.ai/docs/v2/union/llms.txt
* Union.ai is a commercial superset of the open-source Flyte project and uses the same `flyte` SDK and CLI, so questions about Flyte are usually answered here too.
* Only refer to the open-source Flyte docs if you are explicitly looking for information about the open-source Flyte project, particularly the open-source Flyte backend. The open-source Flyte v2 docs are at https://www.union.ai/docs/v2/flyte/llms.txt
* These are the v2 docs. v1 is out of date and deprecated. Only refer to v1 docs if asked explicitly about v1. The v1 index is at https://www.union.ai/docs/v1/union/llms.txt

Plugins and connectors that extend Flyte to specialized compute, data systems, agent frameworks, and observability backends.

# Integrations

Flyte 2 is designed to be extensible by default. While the core platform covers the most common orchestration needs, many production workloads require specialized infrastructure, external services or execution semantics that go beyond the core runtime.

Flyte 2 exposes these capabilities through integrations.

Under the hood, integrations are implemented using Flyte 2's plugin system, which provides a consistent way to extend the platform without modifying core execution logic.

An integration allows you to declaratively enable new capabilities such as distributed compute frameworks or third-party services without manually managing infrastructure. You specify what you need, and Flyte takes care of how it is provisioned, used and cleaned up.

This page covers:

- The types of integrations Flyte 2 supports today
- How integrations fit into Flyte 2's execution model
- How to use integrations in your tasks
- The integrations available out of the box

If you need functionality that doesn't exist yet, Flyte 2's plugin system is intentionally open-ended. You can build and register your own integrations using the same architecture described here.

## Integration categories

Flyte 2 integrations fall into the following categories:

1. **Distributed compute**: Provision transient compute clusters to run tasks across multiple nodes, with automatic lifecycle management.
2. **Agentic AI**: Support for various common aspects of agentic AI applications.
3. **Software development tools**: Receive webhooks from GitHub, Slack, Linear, ClickUp, and Jira, and launch runs from them.
4. **Configuration**: Compose and pass hierarchical configuration objects between tasks, with type-safe schemas and CLI/YAML composition.
5. **Data transformation**: Run data transformation tools as Flyte tasks and capture their results.
6. **Experiment tracking**: Integrate with experiment tracking platforms for logging metrics, parameters, and artifacts.
7. **Data validation**: Enforce schema contracts on dataframes flowing between tasks, with automatic validation reports.
8. **Data types**: Add native support for additional file and dataframe types as task inputs and outputs.
9. **Connectors**: Stateless, long-running services that receive execution requests via gRPC and then submit work to external (or internal) systems.
10. **LLM Serving**: Deploy and serve large language models with an OpenAI-compatible API.
11. **Notebook execution**: Run parameterized Jupyter notebooks as typed Flyte tasks with cell-level reports.
12. **Observability**: Export task and agent telemetry to external tracing and observability backends.
13. **Profiling**: Profile GPU tasks and get the results back in the task's report.

## Distributed compute

Distributed compute integrations allow tasks to run on dynamically provisioned clusters. These clusters are created just-in-time, scoped to the task execution and torn down automatically when the task completes.

This enables large-scale parallelism without requiring users to operate or maintain long-running infrastructure.

### Supported distributed compute integrations

| Plugin                      | Description                                      | Common use cases                                       |
| --------------------------- | ------------------------------------------------ | ------------------------------------------------------ |
| [Ray](https://www.union.ai/docs/v2/union/integrations/ray.md)         | Provisions Ray clusters via KubeRay              | Distributed Python, ML training, hyperparameter tuning |
| [Spark](https://www.union.ai/docs/v2/union/integrations/spark.md)     | Provisions Spark clusters via Spark Operator     | Large-scale data processing, ETL pipelines             |
| [Dask](https://www.union.ai/docs/v2/union/integrations/dask.md)       | Provisions Dask clusters via Dask Operator       | Parallel Python workloads, dataframe operations        |
| [PyTorch](https://www.union.ai/docs/v2/union/integrations/pytorch.md) | Distributed PyTorch training with elastic launch | Single-node and multi-node training                    |

Each plugin encapsulates:

- Cluster provisioning
- Resource configuration
- Networking and service discovery
- Lifecycle management and teardown

From the task author's perspective, these details are abstracted away.

### How the plugin system works

At a high level, Flyte 2's distributed compute plugin architecture follows a simple and consistent pattern.

#### 1. Registration

Each plugin registers itself with Flyte 2's core plugin registry:

- **`TaskPluginRegistry`**: The central registry for all distributed compute plugins
- Each plugin declares:
  - Its configuration schema
  - How that configuration maps to execution behavior

This registration step makes the plugin discoverable by the runtime.

#### 2. Task environments and plugin configuration

Integrations are activated through a `TaskEnvironment`.

A `TaskEnvironment` bundles:

- A container image
- Execution settings
- A plugin configuration object enabled with `plugin_config`

The plugin configuration describes _what_ infrastructure or integration the task requires.

#### 3. Automatic provisioning and execution

When a task associated with a `TaskEnvironment` runs:

1. Flyte inspects the environment's plugin configuration
2. The plugin provisions the required infrastructure or integration
3. The task executes with access to that capability
4. Flyte cleans up all transient resources after completion

### Example: Using the Dask plugin

Below is a complete example showing how a task gains access to a Dask cluster simply by running inside an environment configured with the Dask plugin.

```python
from flyteplugins.dask import Dask, WorkerGroup
import flyte

# Define the Dask cluster configuration
dask_config = Dask(
    workers=WorkerGroup(number_of_workers=4)
)

# Create a task environment that enables Dask
env = flyte.TaskEnvironment(
    name="dask_env",
    plugin_config=dask_config,
    image=image,
)

# Any task in this environment has access to the Dask cluster
@env.task
async def process_data(data: list) -> list:
    from distributed import Client

    client = Client()  # Automatically connects to the provisioned cluster
    futures = client.map(transform, data)
    return client.gather(futures)
```

When `process_data` executes, Flyte performs the following steps:

1. Provisions a Dask cluster with 4 workers
2. Executes the task with network access to the cluster
3. Tears down the cluster once the task completes

No cluster management logic appears in the task code. The task only expresses intent.

### Key design principle

All distributed compute integrations follow the same mental model:

- You declare the required capability via configuration
- You attach that configuration to a task environment
- Tasks decorated with that environment automatically gain access to the capability

This makes it easy to swap execution backends or introduce distributed compute incrementally without rewriting workflows.

## Agentic AI

Agentic AI integrations let you run agents written in a third-party framework as durable Flyte tasks. You keep the framework's own idioms; Flyte supplies the runtime underneath, so tool calls become containerized child actions with their own resources, retries and caching, completed model turns replay instead of re-billing, and conversations persist across runs.

### Supported agentic AI integrations

| Plugin                              | Description                                                                                                              | Common use cases                                 |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------ |
| [Agent frameworks](https://www.union.ai/docs/v2/union/integrations/agents.md) | Adapters for ten agent SDKs, including OpenAI, Claude, Google ADK, Mistral, LangChain, LangGraph, CrewAI and Pydantic AI | Durable agents, tools as tasks, cross-run memory |
| [Code generation](https://www.union.ai/docs/v2/union/integrations/codegen.md) | LLM-driven code generation with automatic testing in sandboxes                                                           | Data processing, ETL, analysis pipelines         |
| [TypeSafe AI](https://www.union.ai/docs/v2/union/integrations/typesafe-ai.md)  | Typed, confidence-scored answers from a System One model, as task inputs and outputs                                     | Model-based I/O guards, intent routing, typed agent control flow |

## Software development tools

Software development tool integrations launch Flyte runs from events in GitHub, Slack, Linear, ClickUp, and Jira, such as a pull request opening or a ticket changing status. One app receives webhooks from any combination of these products, verifies each delivery, and launches one run per event. The receiver ships with Flyte in `flyte.extras.webhooks`; each package adds support for one product.

### Supported software development tool integrations

| Plugin | Description | Common use cases |
| ------ | ----------- | ---------------- |
| [GitHub](https://www.union.ai/docs/v2/union/integrations/software-development-tools/github.md) | Webhooks, human review gates, and GitHub App installation tokens | PR triage, gated merges and releases, coding agents |
| [Slack](https://www.union.ai/docs/v2/union/integrations/software-development-tools/slack.md) | Events, interactivity and slash commands in; messages and button approvals out | ChatOps, deploy approvals, run notifications |
| [Linear](https://www.union.ai/docs/v2/union/integrations/software-development-tools/linear.md) | Issue, comment, and project webhooks | Issue triage, backlog automation |
| [ClickUp](https://www.union.ai/docs/v2/union/integrations/software-development-tools/clickup.md) | Task, list, and goal webhooks | Ticket automation, status-driven workflows |
| [Jira](https://www.union.ai/docs/v2/union/integrations/software-development-tools/jira.md) | Jira Cloud webhooks, authenticated with a shared token | Issue triage, release bookkeeping |

See [Software development tools](https://www.union.ai/docs/v2/union/integrations/software-development-tools.md) for the shared event model, launch-once semantics, and scoping, and the [Software development lifecycle](https://www.union.ai/docs/v2/union/user-guide/software-development-lifecycle.md) guide for what to build on top.

## Data transformation

Data transformation integrations let you run transformation tools as Flyte tasks, compose them with Python tasks, and capture task reports and typed outputs.

### Supported data transformation integrations

| Plugin                  | Description                       | Common use cases                                      |
| ----------------------- | --------------------------------- | ----------------------------------------------------- |
| [dbt](https://www.union.ai/docs/v2/union/integrations/dbt.md)     | Run dbt CLI invocations as tasks  | Analytics engineering, data quality checks, reporting |

## Experiment tracking

Experiment tracking integrations let you log metrics, parameters, and artifacts to external tracking platforms during Flyte task execution.

### Supported experiment tracking integrations

| Plugin                               | Description                  | Common use cases                                 |
| ------------------------------------ | ---------------------------- | ------------------------------------------------ |
| [MLflow](https://www.union.ai/docs/v2/union/integrations/mlflow.md)            | MLflow experiment tracking   | Experiment tracking, autologging, model registry |
| [Weights and Biases](https://www.union.ai/docs/v2/union/integrations/wandb.md) | Weights & Biases integration | Experiment tracking and hyperparameter tuning    |

## Configuration

Configuration integrations let you compose and pass hierarchical configuration objects between Flyte tasks, with type-safe schemas and CLI/YAML composition.

### Supported configuration integrations

| Plugin                          | Description                                                       | Common use cases                                                                  |
| ------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| [OmegaConf](https://www.union.ai/docs/v2/union/integrations/omegaconf.md) | `DictConfig` / `ListConfig` as native task input and output types | Passing composed configs between tasks, structured configs, YAML-driven pipelines |
| [Hydra](https://www.union.ai/docs/v2/union/integrations/hydra.md)         | Hydra config composition and sweep submission for Flyte tasks     | YAML-driven experiment composition, grid and Bayesian sweeps, hardware presets    |

## Data validation

Data validation integrations enforce schema contracts on the dataframes flowing between tasks. They validate data at task boundaries, catch type and constraint violations early, and produce HTML reports visible in the Flyte UI.

### Supported data validation integrations

| Plugin                      | Description                                                | Common use cases                                            |
| --------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------- |
| [Pandera](https://www.union.ai/docs/v2/union/integrations/pandera.md) | Validates dataframes with pandera `DataFrameModel` schemas | Schema enforcement, data quality checks, validation reports |

## Data types

Data type integrations add native support for additional file and dataframe types as task inputs and outputs. They register typed encoders and decoders with Flyte's type engine, so you can annotate task signatures with the type directly and let Flyte handle serialization.

### Supported data type integrations

| Plugin                    | Description                                                  | Common use cases                                            |
| ------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------- |
| [Hugging Face](https://www.union.ai/docs/v2/union/integrations/huggingface.md) | `datasets.Dataset` / `datasets.IterableDataset` support, plus Hub datasets by reference | Training data from the Hub, tokenized dataset handoff, streamed evaluation |
| [JSONL](https://www.union.ai/docs/v2/union/integrations/jsonl.md)   | Typed `JsonlFile` / `JsonlDir` for streaming JSON Lines data | LLM dataset pipelines, event logs, large line-delimited I/O |
| [Lance](https://www.union.ai/docs/v2/union/integrations/lance.md)   | `lance.LanceDataset` as a streaming, multimodal dataframe format | Shuffled training data, vector search, point lookups     |
| [Polars](https://www.union.ai/docs/v2/union/integrations/polars.md) | Native `pl.DataFrame` / `pl.LazyFrame` support via Parquet   | High-performance dataframe ETL, feature engineering         |

## Connectors

Connectors are stateless, long-running services that receive execution requests via gRPC and then submit work to external (or internal) systems. Each connector runs as its own Kubernetes deployment, and is triggered when a Flyte task of the matching type is executed.

Although they normally run inside the data plane, you can also run connectors locally as long as the required secrets/credentials are present locally. This is useful because connectors are just Python services that can be spawned in-process.

Connectors are designed to scale horizontally and reduce load on the core Flyte backend because they execute _outside_ the core system. This decoupling makes connectors efficient, resilient, and easy to iterate on. You can even test them locally without modifying backend configuration, which reduces friction during development.

### Supported connectors

| Connector                         | Description                                 | Common use cases                         |
| --------------------------------- | ------------------------------------------- | ---------------------------------------- |
| [Snowflake](https://www.union.ai/docs/v2/union/integrations/snowflake.md)   | Run SQL queries on Snowflake asynchronously | Data warehousing, ETL, analytics queries |
| [BigQuery](https://www.union.ai/docs/v2/union/integrations/bigquery.md)     | Run SQL queries on Google BigQuery          | Data warehousing, ETL, analytics queries |
| [Databricks](https://www.union.ai/docs/v2/union/integrations/databricks.md) | Run PySpark jobs on Databricks clusters     | Large-scale data processing, Spark ETL   |
| [Slurm](https://www.union.ai/docs/v2/union/integrations/slurm.md)           | Run tasks as jobs on an existing Slurm cluster | HPC and GPU model training on Slurm-managed clusters |

### Creating a new connector

If none of the existing connectors meet your needs, you can build your own.

> [!NOTE]
> Connectors communicate via Protobuf, so in theory they can be implemented in any language.
> Today, only **Python** connectors are supported.

### Async connector interface

To implement a new async connector, extend `AsyncConnector` and implement the following methods, all of which must be idempotent:

| Method     | Purpose                                                     |
| ---------- | ----------------------------------------------------------- |
| `create`   | Launch the external job (via REST, gRPC, SDK, or other API) |
| `get`      | Fetch current job state (return job status or output)       |
| `delete`   | Delete / cancel the external job                            |
| `get_logs` | Stream paginated log lines to the Flyte UI                  |

To test the connector locally, the connector task should inherit from
[AsyncConnectorExecutorMixin](https://github.com/flyteorg/flyte-sdk/blob/1d49299294cd5e15385fe8c48089b3454b7a4cd1/src/flyte/connectors/_connector.py#L206). This mixin simulates how the Flyte 2 system executes asynchronous connector tasks, making it easier to validate your connector implementation before deploying it.

### Example: Batch job connector

The following example implements a connector that simulates submitting and polling an external batch job. Replace the mock logic with real API calls for your use case.

**Connector** (`my_connector/connector.py`):

```
import time
import uuid
from dataclasses import dataclass
from typing import Any, Dict, Optional

from flyteidl2.connector.connector_pb2 import (
    GetTaskLogsResponse,
    GetTaskLogsResponseBody,
    GetTaskLogsResponseHeader,
)
from flyteidl2.core.execution_pb2 import TaskExecution
from flyteidl2.logs.dataplane.payload_pb2 import LogLine, LogLineOriginator
from google.protobuf.timestamp_pb2 import Timestamp

from flyte import logger
from flyte.connectors import AsyncConnector, ConnectorRegistry, Resource, ResourceMeta

@dataclass
class BatchJobMetadata(ResourceMeta):
    job_id: str
    created_at: float

class BatchJobConnector(AsyncConnector):
    name = "Batch Job Connector"
    task_type_name = "batch_job"
    metadata_type = BatchJobMetadata

    async def create(self, task_template, inputs: Optional[Dict[str, Any]] = None, **kwargs) -> BatchJobMetadata:
        job_id = str(uuid.uuid4())[:8]
        logger.info(f"Submitted batch job {job_id}")
        return BatchJobMetadata(job_id=job_id, created_at=time.time())

    async def get(self, resource_meta: BatchJobMetadata, **kwargs) -> Resource:
        elapsed = time.time() - resource_meta.created_at
        if elapsed < 5:
            return Resource(phase=TaskExecution.RUNNING, message="Job in progress")
        return Resource(
            phase=TaskExecution.SUCCEEDED,
            message="Job completed",
            outputs={"result": f"output-from-{resource_meta.job_id}"},
        )

    async def delete(self, resource_meta: BatchJobMetadata, **kwargs):
        logger.info(f"Cancelled job {resource_meta.job_id}")

    async def get_logs(self, resource_meta: BatchJobMetadata, token: str = "", **kwargs):
        def line(message: str, ts: float) -> LogLine:
            t = Timestamp()
            t.FromSeconds(int(ts))
            return LogLine(timestamp=t, message=message, originator=LogLineOriginator.USER)

        start = resource_meta.created_at
        job_id = resource_meta.job_id
        pages = {
            "": GetTaskLogsResponseBody(lines=[
                line(f"[INFO] Job {job_id} submitted", start),
                line(f"[INFO] Job {job_id} started", start + 1),
            ]),
            "page-2": GetTaskLogsResponseBody(lines=[
                line(f"[INFO] Job {job_id} finished", start + 5),
            ]),
        }
        next_tokens = {"": "page-2", "page-2": ""}
        yield GetTaskLogsResponse(body=pages.get(token, GetTaskLogsResponseBody(lines=[])))
        next_token = next_tokens.get(token, "")
        if next_token:
            yield GetTaskLogsResponse(header=GetTaskLogsResponseHeader(token=next_token))

ConnectorRegistry.register(BatchJobConnector())
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/integrations/connectors/batch_job/connector.py*

**Task plugin** (`my_connector/task.py`):

```
from dataclasses import dataclass
from typing import Any, Dict, Optional, Type

from flyte.connectors import AsyncConnectorExecutorMixin
from flyte.extend import TaskTemplate
from flyte.models import NativeInterface, SerializationContext

@dataclass
class BatchJobConfig:
    timeout_seconds: int = 300

class BatchJobTask(AsyncConnectorExecutorMixin, TaskTemplate):
    _TASK_TYPE = "batch_job"

    def __init__(self, name: str, plugin_config: BatchJobConfig,
                 inputs: Optional[Dict[str, Type]] = None,
                 outputs: Optional[Dict[str, Type]] = None, **kwargs):
        super().__init__(
            name=name,
            interface=NativeInterface(
                {k: (v, None) for k, v in inputs.items()} if inputs else {},
                outputs or {},
            ),
            task_type=self._TASK_TYPE,
            image=None,
            **kwargs,
        )
        self.plugin_config = plugin_config

    def custom_config(self, sctx: SerializationContext) -> Optional[Dict[str, Any]]:
        return {"timeout_seconds": self.plugin_config.timeout_seconds}
```

*Source: https://github.com/unionai/unionai-examples/blob/main/v2/integrations/connectors/batch_job/task.py*

**Usage**:

```python
import flyte
from my_connector.task import BatchJobConfig, BatchJobTask

batch_job = BatchJobTask(
    name="my_batch_job",
    plugin_config=BatchJobConfig(timeout_seconds=60),
    inputs={"name": str},
    outputs={"result": str},
)

flyte.TaskEnvironment.from_task("batch-job-env", batch_job)
```

### Connector-level secrets

If your connector needs credentials (API keys, tokens) shared across all tasks, pass them as environment variables into the connector process.

Add secrets to `ConnectorEnvironment`:

```python
connector = flyte.app.ConnectorEnvironment(
    name="batch-job-connector",
    image=image,
    include=["my_connector"],
    secrets=[flyte.Secret(key="MY_API_KEY")],
)
```

Inside the connector, read the secret from the environment:

```python
import os

api_key = os.environ["MY_API_KEY"]
```

See [Secrets](https://www.union.ai/docs/v2/union/user-guide/tasks/task-configuration/secrets.md) for how to store and manage secrets.

### Deploy a custom connector

Deploy your connector as a long-running service using `flyte.app.ConnectorEnvironment`. Union handles building the image, pushing it, and keeping the service running: no manual Kubernetes configuration required.

See the **Connector app** guide (`user-guide/build-apps/connector-app`) for a complete walkthrough.

## LLM serving

LLM serving integrations let you deploy and serve large language models as Flyte apps with an OpenAI-compatible API. They handle model loading, GPU management, and autoscaling.

### Supported LLM serving integrations

| Plugin                                                          | Description                                         | Common use cases             |
| --------------------------------------------------------------- | --------------------------------------------------- | ---------------------------- |
| [SGLang](https://www.union.ai/docs/v2/union/user-guide/apps/native-app-integrations/sglang-app.md) | Deploy models with SGLang's high-throughput runtime | LLM inference, model serving |
| [vLLM](https://www.union.ai/docs/v2/union/user-guide/apps/native-app-integrations/vllm-app.md)     | Deploy models with vLLM's PagedAttention engine     | LLM inference, model serving |

For full setup instructions including multi-GPU deployment, model prefetching, and autoscaling, see the [SGLang app](https://www.union.ai/docs/v2/union/user-guide/apps/native-app-integrations/sglang-app.md) and [vLLM app](https://www.union.ai/docs/v2/union/user-guide/apps/native-app-integrations/vllm-app.md) pages.

## Notebook execution

Notebook execution integrations let you run Jupyter notebooks as first-class Flyte tasks with typed inputs and outputs, HTML reports surfaced in the Flyte UI, and the ability to call other Flyte tasks from within the notebook.

### Supported notebook execution integrations

| Plugin                          | Description                                                                                | Common use cases                                                                                     |
| ------------------------------- | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| [Papermill](https://www.union.ai/docs/v2/union/integrations/papermill.md) | Parameterize and execute `.ipynb` files via [papermill](https://papermill.readthedocs.io/) | Productionizing exploratory notebooks, cell-by-cell HTML reports, notebook-driven analysis pipelines |

## Observability

Observability integrations export telemetry from a Flyte run to an external backend. They understand that a durable run is several processes over time, so a run that crashes and resumes arrives as one trace rather than several, and steps replayed from the durable log are still recorded.

### Supported observability integrations

| Plugin                                                              | Description                                                                                 | Common use cases                                                     |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [OpenTelemetry](https://www.union.ai/docs/v2/union/integrations/opentelemetry.md)                             | Records tasks and traced steps as OpenTelemetry spans and exports them over OTLP            | Distributed tracing, debugging cross-service latency, durable traces |
| [Grafana Agent Observability](https://www.union.ai/docs/v2/union/integrations/grafana-agent-observability.md) | Sends agent generations, tool calls, token usage, and cost to Grafana, grouped by Flyte run | LLM cost tracking, prompt iteration, agent debugging                 |

Both carry trace context across task boundaries using Flyte's [custom context](https://www.union.ai/docs/v2/union/user-guide/tasks/task-programming/custom-context.md) primitive, so a run submitted from inside a caller's span joins that caller's trace.

## Profiling

Profiling integrations run a task under a profiler and attach the results to the run.

### Supported profiling integrations

| Plugin                                  | Description                                                              | Common use cases                                                    |
| --------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------- |
| [NVIDIA Nsight Systems](https://www.union.ai/docs/v2/union/integrations/nsight.md) | Runs a task under `nsys` and adds a GPU report and a downloadable trace | Finding slow CUDA kernels, data-loading stalls, distributed training bottlenecks |

{{< subpage-cards >}}

## Subpages

- [Agent frameworks](https://www.union.ai/docs/v2/union/integrations/agents.md) - Run agents from OpenAI, Claude, LangGraph, CrewAI, and more as durable Flyte tasks.
  - Supported frameworks
  - Two decorators
  - Quick start
  - What Flyte adds
  - Capability matrix
  - Choosing a framework
  - Next steps
- [Software development tools](https://www.union.ai/docs/v2/union/integrations/software-development-tools.md) - Receive webhooks from GitHub, Slack, Linear, ClickUp, and Jira, and turn them into Flyte runs.
  - Supported products
  - Quick start
  - The event model
  - Launch one run per event
  - Restrict which events launch runs
  - Test without an account
  - Deploy
  - Calling the product's API
  - Related features
  - Next steps
- [BigQuery](https://www.union.ai/docs/v2/union/integrations/bigquery.md) - Run SQL queries against Google BigQuery from a task, submitted asynchronously through the Jobs API.
  - Installation
  - Quick start
  - Configuration
  - `BigQueryConfig` parameters
  - `BigQueryTask` parameters
  - Authentication
  - Query templating
  - Supported input types
  - Parameterized query example
  - Retrieving query results
  - API reference
- [Code generation](https://www.union.ai/docs/v2/union/integrations/codegen.md) - Turn a natural-language prompt, sample data, and constraints into tested Python code.
  - Installation
  - Quick start
  - Two execution backends
  - LiteLLM (default)
  - Agent (Claude)
  - Providing data
  - Sample data
  - Schema and constraints
  - Inputs and outputs
  - Running generated code
  - One-shot execution with `result.run()`
  - Reusable task with `result.as_task()`
  - Error diagnosis
  - Durable execution
  - Replay logs
  - Caching
  - Non-determinism in agent mode
  - Observability
  - LiteLLM backend
  - Agent backend
  - Examples
  - Processing CSVs with different schemas
  - DataFrame analysis with constraints
  - Agent mode
  - Configuration
  - LiteLLM parameters
  - Image configuration
  - Skipping tests
  - Base packages
  - Best practices
  - API reference
  - `AutoCoderAgent` constructor
  - `generate()` parameters
  - `CodeGenEvalResult` fields
  - `CodeGenEvalResult` methods
- [Dask](https://www.union.ai/docs/v2/union/integrations/dask.md) - Run Dask jobs on a transient Kubernetes cluster provisioned per task execution.
  - When to use this plugin
  - Installation
  - Configuration
  - `Dask` parameters
  - `Scheduler` parameters
  - `WorkerGroup` parameters
  - Accessing the Dask client
  - Example
  - API reference
- [Databricks](https://www.union.ai/docs/v2/union/integrations/databricks.md) - Submit PySpark jobs written in a Flyte task to a Databricks cluster.
  - Installation
  - Quick start
  - Configuration
  - Spark fields (inherited)
  - Databricks-specific fields
  - `databricks_conf` structure
  - Authentication
  - Accessing the Spark session
  - API reference
- [DBT](https://www.union.ai/docs/v2/union/integrations/dbt.md) - Run dbt CLI commands as Flyte tasks and capture node results and reports.
  - Installation
  - Quick start
  - Commands and arguments
  - Chaining dbt tasks
  - Outputs
  - Reports and failures
  - Custom dbt callbacks
  - Reference
  - `DbtTask`
  - Invocation inputs
- [Grafana Agent Observability](https://www.union.ai/docs/v2/union/integrations/grafana-agent-observability.md) - Send agent generations, tool calls, token usage, and cost to Grafana Agent Observability.
  - Installation
  - What you get beyond agento11y on its own
  - Generations nest inside the task span
  - Flyte identity is bound onto agento11y's context
  - Durability is preserved end to end
  - Framework coverage
  - Configuration
  - `init()` parameters
  - Linking back from Grafana
  - Recording generations by hand
  - Content capture
  - Instrumenting a different backend
  - Limitations
  - Related
- [Hugging Face](https://www.union.ai/docs/v2/union/integrations/huggingface.md) - Pass Hugging Face datasets.Dataset and IterableDataset values between tasks as typed inputs and outputs.
  - When to use this plugin
  - Installation
  - What the plugin registers
  - Referencing a dataset on the Hub
  - It is a value, not just a default
  - Configs and splits
  - Config resolution
  - Omitting `split` concatenates everything
  - Reusing downloads across runs
  - Two caches, keyed differently
  - Reading only the columns you need
  - `Dataset` or `IterableDataset`
  - Passing datasets between tasks
  - Private and gated datasets
  - End-to-end: fine-tuning on IMDB
  - Running the examples
  - Common use cases
  - Hugging Face buckets as raw data storage
- [Hydra](https://www.union.ai/docs/v2/union/integrations/hydra.md) - Pass Hydra configuration trees into tasks as first-class typed inputs.
  - Installation
  - Requirements on tasks
  - A walkthrough config
  - Execution mode
  - Hydra launcher (`@hydra.main` scripts)
  - Python SDK
  - Single run
  - Grid sweep
  - Custom sweepers
  - Forwarding `flyte.with_runcontext` options
  - Flyte CLI (`flyte hydra run`)
  - Single run
  - Grid sweep
  - App-level vs Hydra-namespace overrides
  - `--follow` and `--no-wait`
  - Shell completion
  - Override grammar
  - Sweeps
  - Grid sweeps (BasicSweeper)
  - Bayesian / TPE sweeps (Optuna)
  - Sweep output directories
  - Task environment overrides
  - Prebuilt images
  - Applying overrides to child tasks
  - Renaming the task-env key
  - What `task_env` should not model
  - Structured configs (without YAML)
- [JSONL](https://www.union.ai/docs/v2/union/integrations/jsonl.md) - Typed JSON Lines inputs and outputs, for a single file or a directory of them.
  - When to use this plugin
  - Installation
  - Working with `JsonlFile`
  - Bulk and pre-serialized writes
  - Working with `JsonlDir`
  - Configuration and options
  - Compression
  - Write buffering
  - Prefetch
  - Error handling on read
  - Arrow batches
  - Common use cases
  - API reference
- [Lance](https://www.union.ai/docs/v2/union/integrations/lance.md) - Pass a Lance dataset between tasks as a lazy, typed DataFrame format.
  - Lance vs. Parquet
  - Which one should you use?
  - How much does it matter?
  - Installation
  - What the plugin registers
  - Other dataframe types
  - Passing a dataset between tasks
  - Return a live handle
  - Hand off by reference
  - Choosing between them
  - Arrow tables and the explicit `lance` format
  - Reading only what you need
  - Multimodal data
  - Streaming a shuffled epoch
  - Reading the labels without the images
  - Parallel reads across fragments
  - Indices and schema evolution
  - Object storage and credentials
  - Measuring the difference
  - Running the examples
  - Common use cases
- [MLflow](https://www.union.ai/docs/v2/union/integrations/mlflow.md) - Track experiments with MLflow from inside a task using the mlflow_run decorator.
  - Installation
  - Quick start
  - Autologging
  - Generic autologging
  - Framework-specific autologging
  - Run modes
  - Sharing a run across tasks
  - Creating independent runs
  - Nested runs
  - Workflow-level configuration
  - Per-task overrides
  - Configuration priority
  - Distributed training
  - MLflow UI links
  - Setup
  - Custom URL templates
  - Explicit links
  - Link behavior by run mode
  - Automatic Flyte tags
  - API reference
  - `mlflow_run` and `mlflow_config`
  - `get_mlflow_run`
  - `get_mlflow_context`
  - `Mlflow`
- [NVIDIA Nsight Systems](https://www.union.ai/docs/v2/union/integrations/nsight.md) - Profile a GPU task with NVIDIA Nsight Systems by adding one decorator, and get a GPU report and a downloadable trace back.
  - Installation
  - Quick start
  - Reading the profile
  - Profiling part of a task
  - Labeling the timeline with NVTX
  - Choosing what to trace
  - Profiling distributed training
  - Turning profiling off
  - Permissions
  - Troubleshooting
  - How it works
  - Reference
- [OmegaConf](https://www.union.ai/docs/v2/union/integrations/omegaconf.md) - Pass OmegaConf hierarchical configuration objects into tasks as typed inputs.
  - Installation
  - Quick start
  - When to use this plugin
  - Building a DictConfig
  - From a plain dict
  - From a YAML file
  - From a dataclass (structured config)
  - From a base config plus overrides
  - Variable interpolation
  - Nested and deeply structured configs
  - DictConfigs that contain lists
  - ListConfig as input and output
  - Lists of primitives
  - Building a schedule from another task
  - Nested lists (list of lists)
  - Lists of DictConfigs
  - Lists of dataclass instances
  - Structured configs
  - Basic structured config
  - Schema reconstruction in the receiving task
  - Required (`MISSING`) fields
  - Advanced field types
  - Merging overrides on top of a structured base
  - Embedding rich Python values inside a plain DictConfig
  - Reserved-looking keys
  - YAML reports
  - Wire format
  - End-to-end example
- [OpenTelemetry](https://www.union.ai/docs/v2/union/integrations/opentelemetry.md) - Turn a Flyte run into an OpenTelemetry trace, with each task and traced function a span.
  - Installation
  - Quick start
  - What becomes a span
  - Span attributes
  - Your own spans
  - What's next
- [Pandera](https://www.union.ai/docs/v2/union/integrations/pandera.md) - Validate dataframes against a Pandera DataFrameModel schema at task boundaries.
  - When to use this plugin
  - Installation
  - pandas
  - Polars
  - PySpark SQL
  - Defining schemas
  - Using schemas in tasks
  - Error handling with `ValidationConfig`
  - Image configuration
  - Pandas
  - Polars
  - PySpark SQL
  - Polars lazy frames
  - Examples
  - pandas
  - Polars
  - PySpark SQL
- [Papermill](https://www.union.ai/docs/v2/union/integrations/papermill.md) - Run a parameterized Jupyter notebook as a task and capture its outputs as typed values.
  - When to use this plugin
  - Installation
  - Quick start
  - Notebook setup
  - `parameters` cell
  - `outputs` cell
  - Inputs and outputs
  - Supported input types
  - Complex types: File, Dir, DataFrame
  - Outputs: single, multiple, none
  - Calling Flyte tasks from notebooks
  - Workflow patterns
  - Chaining notebooks
  - Mixing notebooks with regular tasks
  - Inline definition
  - Calling from sync vs. async tasks
  - Running a NotebookTask directly as the entrypoint
  - Reports and notebook artifacts
  - HTML report (default)
  - Notebook artifacts
  - Clean reports
  - Failure reports
  - Spark notebooks
  - Local testing
  - Execution options
  - `NotebookTask` reference
  - Helper functions
- [Polars](https://www.union.ai/docs/v2/union/integrations/polars.md) - Pass Polars DataFrame and LazyFrame values between tasks, serialized through Parquet.
  - When to use this plugin
  - Installation
  - Using Polars DataFrames
  - Using LazyFrames
  - Interoperating with `flyte.io.DataFrame`
  - Common use cases
  - API reference
- [PyTorch](https://www.union.ai/docs/v2/union/integrations/pytorch.md) - Run distributed PyTorch training on Kubernetes through the Kubeflow Training Operator.
  - When to use this plugin
  - Installation
  - Configuration
  - `Elastic` parameters
  - `RunPolicy` parameters
  - NCCL tuning parameters
  - Writing a distributed training task
  - Example
  - API reference
- [Ray](https://www.union.ai/docs/v2/union/integrations/ray.md) - Run Ray jobs on a transient KubeRay cluster provisioned per task execution.
  - When to use this plugin
  - Installation
  - Configuration
  - `RayJobConfig` parameters
  - `WorkerNodeConfig` parameters
  - `HeadNodeConfig` parameters
  - `AutoscalerOptionsConfig` parameters
  - Connecting to an existing cluster
  - Reusable Ray clusters
  - Reuse scope
  - Cleanup
  - Constraints
  - Examples
  - API reference
- [Slurm](https://www.union.ai/docs/v2/union/integrations/slurm.md) - Run Flyte tasks as Slurm jobs on an existing HPC or GPU cluster, submitted over SSH.
  - Installation
  - Task types
  - Quick start
  - Running an existing sbatch script
  - Outputs from a script task
  - Configuration
  - Scheduling
  - Container and execution
  - Connection
  - Container runtimes
  - Container images
  - Object storage from inside the job
  - Outputs and caching
  - Output types
  - Caching
  - Job state mapping
  - Inspecting a job
  - Deployment
  - Connector image
  - Data plane values
  - The connector's own object-storage access
  - Verifying the connector
  - Credentials
  - Network
  - Known gaps
  - Execution
  - Data and I/O
  - Operations
  - API reference
- [Snowflake](https://www.union.ai/docs/v2/union/integrations/snowflake.md) - Run SQL queries against Snowflake from a task, submitted asynchronously and polled for completion.
  - Installation
  - Quick start
  - Configuration
  - Required fields
  - Additional connection parameters
  - Authentication
  - Key-pair authentication
  - Password authentication
  - OAuth authentication
  - Query templating
  - Supported input types
  - Batched `INSERT` with list inputs
  - Parameterized `SELECT`
  - Multiple inputs
  - Retrieving query results
  - End-to-end example
- [Spark](https://www.union.ai/docs/v2/union/integrations/spark.md) - Run Apache Spark jobs on Kubernetes, with Flyte managing the full transient cluster lifecycle.
  - When to use this plugin
  - Installation
  - Configuration
  - `Spark` parameters
  - Pod templates
  - Accessing the Spark session
  - Overriding configuration at runtime
  - Example
  - API reference
- [TypeSafe AI](https://www.union.ai/docs/v2/union/integrations/typesafe-ai.md) - Run TypeSafe's System One model inside durable Flyte tasks, and carry its typed answers across task boundaries.
  - When to use this plugin
  - The three answer types
  - Setup
  - Declaring questions
  - Three ways to ask
  - Branching on the answers
  - Answers as task inputs and outputs
  - Two kinds of parallelism
  - A note on `IntEnum`
  - Next steps
- [Weights & Biases](https://www.union.ai/docs/v2/union/integrations/wandb.md) - Track experiments, visualize metrics, and run hyperparameter sweeps with Weights and Biases.
  - Installation
  - Quick start
  - What's next

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/integrations/_index.md
**HTML**: https://www.union.ai/docs/v2/union/integrations/