Integrations

We currently offer a select few integrations that seamlessly connect Union Cloud to some of the most popular data and ML tools. However, we understand that every business has unique needs and workflows, which is why we also have a library of plugins that are available upon request.

These plugins are already a part of Flyte™, and we are more than happy to work with you to integrate them into your workflow. Simply fill out the request form and we will be happy to assist you.

DataFrames

Crunch data however you want. Automatically transition data from one dataframe type to the other using Structured Dataset.

HuggingFace

Use HuggingFace dataset as a native Flyte™ type.

pip install flytekitplugins-huggingface

Vaex

Visualize and explore big tabular datasets.

pip install flytekitplugins-vaex

Polars

Use Polars dataframe as a native Flyte™ type.

pip install flytekitplugins-polars
pip install flytekitplugins-modin

Data Validation

Validate data at every step of your Flyte™ workflow.

Pandera

Validate dataframe-like objects.

pip install flytekitplugins-pandera

Great Expectations

Validate your data with expectations.

pip install flytekitplugins-great-expectations

Databases & Data Warehouses

Manage and connect to databases and warehouses seamlessly.

SQLAlchemy

Execute SQL queries as Flyte™ tasks.

pip install flytekitplugins-sqlalchemy

Snowflake

Query a Snowflake service.

pip install flytekitplugins-snowflake
pip install flytekitplugins-hive

DuckDB

Run intricate analytical queries with DuckDB.

pip install flytekitplugins-duckdb

DoltHub

Apply git-like versioning to your SQL databases.

pip install flytekitplugins-dolt
pip install flytekitplugins-bigquery

Data Processing

Process and analyze your data with data-crunchers.

Databricks

Schedule, monitor and orchestrate Databricks jobs.

pip install flytekitplugins-spark

DBT

Transform data in your warehouses with DBT.

pip install flytekitplugins-dbt

Apache Spark

Run Spark jobs on ephemeral clusters.

pip install flytekitplugins-spark

AWS Athena

Query an AWS Athena service.

pip install flytekitplugins-athena

Feature Stores

Store, share and manage features for ML models.

Feast

Manage and serve ML features with Feast.

Model Training

Simplify the model training process.

AWS Sagemaker

Train ML models on Sagemaker from within Flyte™.

pip install flytekitplugins-awssagemaker

Distributed Model Training

Perform distributed model training to speed up the model development process.

Ray

Connect to Ray cluster to perform distributed model training and hyperparameter tuning.

pip install flytekitplugins-ray

Kubeflow PyTorch

Run distributed PyTorch training jobs.

pip install flytekitplugins-kfpytorch

Kubeflow TensorFlow

Run distributed TensorFlow training jobs.

pip install flytekitplugins-kftenorflow

Kubeflow MPI

Run distributed training with an MPI operator.

pip install flytekitplugins-kfmpi

Horovod

Run distributed deep learning workflows.

Dask

Run Dask jobs natively on a Kubernetes cluster.

pip install flytekitplugins-dask

Model Deployment

Streamline the model deployment process.

ONNX TensorFlow

Generate ONNX models from TensorFlow models.

pip install flytekitplugins-onnxtensorflow

ONNX Scikit Learn

Generate ONNX models from Scikit Learn models.

pip install flytekitplugins-onnxscikitlearn

ONNX PyTorch

Generate ONNX models from PyTorch models.

pip install flytekitplugins-onnxpytorch

Monitoring

Monitor data and models from within Flyte™.

Whylogs

Log any kind of data and generate summaries of datasets.

pip install flytekitplugins-whylogs

MLFlow

Track your machine learning metrics with MLFlow

pip install flytekitplugins-mlflow

Kubernetes-Native

Exercise greater control over Kubernetes resources.

Kubernetes Pods

Configure pods for arbitrary workloads.

pip install flytekitplugins-pod

Miscellaneous

Papermill

Execute Jupyter notebooks.

pip install flytekitplugins-papermill
pip install flytekitplugins-awsbatch

Build your own integration

Create your own integration and submit it to the Flyte™ repository.

Join our community to discuss and share ideas for integrations