# Platform deployment
> This bundle contains all pages in the Platform deployment section.
> Source: https://www.union.ai/docs/v2/flyte/oss-deployment/
=== PAGE: https://www.union.ai/docs/v2/flyte/oss-deployment ===
# Platform deployment
> **π Note**
>
> An LLM-optimized bundle of this entire section is available at [`section.md`](section.md).
> This single file contains all pages in this section, optimized for AI coding agent context.
This section covers how to deploy **Flyte** to your own Kubernetes cluster using the
`flyte-binary` Helm chart.
Flyte ships as a single unified binary that bundles the runs service, the
task/actions controller, the data proxy, and the app service, served alongside the
Flyte web console. You point it at three things you provision yourself β a Kubernetes
cluster, a PostgreSQL database, and an object store β and it runs as one Deployment
that you scale vertically.
Read these pages in order:
1. **Deployment overview** β architecture and the external
dependencies you need to provision.
2. [Kind deployment](./kind-deployment/_index) β spin up the whole stack on a kind cluster
(on your machine or a DigitalOcean cloud VM) for evaluation,
including an optional self-contained authentication setup with
**Kind deployment > Set up local OIDC provider** (or an
**Kind deployment > Set up external OIDC provider**).
3. **AWS deployment** β a minimal `values.yaml`, the `helm install`
command, object-storage access, ingress, and a worked AWS/EKS example.
4. **Authentication and SSO** β securing the API and putting single
sign-on in front of the console.
5. **Enable app serving** β running long-running apps on Knative,
including how to install the Knative Serving prerequisite.
=== PAGE: https://www.union.ai/docs/v2/flyte/oss-deployment/overview ===
# Deployment overview
Before you install Flyte, it helps to understand what the deployment is made of and
which external dependencies you need to provide yourself.
## Architecture
Flyte runs as a **single unified binary**. One container image bundles all of the
backend components into one process:
| Component | Responsibility |
|---|---|
| Runs service | Accepts and stores run/task/workflow requests; owns the database. |
| Executor | Reconciles task runs onto Kubernetes. |
| Data proxy | Issues signed URLs for uploading and downloading data to the object store. |
| App service | Manages apps β long-running serving deployments β and their lifecycle. |
| Action service | Manages actions β short-lived task runs β and their lifecycle. |
| Cache service | Manages the cache for task runs and actions. |
At a high level, clients reach Flyte through a single HTTP ingress that fronts the console and the Flyte binary. The binary bundles the backend services into one process, reconciles task pods onto Kubernetes, and depends on an external database and object store:
```mermaid
flowchart TB
client["SDK & CLI"]
subgraph cluster["Kubernetes cluster"]
ingress["HTTP ingress"]
console["Console
(static SPA)"]
subgraph binary["Flyte binary (unified)"]
runs["Runs service"]
controller["Actions / task controller"]
dataproxy["Data proxy"]
apps["App service"]
end
pods["Task pods & app deployments"]
end
db[("PostgreSQL")]
store[("Object store
S3 / GCS / Azure")]
client -->|Connect over HTTP| ingress
ingress -->|/v2| console
ingress -->|flyteidl2.* + auth| binary
console -.->|API on same origin| ingress
runs --> db
controller --> pods
apps --> pods
dataproxy -->|signed URLs| store
client -.->|upload / download| store
pods --> store
```
A second image serves the web **console** as a static single-page application. It is
deployed as its own Deployment and Service but has no backend configuration of its
own β it talks to the Flyte API on the same origin (same ingress host) and is served
under a base path (default `/v2`, configurable via `console.basePath`).
### Networking
Flyte clients (the SDK and CLI) speak [buf Connect](https://connectrpc.com/) **over
HTTP**, so a **single HTTP ingress** serves the console, the API, and the
auth-discovery endpoints β there is no separate gRPC port to expose. The single
ingress routes, by path, to two backends:
- **Console Service**: `console.basePath` (default `/v2`, and `/v2/*`)
- **Flyte HTTP Service**: this service consists of:
- The `flyteidl2.*` Connect service paths, for example:
`/flyteidl2.project.ProjectService`, `/flyteidl2.workflow.RunService`,
`/flyteidl2.task.TaskService`, `/flyteidl2.dataproxy.DataProxyService`
- The auth-discovery paths, for example:
`/.well-known/oauth-authorization-server`, `/flyteidl2.auth.*`
## External dependencies
The chart deploys the Flyte binary and console, but it does **not** provision the
infrastructure they depend on. Provision these before installing, regardless of cloud:
### Kubernetes cluster
Use a [supported Kubernetes version](https://kubernetes.io/releases/version-skew-policy/#supported-versions).
Flyte does not constrain the provider or how you stand the cluster up β managed
offerings (EKS, GKE, AKS), self-managed clusters, and on-prem all work.
### Relational database
Flyte stores its persistent records in **PostgreSQL** (12 or newer). The binary reads
a single database configuration, rendered by the chart under `runs.database` from your
`configuration.database` values. Provision a database and a user before installing;
the chart can create the database password Secret for you, or you can mount it from
your own secret manager.
### Object store
Flyte uses an object store to hold task inputs/outputs, metadata, and uploaded data.
The chart supports **S3, GCS, and Azure Blob Storage**. You need at least one bucket,
and the identity Flyte runs as needs these minimum permissions on it:
- `GetObject`
- `PutObject`
- `ListBucket`
- `DeleteObject`
Configure it as `metadataContainer` β Flyte stores everything (metadata, task
inputs/outputs, and uploads) in that one bucket.
## Optional dependencies
Flyte runs without these, but integrates with them when present.
### Ingress controller
To expose Flyte beyond `kubectl port-forward`, you need an **ingress controller**
installed in the cluster: the chart creates a standard Kubernetes `Ingress` resource,
but a controller has to reconcile it into an actual load balancer.
On a managed cloud you don't need to run your own controller β each cloud has a native
one that reconciles the `Ingress` and provisions the cloud's own load balancer:
| Environment | Ingress controller | Provisions |
|---|---|---|
| AWS (EKS) | [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/) | Application Load Balancer (ALB) |
| GCP (GKE) | [GKE Ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress) (built in) | Google Cloud HTTP(S) Load Balancer |
| Azure (AKS) | [Application Gateway Ingress Controller (AGIC)](https://learn.microsoft.com/azure/application-gateway/ingress-controller-overview) | Azure Application Gateway |
| On-prem / any | [Contour](https://projectcontour.io/) or [Traefik](https://traefik.io/) (with MetalLB or a NodePort for the external address) | self-managed |
### DNS
For anything beyond local testing, point a DNS record at your ingress host so clients
and the console reach Flyte at a stable address instead of `localhost`.
### SSL/TLS
Use a valid certificate to secure traffic between clients and Flyte. On AWS this is
typically an ACM certificate attached to the ALB; elsewhere it is a TLS Secret
referenced from the Ingress.
## Container images
The chart deploys two images, configurable under `deployment.image` and
`console.image`. You normally don't need to set these β the chart ships with working
defaults:
| Image | Default repository |
|---|---|
| Flyte binary | `cr.flyte.org/flyteorg/flyte-binary-v2` |
| Console | `ghcr.io/unionai-oss/flyteconsole-v2` |
When you're ready, continue to the
[Kind deployment](./kind-deployment/_index) to try Flyte on a kind cluster,
or the [AWS deployment](./aws-deployment) guide for a real deployment.
=== PAGE: https://www.union.ai/docs/v2/flyte/oss-deployment/kind-deployment ===
# Kind deployment
This guide spins up a complete Flyte stack β the Flyte binary on a
[kind](https://kind.sigs.k8s.io/) cluster, backed by a hosted PostgreSQL and an
S3-compatible object store. kind runs anywhere Docker runs, so the same steps work on
**your own machine** or on a **cloud VM** (a DigitalOcean Droplet is shown here). Either
way it's a fast way to try Flyte without running a production-grade control plane.
> [!WARNING] For evaluation only
> This runs on a single-node kind cluster with static credentials (no workload identity),
> and the optional **Kind deployment > 7. Add authentication with a local IdP (optional)**
> uses a self-signed cert the SDK only accepts via `insecureSkipVerify`. Use it to try
> Flyte β not as a template for a production deployment. For that, see
> **AWS deployment**. On a cloud VM, remember the stack is reachable
> from the public internet: restrict ports `80`/`443` (and `22`) to your own IP with a
> [cloud firewall](https://docs.digitalocean.com/products/networking/firewalls/) while
> you evaluate.
## 1. Prerequisites
### Local machine
Install these on your machine:
- [Docker](https://docs.docker.com/get-docker/) (kind runs the cluster in containers)
- [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)
- [`kubectl`](https://kubernetes.io/docs/tasks/tools/)
- [`helm`](https://helm.sh/docs/intro/install/)
### DigitalOcean VM
Create a Droplet to host the cluster β kind needs a few GB of headroom, so pick at
least 4 vCPUs / 8 GB (e.g. `s-4vcpu-8gb`). Use the dashboard or
[`doctl`](https://docs.digitalocean.com/reference/doctl/):
```bash
doctl compute droplet create flyte-kind \
--image ubuntu-24-04-x64 --size s-4vcpu-8gb --region nyc1 \
--ssh-keys
```
SSH in and install Docker, kind, `kubectl`, and `helm` on the droplet:
```bash
ssh root@
# Docker
curl -fsSL https://get.docker.com | sh
# kind
curl -Lo /usr/local/bin/kind \
https://github.com/kubernetes-sigs/kind/releases/latest/download/kind-linux-amd64 \
&& chmod +x /usr/local/bin/kind
# kubectl
curl -Lo /usr/local/bin/kubectl \
"https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" \
&& chmod +x /usr/local/bin/kubectl
# helm
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
```
From here on, run every `kind`, `kubectl`, and `helm` command in this guide **on the
droplet**; only the SDK/CLI and browser run on your own machine.
## 2. Create the kind cluster
Create the cluster with two host-port mappings up front. kind fixes a cluster's port
mappings **at creation time** β they can't be added later β so map them now even
though they're only used in the optional auth step:
```bash
kind create cluster --name flyte --config - <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30080 # Traefik's web (HTTP) nodePort (step 7) β browser ingress
hostPort: 80 # reach the ingress at http://flyte.local
protocol: TCP
- containerPort: 30443 # Traefik's websecure (HTTPS) nodePort (step 7) β SDK auth
hostPort: 443 # reach the TLS ingress at https://flyte.local
protocol: TCP
EOF
```
This creates a single-node cluster and points your `kubectl` context at it. Confirm it
is up:
```bash
kubectl cluster-info --context kind-flyte
```
> [!NOTE] What the two mappings are for
> - **`30080 β 80`** lets the **browser** reach the Traefik ingress (plain HTTP) used by
> the optional **Kind deployment > 7. Add authentication with a local IdP (optional)** at
> `http://flyte.local`.
> - **`30443 β 443`** lets the **SDK/CLI** reach the Traefik ingress over **TLS** at
> `https://flyte.local`. The SDK only authenticates over HTTPS (see
> **Kind deployment > Letting the SDK/CLI authenticate (with auth enabled)**),
> so this mapping is required if you enable ingress auth *and* want to submit runs from
> the SDK. Harmless if you never enable auth.
>
> On a **DigitalOcean droplet** the same mappings bind to the droplet's **public IP** β
> in step 7, `flyte.local` points at that IP instead of `127.0.0.1`, and the two ports
> are open to the internet unless you restrict them with a cloud firewall (see the
> warning at the top).
>
> If you already created a plain `kind create cluster --name flyte` without these, delete
> it (`kind delete cluster --name flyte`) and recreate it with the config above.
## 3. Deploy the dependencies
kind runs only the Flyte binary; the database and object store are hosted. You need a
PostgreSQL (Supabase or another external/self-hosted instance) and an S3-compatible
object store (e.g. AWS S3, Cloudflare R2, etc). The two choices are independent. Each config
block below plugs into **Kind deployment > 4. Write the values file**.
```bash
kubectl create namespace flyte
```
### PostgreSQL
Create a project at [supabase.com](https://supabase.com/), then open **Project
Settings β Database β Connection string** and switch the tab to **Session pooler**.
Use that string, **not** the direct connection. (Another external or self-hosted
PostgreSQL works the same way β supply its host, database, user, and password, with
`sslmode` to match. For a DB on your host machine, use `host.docker.internal` as the
host. The database must already exist.)
> [!WARNING] Use the session pooler, not the direct connection
> Supabase's direct host (`db..supabase.co`) resolves to **IPv6 only**.
> A kind cluster is IPv4-only, so the Flyte pod can't reach it β `wait-for-db` passes
> (it only probes the port) but Flyte then crash-loops on `failed to connect`. The
> **session pooler** host (`aws--.pooler.supabase.com`) has IPv4, so use it.
> Use the **session** pooler on port `5432`, not the transaction pooler (`6543`) β
> Flyte's migrations need session semantics.
>
> Two things the pooler changes versus the direct string, both shown verbatim in the
> Session pooler tab β **copy them, don't guess**:
> - **Username carries the project ref**: `postgres.`, not bare `postgres`.
> - **The region must match your project's**: `aws--.pooler.supabase.com`.
> A mismatched region connects but is rejected with `tenant/user not found`.
```yaml
database:
postgres:
# From the "Session pooler" connection string β copy host and username verbatim.
host: aws-1-ap-northeast-1.pooler.supabase.com # <- your project's pooler host
port: 5432 # session mode (not 6543 transaction mode)
dbname: postgres # Supabase's default database
username: postgres. # pooler requires the ref-qualified username
password:
options: "sslmode=require" # Supabase requires TLS
```
### Object store
Both AWS S3 and Cloudflare R2 have publicly-resolvable endpoints, so the off-cluster
SDK uploads code bundles to presigned URLs directly β no nodePort or `signedURL`
override is needed.
### AWS S3
Create an S3 bucket in your AWS account and an IAM user (or access key) that can read
and write it.
```yaml
storage:
metadataContainer:
userDataContainer:
provider: s3
providerConfig:
s3:
region: # e.g. us-east-1
authType: accesskey
accessKey:
secretKey:
```
### Cloudflare R2
Create an R2 bucket and an R2 API token (Access Key ID + Secret) in the Cloudflare
dashboard. R2 is S3-compatible: point the `endpoint` at your account's R2 URL and use
`auto` for the region.
```yaml
storage:
metadataContainer:
userDataContainer:
provider: s3
providerConfig:
s3:
endpoint: https://.r2.cloudflarestorage.com
region: auto # R2 ignores region; "auto" is conventional
authType: accesskey
accessKey:
secretKey:
v2Signing: false
```
## 4. Write the values file
Create `values-local.yaml` by dropping in the `database` and `storage` blocks from
**Kind deployment > 3. Deploy the dependencies**. It uses static access keys (no cloud workload
identity) and skips the ingress β you'll reach Flyte with `kubectl port-forward`:
```yaml
# values-local.yaml β local kind deployment
fullnameOverride: flyte
configuration:
# ββ your PostgreSQL block (from step 3) β Supabase shown ββ
database:
postgres:
host: aws-1-ap-northeast-1.pooler.supabase.com # <- your project's pooler host
port: 5432
dbname: postgres
username: postgres.
password:
options: "sslmode=require"
# ββ your object-store block (from step 3) β AWS S3 shown ββ
storage:
metadataContainer:
userDataContainer:
provider: s3
providerConfig:
s3:
region:
authType: accesskey
accessKey:
secretKey:
serviceAccount:
create: true
annotations: {} # no IRSA/Workload Identity locally
ingress:
create: false # reach Flyte via port-forward instead
```
Swap in the R2 storage block if you chose Cloudflare R2.
## 5. Install Flyte
```bash
helm repo add flyteorg https://flyteorg.github.io/flyte
helm repo update
helm install flyte flyteorg/flyte-binary -n flyte -f values-local.yaml
```
Watch the rollout:
```bash
kubectl -n flyte rollout status deploy/flyte
kubectl -n flyte get pods
```
A `wait-for-db` init container blocks startup until PostgreSQL is reachable, so a pod
stuck in `Init` usually means the database isn't up yet or the host/credentials are
wrong.
## 6. Access Flyte
Make the API reachable at `localhost:8090` on the machine where you run the SDK/CLI:
### Local machine
Port-forward the API service:
```bash
kubectl -n flyte port-forward service/flyte-http 8090:8090
```
### DigitalOcean VM
The port-forward runs on the droplet, so tunnel it back to your own machine over SSH.
This single command starts the port-forward on the droplet *and* exposes it at
`localhost:8090` locally:
```bash
ssh -L 8090:localhost:8090 root@ \
kubectl -n flyte port-forward service/flyte-http 8090:8090
```
Keep it running while you use the SDK/CLI β everything below works identically through
the tunnel.
```bash
# In another terminal β list projects over the Connect (HTTP) API:
curl -s -X POST \
http://localhost:8090/flyteidl2.project.ProjectService/ListProjects \
-H 'Content-Type: application/json' -d '{}'
```
A JSON response (rather than a connection error) confirms the binary is up and talking
to its database.
### Submitting runs from the SDK
Point the SDK at the API forward (or tunnel) you just started. Write
`~/.flyte/config.yaml`:
```yaml
# ~/.flyte/config.yaml
admin:
endpoint: dns:///localhost:8090 # the port-forwarded API
insecure: True # plain HTTP, no TLS
task:
org: local
domain: development
project: flytesnacks
```
The code-bundle upload needs no extra setup β only the one API port-forward is required,
never a second one for the object store. The S3/R2 endpoint is publicly resolvable, so
the SDK uploads to the presigned URL directly.
## 7. Add authentication with a local IdP (optional)
The cloud worked examples in **AWS deployment** gate the console with
OIDC single sign-on at the load balancer β on AWS that's the ALB, configured through
`alb.ingress.kubernetes.io/auth-*` annotations. Those annotations are instructions to
the *AWS Load Balancer Controller* and do nothing on kind, which has no ALB.
The pattern is the same on kind; only the ingress controller changes. Here you run
[Traefik](https://doc.traefik.io/traefik/) and delegate auth to
[oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/): Traefik intercepts each
request through a `ForwardAuth` middleware, asks oauth2-proxy whether the caller is
logged in, and redirects to your IdP if not. oauth2-proxy plays the role the ALB played
β an auth proxy at the edge.
For the IdP itself, this step runs [Dex](https://dexidp.io/) **inside the same kind
cluster**, so you can test the whole authentication flow with no cloud account and no
real users β the Dex deployment itself is covered in
**Kind deployment > Set up local OIDC provider**.
> [!NOTE] Prefer an external provider?
> To validate against a real OIDC provider (Okta, Google, Auth0, β¦) instead of the
> in-cluster Dex, see **Kind deployment > Set up external OIDC provider**. It follows
> this same step with the Dex-specific parts swapped out.
### Install the ingress controller
Install Traefik with its Helm chart and expose **both** entrypoints on the kind node's
nodePorts β `web` (HTTP) on `30080` and `websecure` (HTTPS) on `30443`, the ports the
cluster maps to host `80` and `443` (see the warning in
**Kind deployment > 2. Create the kind cluster**). The browser uses HTTP; **the SDK needs HTTPS** β
expose both up front so you don't have to reinstall later.
```bash
helm repo add traefik https://traefik.github.io/charts
helm repo update
helm install traefik traefik/traefik -n traefik --create-namespace \
--set "service.type=NodePort" \
--set "ports.web.nodePort=30080" \
--set "ports.websecure.nodePort=30443"
```
Traefik installs its CRDs (including `Middleware`), registers a `traefik` IngressClass,
and serves a **default self-signed certificate** on `websecure`. If your cluster lacks
the `30080 β 80` / `30443 β 443` mappings from **Kind deployment > 2. Create the kind cluster**,
Traefik is unreachable at `flyte.local` and you'll need to recreate the cluster.
#### Replace the default cert with one for `flyte.local`
Traefik's built-in cert is rejected by the SDK for two reasons, hit in sequence if you
only set `insecureSkipVerify` later:
- Its SAN is `*.traefik.default`, so the hostname check fails with
`certificate not valid for name "flyte.local"`. The SDK validates the SAN **even with
`insecureSkipVerify`** (that flag relaxes CA trust, not the hostname).
- The SDK implements `insecureSkipVerify` by fetching the server's cert chain and
**pinning it as the CA**. A bare self-signed leaf then fails with `CaUsedAsEndEntity` β
rustls won't use a leaf cert as a CA.
The fix is a **two-tier chain**: a self-signed root CA that signs a leaf carrying
`SAN=flyte.local`. Traefik serves `leaf + CA`; the SDK pins the root as CA and validates
the leaf against it. (You can skip this if you only need the browser console β it's the
SDK that requires a trusted-chain TLS cert.)
```bash
# 1. Root CA (CA:TRUE)
openssl req -x509 -nodes -newkey rsa:2048 -days 3650 \
-keyout ca.key -out ca.crt -subj "/CN=flyte-local-ca" \
-addext "basicConstraints=critical,CA:TRUE" \
-addext "keyUsage=critical,keyCertSign,cRLSign"
# 2. Leaf key + CSR
openssl req -nodes -newkey rsa:2048 -keyout leaf.key -out leaf.csr \
-subj "/CN=flyte.local"
# 3. CA signs the leaf (CA:FALSE, SAN=flyte.local, server auth)
openssl x509 -req -in leaf.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 3650 -out leaf.crt \
-extfile <(printf "subjectAltName=DNS:flyte.local\nbasicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth")
# 4. Secret holds the full chain (leaf + CA) so Traefik serves both
cat leaf.crt ca.crt > fullchain.crt
kubectl -n traefik create secret tls flyte-local-tls \
--cert=fullchain.crt --key=leaf.key
```
Point Traefik's cluster-wide default cert at that secret with a `TLSStore` named
`default` (the only name Traefik honours for the fallback cert), then restart Traefik:
```bash
kubectl apply -f - <<'EOF'
apiVersion: traefik.io/v1alpha1
kind: TLSStore
metadata:
name: default
namespace: traefik
spec:
defaultCertificate:
secretName: flyte-local-tls
EOF
kubectl -n traefik rollout restart deploy/traefik
```
> [!NOTE] A self-signed CA still needs `insecureSkipVerify`
> This cert chains to a **self-signed root the SDK doesn't trust**, so the SDK config
> **Kind deployment > Point the SDK at the HTTPS endpoint** still sets `insecureSkipVerify` to accept
> it β inherent to any private/self-signed CA (AWS Private CA included), not a Traefik
> limitation. To drop it entirely, either install `ca.crt` into each client's system
> trust store, or front Flyte with a **publicly-resolvable domain** and a publicly-trusted
> cert (e.g. Traefik's ACME / Let's Encrypt resolver β the cloud ALB's ACM equivalent).
> Neither is possible for a purely-local `flyte.local`. On a **DigitalOcean droplet** the
> domain route *is* attainable β point a real DNS record at the droplet and substitute
> that hostname for `flyte.local` throughout β but this guide sticks with the self-signed
> chain so the local and cloud-VM paths stay identical.
### Deploy Dex
Deploy Dex into the cluster and route its issuer URL (`http://flyte.local/dex`)
through Traefik, following
**Kind deployment > Set up local OIDC provider**. Come back here once the discovery
document resolves through Traefik:
```bash
curl -s http://flyte.local/dex/.well-known/openid-configuration | head
# β a JSON document with "issuer":"http://flyte.local/dex"
```
### Deploy oauth2-proxy
Give oauth2-proxy the Dex client details and a random cookie secret. `--set-xauthrequest`
makes it emit the `X-Auth-Request-*` headers Traefik forwards downstream, and
`--reverse-proxy` tells it to trust the forwarded host/proto from Traefik.
`skip-jwt-bearer-tokens`, `oidc-extra-audience`, and `bearer-token-login-fallback` let
the **SDK/CLI** authenticate too (not just the browser) β set them now so you don't have
to upgrade later. The `hostAliases` setting is the one addition an in-cluster Dex needs β
see the warning below for why:
```bash
# 32-byte cookie secret. Must decode to 16/24/32 bytes β head -c 32 trims the
# base64 string, since a raw 44-char value fails oauth2-proxy's length check.
COOKIE_SECRET=$(openssl rand -base64 32 | head -c 32)
# Dex's issuer is flyte.local, which the pod can't otherwise resolve β point it at Traefik.
TRAEFIK_IP=$(kubectl -n traefik get svc traefik -o jsonpath='{.spec.clusterIP}')
helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm repo update
helm install oauth2-proxy oauth2-proxy/oauth2-proxy -n flyte \
--set config.clientID='oauth2-proxy' \
--set config.clientSecret='oauth2-proxy-secret' \
--set config.cookieSecret="$COOKIE_SECRET" \
--set extraArgs.provider=oidc \
--set extraArgs.oidc-issuer-url='http://flyte.local/dex' \
--set extraArgs.upstream='static://202' \
--set extraArgs.reverse-proxy='true' \
--set extraArgs.set-xauthrequest='true' \
--set extraArgs.email-domain='*' \
--set extraArgs.cookie-secure='false' \
--set extraArgs.skip-jwt-bearer-tokens='true' \
--set extraArgs.oidc-extra-audience='flytectl' \
--set extraArgs.bearer-token-login-fallback='false' \
--set "hostAliases[0].ip=$TRAEFIK_IP" \
--set "hostAliases[0].hostnames[0]=flyte.local"
```
A few of these flags deserve explanation:
- `cookie-secure='false'` β the browser flow runs over local HTTP, not HTTPS.
- `skip-jwt-bearer-tokens='true'` β the browser path uses the session cookie, but the
**SDK** sends an `Authorization: Bearer` JWT instead. This makes oauth2-proxy verify
that JWT against Dex's JWKS and pass it through.
- `oidc-extra-audience='flytectl'` β must list the **public client ID** the SDK uses
(the `flytectl` static client from the **Kind deployment > Set up local OIDC provider**;
also the `flyteClient.clientId` you'll advertise in `authMetadata` below) β its
tokens carry that as their audience.
The flag is **singular** β `oidc-extra-audiences` (plural) is not valid and
crash-loops oauth2-proxy with `unknown flag`.
- `bearer-token-login-fallback='false'` β an invalid token gets a `403`, not an HTML
login page the SDK can't parse.
Without the SDK flags, the SDK is rejected and `flyte.run` fails the upload with
`Unauthorized`. See
**Kind deployment > Letting the SDK/CLI authenticate (with auth enabled)** for
the rest of the SDK setup.
> [!WARNING] Why `hostAliases` is required for Dex
> Unlike an external IdP, Dex's issuer is `flyte.local` β a name that resolves on your
> host (via `/etc/hosts`) but **not inside the cluster**, where CoreDNS doesn't know it.
> Without help, oauth2-proxy hangs on `Performing OIDC Discovery...` at startup and
> `CrashLoopBackOff`s. The `hostAliases` flag above adds `flyte.local β Traefik's
> ClusterIP` to the pod's `/etc/hosts`, so `flyte.local/dex` resolves to the same issuer
> from both the pod and the browser, as OIDC requires.
>
> This pins the current ClusterIP. If the Traefik service is recreated with a new IP,
> `helm upgrade` oauth2-proxy with the new value.
### Create the ForwardAuth middleware
Two Traefik `Middleware` objects: one sends each request to oauth2-proxy for a verdict
and forwards the identity headers; the other catches the `401` an unauthenticated
request gets and redirects it to the oauth2-proxy sign-in page.
```yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: oauth2-auth
namespace: flyte
spec:
forwardAuth:
address: http://oauth2-proxy.flyte.svc.cluster.local/oauth2/auth
trustForwardHeader: true
# Forwarded to Flyte; these feed executed_by run attribution.
authResponseHeaders:
- X-Auth-Request-User
- X-Auth-Request-Email
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: oauth2-signin
namespace: flyte
spec:
errors:
status:
- "401"
service:
name: oauth2-proxy
port: 80
query: "/oauth2/sign_in?rd={url}"
```
### Create the Flyte ingress with the middleware
Re-render Flyte with the ingress enabled and pointed at Traefik. The
`traefik.ingress.kubernetes.io/router.middlewares` annotation chains both middlewares
onto every route β this is the Traefik equivalent of the ALB's `auth-type: oidc`. The
reference format is `-@kubernetescrd`:
```yaml
# add to values-local.yaml, replacing the `ingress.create: false` block
ingress:
create: true
host: flyte.local # add "127.0.0.1 flyte.local" to /etc/hosts
ingressClassName: traefik
httpAnnotations:
traefik.ingress.kubernetes.io/router.middlewares: flyte-oauth2-signin@kubernetescrd,flyte-oauth2-auth@kubernetescrd
```
You also need a route that sends `/oauth2` to oauth2-proxy itself so the sign-in
redirect resolves. Apply it alongside the Flyte release:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: oauth2-proxy
namespace: flyte
spec:
ingressClassName: traefik
rules:
- host: flyte.local
http:
paths:
- path: /oauth2
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
number: 80
```
After `helm upgrade flyte β¦ -f values-local.yaml`, make `flyte.local` resolve to
Traefik from the machine your **browser and SDK** run on (do this once):
### Local machine
Traefik's node ports are mapped to your own host, so point `flyte.local` at loopback:
```bash
echo "127.0.0.1 flyte.local" | sudo tee -a /etc/hosts
```
### DigitalOcean VM
Traefik's node ports are bound to the droplet's public IP, so point `flyte.local` there
in **your own machine's** `/etc/hosts` (not the droplet's):
```bash
echo " flyte.local" | sudo tee -a /etc/hosts
```
Every other `flyte.local` reference in this step (Dex issuer, redirect URIs, cert SAN,
ingress host) stays exactly the same β only this mapping differs. Alternatively, create
a real DNS A record pointing at the droplet and substitute that hostname everywhere
`flyte.local` appears.
Opening the console by raw IP won't work in its place β Traefik has no route for that
host, and the OIDC issuer is `flyte.local`, so login fails on an issuer mismatch. Then
open `http://flyte.local/v2` β Traefik bounces you through Dex and back into the console.
#### Split the API and discovery paths off the browser middleware
This gates the **browser** correctly, but the same `oauth2-signin` redirect on **every**
path breaks the SDK (the cloud walkthrough avoids this by splitting into three ingresses β
`ingress` / `apiJwtIngress` / `wellknownIngress` β since ALB can't mix cookie-OIDC and JWT
auth on one). Two path groups need different handling:
- **Auth-discovery** (`AuthMetadataService`, `IdentityService`) β the SDK reads these
*before* it has a token, so they must **bypass auth**. Gated, they return a `text/plain`
401 that ConnectRPC reports as `UNAVAILABLE`, and the SDK never starts login.
- **The `flyteidl2.*` API** β needs `oauth2-auth` (Bearer validation) but **not**
`oauth2-signin`, so an unauthenticated call gets a clean gRPC 401 the SDK retries after
login, not sign-in HTML.
Add two higher-priority `IngressRoute`s β Traefik matches the highest `priority` first,
so these win over the `flyte-http` Ingress for their paths:
```bash
kubectl apply -f - <<'EOF'
# Discovery β highest priority, NO middleware. The SDK reads these before it has a
# token; they must reach Flyte directly so it returns the real metadata (issuer,
# client ID), not oauth2-proxy's unparseable 401. (Equivalent of wellknownIngress.)
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: flyte-auth-discovery
namespace: flyte
spec:
entryPoints: [web, websecure]
routes:
- kind: Rule
priority: 300
match: Host(`flyte.local`) && (PathPrefix(`/flyteidl2.auth.AuthMetadataService`) || PathPrefix(`/flyteidl2.auth.IdentityService`))
services:
- name: flyte-http
port: 8090
scheme: h2c # gRPC over cleartext HTTP/2 to the backend
---
# API β oauth2-auth only (validates Bearer tokens), no oauth2-signin redirect, so an
# unauthenticated call gets a clean 401 the SDK can act on. (Equivalent of apiJwtIngress.)
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: flyte-api-bearer
namespace: flyte
spec:
entryPoints: [web, websecure]
routes:
- kind: Rule
priority: 100
match: Host(`flyte.local`) && PathPrefix(`/flyteidl2.`)
middlewares:
- name: oauth2-auth
services:
- name: flyte-http
port: 8090
scheme: h2c
EOF
```
Now `flyte.local` routes three ways by precedence: discovery (300) bypasses auth, the
API (100) requires a Bearer token, and everything else β the `/v2` console β falls
through to the `flyte-http` Ingress with the full browser middleware chain. Verify the
discovery path returns JSON rather than oauth2-proxy's 401:
```bash
curl -s -X POST --resolve flyte.local:443:127.0.0.1 -k \
https://flyte.local/flyteidl2.auth.AuthMetadataService/GetPublicClientConfig \
-H 'Content-Type: application/json' -d '{}' | head -c 120
# β {"clientId":"flytectl", ...} (JSON, not "Unauthorized")
```
### Verify the browser flow
With Dex, oauth2-proxy, and the Flyte ingress all in place, check the flow from the
command line before opening a browser. These use `curl --resolve` to point `flyte.local`
at the Traefik node port, so they work even without the `/etc/hosts` entry (the
browser still needs it):
> [!NOTE] `--resolve` on a DigitalOcean droplet
> The `--resolve flyte.local::127.0.0.1` flags in this guide assume the curl runs
> on the machine hosting kind. That's still true on a droplet β run them in your SSH
> session and `127.0.0.1` works as-is. To run them from your own machine instead,
> substitute the droplet's public IP for `127.0.0.1`.
```bash
# 1. The console is gated β an unauthenticated request is rejected by the auth middleware:
curl -s -o /dev/null -w "%{http_code}\n" --resolve flyte.local:80:127.0.0.1 \
http://flyte.local/v2
# β 401 (the oauth2-auth ForwardAuth middleware rejects it; a browser is then
# redirected to sign-in by the oauth2-signin error middleware)
# 2. The sign-in page is served:
curl -s -o /dev/null -w "%{http_code}\n" --resolve flyte.local:80:127.0.0.1 \
"http://flyte.local/oauth2/sign_in?rd=http://flyte.local/v2"
# β 200
# 3. Starting login redirects all the way to Dex's login page:
curl -s -o /dev/null -w "%{url_effective}\n" -L --max-redirs 5 \
--resolve flyte.local:80:127.0.0.1 "http://flyte.local/oauth2/start?rd=http://flyte.local/v2"
# β http://flyte.local/dex/auth/local/login?... (oauth2-proxy β Dex)
```
> [!NOTE] Why 401 and not 302 on a raw curl
> Traefik's `oauth2-signin` middleware turns the 401 into a sign-in redirect via its
> `errors` handler, which a browser follows automatically. On a plain `curl` you see the
> raw 401 β that still confirms the request is being gated. The redirect itself is
> verified by checks 2 and 3.
Then open `http://flyte.local/v2` in a browser, log in as **`admin@example.com` /
`password`**, and you should land in the console. The `X-Auth-Request-Email` header Dex
supplies flows through oauth2-proxy to Flyte and populates `executed_by` on runs you
create β see **Authentication and SSO**.
### Letting the SDK/CLI authenticate (with auth enabled)
The browser flow above works over plain HTTP, but **the SDK does not**. Like the cloud
walkthrough in **Authentication and SSO** β which requires an **HTTPS
listener** because "OIDC auth only applies to HTTPS rules" β the SDK path here only works
over **TLS**. The reason is in the SDK itself: it attaches its auth interceptors (PKCE
browser login, token injection) **only when the client uses TLS**. With `insecure: True`
it assumes "plaintext endpoint β no auth server" and skips authentication entirely, so an
SDK pointed at `http://flyte.local` sends **no token**, oauth2-proxy rejects every call,
and `flyte.run` fails the code-bundle upload with `Unauthorized` β without ever opening a
browser login.
With oauth2-proxy configured for Bearer tokens (the SDK flags in
**Kind deployment > Deploy oauth2-proxy**), Traefik's TLS listener exposed, and the
`flyte.local` cert in place (**Kind deployment > 7. Add authentication with a local IdP (optional) > Install the ingress controller > Replace the default cert with one for `flyte.local`**),
three things remain: advertise Dex in Flyte's auth metadata, point the SDK at the HTTPS
endpoint, and make Flyte able to reach Dex's discovery document.
#### Advertise Dex to the SDK/CLI
oauth2-proxy gates the **browser** path, but the SDK/CLI discover where to log in from
Flyte's **Authentication and SSO**. Point it at
Dex using the public `flytectl` client from the
**Kind deployment > Set up local OIDC provider**, then `helm upgrade`:
```yaml
# add to values-local.yaml
flyte-core-components:
runs:
authMetadata:
externalAuthServerBaseUrl: http://flyte.local/dex
flyteClient:
clientId: flytectl
redirectUri: http://localhost:53593/callback
scopes:
- openid
- profile
- offline_access
```
This is the same `authMetadata` block as a real IdP β only the issuer URL points at the
in-cluster Dex. V2 has no auth server of its own; it just advertises Dex.
#### Point the SDK at the HTTPS endpoint
In `~/.flyte/config.yaml`, reach Flyte over TLS and accept the self-signed CA from the
**Kind deployment > 7. Add authentication with a local IdP (optional) > Install the ingress controller > Replace the default cert with one for `flyte.local`**:
```yaml
admin:
endpoint: dns:///flyte.local # must match SelectCluster's clusterEndpoint (no :80)
insecure: False # use TLS β the SDK only authenticates over TLS
insecureSkipVerify: True # accept the self-signed CA (see the note above)
authType: Pkce
task:
org: local
domain: development
project: flytesnacks
```
> [!WARNING] The key is `insecureSkipVerify` (camelCase)
> The SDK reads `admin.insecureSkipVerify`. The snake_case `insecure_skip_verify` is
> silently ignored, so the SDK keeps full verification and fails on the self-signed cert.
> The SDK also reads the **project-local** `.flyte/config.yaml` (in the directory you run
> from) before `~/.flyte/config.yaml` β make sure the flag is in whichever file actually
> applies.
> [!WARNING] `endpoint` must match what `SelectCluster` returns
> Before uploading, the SDK calls `SelectCluster`; if the returned `clusterEndpoint`
> differs from your `admin.endpoint`, it builds a *separate* per-cluster session for the
> upload that may skip auth. Check the value and match it exactly:
> ```bash
> curl -s -X POST --resolve flyte.local:443:127.0.0.1 -k \
> https://flyte.local/flyteidl2.cluster.ClusterService/SelectCluster \
> -H 'Content-Type: application/json' \
> -d '{"operation":"OPERATION_CREATE_UPLOAD_LOCATION","project":"flytesnacks","domain":"development","org":"local"}'
> # β {"clusterEndpoint":"https://flyte.local"} β endpoint: dns:///flyte.local (no :443)
> ```
#### Let Flyte reach Dex's discovery document
Because Dex runs **in-cluster** with an issuer that only your host's `/etc/hosts` knows
about, two fixes are needed so Flyte's `GetOAuth2Metadata` β which fetches the IdP's
discovery document to tell the SDK where to log in β actually succeeds. (An **external**
IdP is publicly resolvable and serves the standard discovery paths, so it needs neither β
see **Kind deployment > Set up external OIDC provider**.)
**(a) DNS.** Flyte fetches `http://flyte.local/dex/...`, but `flyte.local` isn't
resolvable inside the cluster (it's only in your host's `/etc/hosts`), so the fetch times
out. Point `flyte.local` at Traefik's ClusterIP from inside the Flyte pod:
```bash
TRAEFIK_IP=$(kubectl -n traefik get svc traefik -o jsonpath='{.spec.clusterIP}')
helm upgrade flyte flyteorg/flyte-binary -n flyte -f values-local.yaml \
--set "deployment.extraPodSpec.hostAliases[0].ip=$TRAEFIK_IP" \
--set "deployment.extraPodSpec.hostAliases[0].hostnames[0]=flyte.local"
```
**(b) Discovery path.** Flyte fetches the RFC 8414 path
`/.well-known/oauth-authorization-server`, but Dex only serves the OIDC path
`/.well-known/openid-configuration` (the former returns `404`). The two carry the same
endpoints, so rewrite one to the other at Traefik:
```bash
kubectl apply -f - <<'EOF'
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: dex-wellknown-rewrite
namespace: flyte
spec:
replacePathRegex:
regex: ^/dex/\.well-known/oauth-authorization-server$
replacement: /dex/.well-known/openid-configuration
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: dex-oauth-metadata
namespace: flyte
spec:
entryPoints: [web, websecure]
routes:
- kind: Rule
priority: 200 # match before the /dex Ingress
match: Host(`flyte.local`) && Path(`/dex/.well-known/oauth-authorization-server`)
middlewares:
- name: dex-wellknown-rewrite
services:
- name: dex
port: 5556
EOF
```
Verify both fixes landed β this should return JSON, not a `404` or a timeout:
```bash
curl -s -X POST --resolve flyte.local:443:127.0.0.1 -k \
https://flyte.local/flyteidl2.auth.AuthMetadataService/GetOAuth2Metadata \
-H 'Content-Type: application/json' -d '{}' | head -c 200
```
**First clear any stale SDK token from a previous cluster.** The SDK caches OAuth
tokens in the keyring (macOS Keychain), keyed by endpoint host β `kind delete cluster`
doesn't wipe them. Dex's `storage: memory` mints new signing keys on every restart, so
an old token fails oauth2-proxy's signature check with `403 Forbidden` on
`SelectCluster` and **no browser opens**. Clear it after any cluster/Dex recreate:
```bash
# macOS; "not found" is fine. Linux: keyring del flyte.local access_token / refresh_token
for k in access_token refresh_token; do security delete-generic-password -s flyte.local -a "$k" 2>/dev/null; done
```
Then `flyte.run` opens a browser to Dex, you log in (`admin@example.com` / `password`),
and the SDK submits the run with the resulting token.
### Troubleshooting
| Symptom | Cause and fix |
|---|---|
| oauth2-proxy `CrashLoopBackOff`, logs stuck on `Performing OIDC Discovery...` | The pod can't resolve `flyte.local` in-cluster. Add the `hostAliases` from the **Kind deployment > Deploy oauth2-proxy** mapping `flyte.local` to Traefik's ClusterIP. |
| oauth2-proxy `CrashLoopBackOff`, logs show `could not fetch .well-known` | oauth2-proxy can't reach the issuer. Confirm the discovery curl in **Kind deployment > Set up local OIDC provider** returns the discovery doc and that `oidc-issuer-url` matches `issuer` in the Dex config **exactly**. |
| Browser: `Unregistered redirect_uri` / `redirect_uri did not match` | The `oauth2-proxy` static client's `redirectURIs` must list the callback for the scheme you open the console with β `http://flyte.local/oauth2/callback` **and** `https://flyte.local/oauth2/callback` (opening `/v2` over TLS uses the `https` one). List both. |
| Login succeeds but loops back to sign-in | Issuer mismatch between what the browser saw and what oauth2-proxy validated. Both must be `http://flyte.local/dex` β not a service name, not `localhost`. |
| `flyte.run` fails the upload with `Unauthorized`, **no browser opens** | The SDK is on plain HTTP (`insecure: True`) and skipped auth. Use `insecure: False` + `https://flyte.local`. |
| `InitializationError: Service is unavailable` / `EndpointUnavailable`, **no browser opens** | The SDK couldn't reach the API or discovery paths. Two common causes: the auth-discovery/API paths are still behind `oauth2-signin` (apply the two `IngressRoute`s in **Kind deployment > DigitalOcean VM > Split the API and discovery paths off the browser middleware**); or the TLS cert is rejected (next two rows). |
| `invalid peer certificate: ... not valid for name "flyte.local"` | Traefik is serving its default cert (`SAN=*.traefik.default`). Apply the `flyte.local` cert in **Kind deployment > 7. Add authentication with a local IdP (optional) > Install the ingress controller > Replace the default cert with one for `flyte.local`**. |
| `invalid peer certificate: ... CaUsedAsEndEntity` | The cert is a bare self-signed leaf; the SDK pins it as a CA. Use the two-tier root-CA-signs-leaf chain in **Kind deployment > 7. Add authentication with a local IdP (optional) > Install the ingress controller > Replace the default cert with one for `flyte.local`**. |
| `Connection refused` to `https://flyte.local` | No TLS listener β Traefik's `websecure` isn't exposed, or the cluster lacks the `30443 β 443` mapping. See **Kind deployment > 2. Create the kind cluster** / the **Kind deployment > 7. Add authentication with a local IdP (optional) > Install the ingress controller**. |
| Upload still 401 *after* a successful browser login | oauth2-proxy rejects the Bearer token. Confirm `skip-jwt-bearer-tokens=true` and `oidc-extra-audience=flytectl` (**Kind deployment > Deploy oauth2-proxy**); check its logs for `audience ... does not match`. |
| `403 Forbidden` on `SelectCluster`, **no browser opens** (oauth2-proxy logs `failed to verify id token signature`) | Stale cached token; Dex's in-memory keys changed on restart. Clear the keyring tokens (block above) and rerun. |
| `GetOAuth2Metadata` returns `... 404 ... oauth-authorization-server` | The well-known rewrite isn't applied β fix (b) in **Kind deployment > Let Flyte reach Dex's discovery document**. |
| `GetOAuth2Metadata` times out (`context deadline exceeded`) | Flyte can't resolve `flyte.local` in-cluster. Apply the `hostAliases` β fix (a) in **Kind deployment > Let Flyte reach Dex's discovery document**. |
## 8. Load a local image into kind (optional)
kind nodes can't pull from the host's Docker daemon. If you build a custom task or
Flyte image, load it into the cluster so pods can run it without a registry:
```bash
kind load docker-image : --name flyte
```
On a DigitalOcean droplet, the image must be in the **droplet's** Docker daemon first β
either build it there, or ship it from your machine with
`docker save | ssh root@ docker load`.
Reference that exact `:` in your task config; with the image already
present, the default `IfNotPresent` pull policy won't try to fetch it from a registry.
## 9. Tear down
Delete the cluster and Flyte in one command:
```bash
kind delete cluster --name flyte
```
On DigitalOcean, also destroy the droplet so it stops billing:
```bash
doctl compute droplet delete flyte-kind
```
The hosted PostgreSQL and S3/R2 bucket are untouched β clean those up in their own
consoles.
When you're ready to deploy to a real cluster, continue to
**AWS deployment**.
=== PAGE: https://www.union.ai/docs/v2/flyte/oss-deployment/kind-deployment/local-oidc ===
# Set up local OIDC provider
[Step 7 of the Kind deployment guide](_index#7-add-authentication-with-a-local-idp-optional)
gates the console behind Traefik and oauth2-proxy, using [Dex](https://dexidp.io/)
running **inside the same kind cluster** as the identity provider β so you can test the
whole authentication flow with no cloud account and no real users. This page covers
deploying Dex itself: do it after
[installing Traefik](_index#install-the-ingress-controller) and before
[deploying oauth2-proxy](_index#deploy-oauth2-proxy).
> [!WARNING] For local testing only
> Dex here uses in-memory storage and a static test password baked into its config.
> It is an IdP stand-in for development β never use this configuration anywhere real.
## The issuer-URL problem
OIDC has one constraint that shapes the Dex setup: the **issuer URL must be identical**
everywhere it's seen.
- **oauth2-proxy** (running in-cluster) reaches Dex over a Kubernetes service name.
- **Your browser** reaches Dex to log in, and must land on the *same* issuer the token
was minted for, or validation fails.
A service name like `dex.flyte.svc.cluster.local` isn't resolvable from your browser; a
`localhost` URL isn't resolvable from inside the cluster. The fix is to serve Dex under
the **same host as Flyte** (`flyte.local`) at a sub-path (`/dex`), and route to it
through Traefik. One URL β `http://flyte.local/dex` β works from both sides.
## Dex configuration
The issuer is the through-Traefik URL; the two static clients are oauth2-proxy
(confidential, with a secret) and the Flyte CLI (public, for SDK login).
`staticPasswords` gives you a login with no external user store.
The Dex Helm chart renders whatever you pass under its `config` value into a Secret and
mounts it as `config.yaml` β so write the configuration nested under a top-level
`config:` key in a values file:
```yaml
# dex-values.yaml
config:
issuer: http://flyte.local/dex
storage:
type: memory
web:
http: 0.0.0.0:5556
oauth2:
skipApprovalScreen: true # auto-approve, no consent screen in dev
staticClients:
# oauth2-proxy β confidential client (matches the secret you pass to oauth2-proxy in step 7)
- id: oauth2-proxy
name: oauth2-proxy
secret: oauth2-proxy-secret
redirectURIs:
- 'http://flyte.local/oauth2/callback'
- 'https://flyte.local/oauth2/callback' # console opened over TLS (websecure)
# Flyte CLI β public client for SDK/CLI PKCE login
- id: flytectl
name: 'Flyte CLI'
public: true
redirectURIs:
- 'http://localhost:53593/callback'
enablePasswordDB: true
staticPasswords:
# login: admin@example.com / password
- email: "admin@example.com"
username: "admin"
userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"
# bcrypt hash of the literal string "password" β see the note below
hash: "$2a$10$wi77Jcsjw08l416Q4./OCu6qNvYMaNSvA3Jbo30QeyZAvq9b4BSRK"
```
> [!WARNING] `hash` must be a real bcrypt hash
> Dex validates the hash at startup and crashes (`CrashLoopBackOff`) with
> `malformed bcrypt hash: hashedSecret too short to be a bcrypted password` if it
> isn't a complete 60-character bcrypt string. The hash above is for `password` and
> is known-good β but verify length 60 before pasting (`echo -n "$HASH" | wc -c`); a
> char lost in transit looks fine and crashes Dex. To use a different password:
> ```bash
> htpasswd -bnBC 10 "" 'your-password' | tr -d ':\n' | sed 's/^\$2y/\$2a/'
> ```
## Install Dex
```bash
helm repo add dex https://charts.dexidp.io
helm repo update
helm install dex dex/dex -n flyte -f dex-values.yaml
```
> [!NOTE] Why not a ConfigMap volume mount
> The chart already defines its own `config` volume, so mounting your own with
> `--set volumes[0].name=config` collides with it (`Duplicate value: "config"`). Passing
> the config under the chart's `config` value, as above, is the supported path and avoids
> the clash.
Confirm Dex came up:
```bash
kubectl -n flyte rollout status deploy/dex
kubectl -n flyte get svc dex # note the port (5556 by default)
```
## Route the issuer path through Traefik
Add an ingress so `http://flyte.local/dex` reaches the Dex service. This is what makes
the single issuer URL resolve from the browser:
```yaml
# dex-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dex
namespace: flyte
spec:
ingressClassName: traefik
rules:
- host: flyte.local
http:
paths:
- path: /dex
pathType: Prefix
backend:
service:
name: dex
port:
number: 5556
```
```bash
kubectl apply -f dex-ingress.yaml
```
Check discovery works through the host path β this is the URL oauth2-proxy will fetch:
```bash
# (with "127.0.0.1 flyte.local" in /etc/hosts and Traefik reachable on the node port)
curl -s http://flyte.local/dex/.well-known/openid-configuration | head
```
A JSON document with `"issuer":"http://flyte.local/dex"` confirms Dex is reachable at
the issuer it advertises.
## Next step
With Dex up and its issuer routable, return to the Kind deployment guide and
[deploy oauth2-proxy](_index#deploy-oauth2-proxy) pointed at `http://flyte.local/dex`.
=== PAGE: https://www.union.ai/docs/v2/flyte/oss-deployment/kind-deployment/external-oidc ===
# Set up external OIDC provider
[Step 7 of the Kind deployment guide](_index#7-add-authentication-with-a-local-idp-optional)
gates the console behind Traefik and oauth2-proxy using [Dex](https://dexidp.io/) as a
throwaway in-cluster IdP. This page is the same setup with a **real external OIDC
provider** (Okta, Google, Auth0, β¦) instead of Dex β useful when you want to test the
authentication flow against the provider you'll actually use.
This page assumes you've completed [steps 1β6 of the Kind deployment guide](_index) and are
about to do step 7. Register an app at your provider first: it needs the redirect URI
`http:///oauth2/callback`, and you'll need its client ID and secret ready.
An external IdP is *simpler* than the in-cluster Dex: its issuer is publicly
resolvable and it serves the standard discovery paths, so none of Dex's `hostAliases`
or well-known-rewrite workarounds apply here.
## Install the ingress controller
Install Traefik with its Helm chart and expose **both** entrypoints on the kind node's
nodePorts β `web` (HTTP) on `30080` and `websecure` (HTTPS) on `30443`, the ports the
cluster maps to host `80` and `443` (see
[step 2 of the Kind deployment guide](_index#2-create-the-kind-cluster)). The browser uses
HTTP; **the SDK needs HTTPS** β expose both up front so you don't have to reinstall
later.
```bash
helm repo add traefik https://traefik.github.io/charts
helm repo update
helm install traefik traefik/traefik -n traefik --create-namespace \
--set "service.type=NodePort" \
--set "ports.web.nodePort=30080" \
--set "ports.websecure.nodePort=30443"
```
Traefik installs its CRDs (including `Middleware`), registers a `traefik` IngressClass,
and serves a **default self-signed certificate** on `websecure`. If your cluster lacks
the `30080 β 80` / `30443 β 443` mappings from step 2, Traefik is unreachable at
`flyte.local` and you'll need to recreate the cluster.
Then replace Traefik's default certificate with one for `flyte.local`, exactly as in
[the Kind deployment guide](_index#replace-the-default-cert-with-one-for-flytelocal) β the
SDK rejects the built-in cert, and the fix (a two-tier root-CA-signs-leaf chain) is
identical whichever IdP you use.
## Deploy oauth2-proxy
Give oauth2-proxy your IdP details and a random cookie secret. `--set-xauthrequest`
makes it emit the `X-Auth-Request-*` headers Traefik forwards downstream, and
`--reverse-proxy` tells it to trust the forwarded host/proto from Traefik. The last three
flags let the **SDK/CLI** authenticate too (not just the browser) β set them now so you
don't have to upgrade later:
```bash
# 32-byte cookie secret. Must decode to 16/24/32 bytes β head -c 32 trims the
# base64 string, since a raw 44-char value fails oauth2-proxy's length check.
COOKIE_SECRET=$(openssl rand -base64 32 | head -c 32)
helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm repo update
helm install oauth2-proxy oauth2-proxy/oauth2-proxy -n flyte \
--set config.clientID='' \
--set config.clientSecret='' \
--set config.cookieSecret="$COOKIE_SECRET" \
--set extraArgs.provider=oidc \
--set extraArgs.oidc-issuer-url='https:///oauth2/default' \
--set extraArgs.upstream='static://202' \
--set extraArgs.reverse-proxy='true' \
--set extraArgs.set-xauthrequest='true' \
--set extraArgs.email-domain='*' \
--set extraArgs.cookie-secure='false' \ # local HTTP, not HTTPS
--set extraArgs.skip-jwt-bearer-tokens='true' \ # accept SDK Bearer JWTs
--set extraArgs.oidc-extra-audience='' \ # SDK client's audience (singular flag!)
--set extraArgs.bearer-token-login-fallback='false' # invalid token β 403, not HTML login
```
The browser path uses the session cookie; the **SDK** sends an `Authorization: Bearer`
JWT instead. `skip-jwt-bearer-tokens` makes oauth2-proxy verify that JWT against the IdP's
JWKS and pass it through, while `oidc-extra-audience` must list the **public client ID**
the SDK uses (the `flyteClient.clientId` from your `authMetadata` β see
**Kind deployment > Set up external OIDC provider > Advertise your IdP to the SDK/CLI** below) β its tokens carry that
as their audience. The flag is **singular** β `oidc-extra-audiences` (plural) is not valid
and crash-loops oauth2-proxy with `unknown flag`. Without these, the SDK is rejected and
`flyte.run` fails the upload with `Unauthorized`.
## Create the ForwardAuth middleware
Two Traefik `Middleware` objects: one sends each request to oauth2-proxy for a verdict
and forwards the identity headers; the other catches the `401` an unauthenticated
request gets and redirects it to the oauth2-proxy sign-in page.
```yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: oauth2-auth
namespace: flyte
spec:
forwardAuth:
address: http://oauth2-proxy.flyte.svc.cluster.local/oauth2/auth
trustForwardHeader: true
# Forwarded to Flyte; these feed executed_by run attribution.
authResponseHeaders:
- X-Auth-Request-User
- X-Auth-Request-Email
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: oauth2-signin
namespace: flyte
spec:
errors:
status:
- "401"
service:
name: oauth2-proxy
port: 80
query: "/oauth2/sign_in?rd={url}"
```
## Create the Flyte ingress with the middleware
Re-render Flyte with the ingress enabled and pointed at Traefik. The
`traefik.ingress.kubernetes.io/router.middlewares` annotation chains both middlewares
onto every route β this is the Traefik equivalent of the ALB's `auth-type: oidc`. The
reference format is `-@kubernetescrd`:
```yaml
# add to values-local.yaml, replacing the `ingress.create: false` block
ingress:
create: true
host: flyte.local # add "127.0.0.1 flyte.local" to /etc/hosts
ingressClassName: traefik
httpAnnotations:
traefik.ingress.kubernetes.io/router.middlewares: flyte-oauth2-signin@kubernetescrd,flyte-oauth2-auth@kubernetescrd
```
You also need a route that sends `/oauth2` to oauth2-proxy itself so the sign-in
redirect resolves. Apply it alongside the Flyte release:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: oauth2-proxy
namespace: flyte
spec:
ingressClassName: traefik
rules:
- host: flyte.local
http:
paths:
- path: /oauth2
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
number: 80
```
After `helm upgrade flyte β¦ -f values-local.yaml`, add a hosts entry so the browser can
resolve `flyte.local` to the local Traefik node port (do this once):
```bash
echo "127.0.0.1 flyte.local" | sudo tee -a /etc/hosts
```
`http://127.0.0.1/v2` won't work in its place β Traefik has no route for that host, and
the OIDC issuer is `flyte.local`, so login fails on an issuer mismatch. Then open
`http://flyte.local/v2` β Traefik bounces you through the IdP and back into the console.
### Split the API and discovery paths off the browser middleware
This gates the **browser** correctly, but the same `oauth2-signin` redirect on **every**
path breaks the SDK (the cloud walkthrough avoids this by splitting into three ingresses β
`ingress` / `apiJwtIngress` / `wellknownIngress` β since ALB can't mix cookie-OIDC and JWT
auth on one). Two path groups need different handling:
- **Auth-discovery** (`AuthMetadataService`, `IdentityService`) β the SDK reads these
*before* it has a token, so they must **bypass auth**. Gated, they return a `text/plain`
401 that ConnectRPC reports as `UNAVAILABLE`, and the SDK never starts login.
- **The `flyteidl2.*` API** β needs `oauth2-auth` (Bearer validation) but **not**
`oauth2-signin`, so an unauthenticated call gets a clean gRPC 401 the SDK retries after
login, not sign-in HTML.
Add two higher-priority `IngressRoute`s β Traefik matches the highest `priority` first,
so these win over the `flyte-http` Ingress for their paths:
```bash
kubectl apply -f - <<'EOF'
# Discovery β highest priority, NO middleware. The SDK reads these before it has a
# token; they must reach Flyte directly so it returns the real metadata (issuer,
# client ID), not oauth2-proxy's unparseable 401. (Equivalent of wellknownIngress.)
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: flyte-auth-discovery
namespace: flyte
spec:
entryPoints: [web, websecure]
routes:
- kind: Rule
priority: 300
match: Host(`flyte.local`) && (PathPrefix(`/flyteidl2.auth.AuthMetadataService`) || PathPrefix(`/flyteidl2.auth.IdentityService`))
services:
- name: flyte-http
port: 8090
scheme: h2c # gRPC over cleartext HTTP/2 to the backend
---
# API β oauth2-auth only (validates Bearer tokens), no oauth2-signin redirect, so an
# unauthenticated call gets a clean 401 the SDK can act on. (Equivalent of apiJwtIngress.)
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: flyte-api-bearer
namespace: flyte
spec:
entryPoints: [web, websecure]
routes:
- kind: Rule
priority: 100
match: Host(`flyte.local`) && PathPrefix(`/flyteidl2.`)
middlewares:
- name: oauth2-auth
services:
- name: flyte-http
port: 8090
scheme: h2c
EOF
```
Now `flyte.local` routes three ways by precedence: discovery (300) bypasses auth, the
API (100) requires a Bearer token, and everything else β the `/v2` console β falls
through to the `flyte-http` Ingress with the full browser middleware chain. Verify the
discovery path returns JSON rather than oauth2-proxy's 401:
```bash
curl -s -X POST --resolve flyte.local:443:127.0.0.1 -k \
https://flyte.local/flyteidl2.auth.AuthMetadataService/GetPublicClientConfig \
-H 'Content-Type: application/json' -d '{}' | head -c 120
# β {"clientId":"flytectl", ...} (JSON, not "Unauthorized")
```
## Advertise your IdP to the SDK/CLI
oauth2-proxy gates the **browser** path, but the SDK/CLI discover where to log in from
Flyte's [auth metadata](../authentication#advertise-your-identity-provider). Register
a **public (PKCE) client** at your IdP with the `http://localhost:53593/callback`
redirect, point `authMetadata` at it, then `helm upgrade`:
```yaml
# add to values-local.yaml
flyte-core-components:
runs:
authMetadata:
externalAuthServerBaseUrl: https:///oauth2/default
flyteClient:
clientId:
redirectUri: http://localhost:53593/callback
scopes:
- openid
- profile
- offline_access
```
## Letting the SDK/CLI authenticate
The browser flow above works over plain HTTP, but **the SDK does not** β it attaches its
auth interceptors (PKCE browser login, token injection) **only when the client uses
TLS**. With `insecure: True` it assumes "plaintext endpoint β no auth server" and skips
authentication entirely, so an SDK pointed at `http://flyte.local` sends **no token**,
oauth2-proxy rejects every call, and `flyte.run` fails the code-bundle upload with
`Unauthorized` β without ever opening a browser login.
In `~/.flyte/config.yaml`, reach Flyte over TLS and accept the self-signed CA from the
**Kind deployment > Set up external OIDC provider > Install the ingress controller**:
```yaml
admin:
endpoint: dns:///flyte.local # must match SelectCluster's clusterEndpoint (no :80)
insecure: False # use TLS β the SDK only authenticates over TLS
insecureSkipVerify: True # accept the self-signed CA
authType: Pkce
task:
org: local
domain: development
project: flytesnacks
```
The `insecureSkipVerify` and `SelectCluster` caveats from
[the Kind deployment guide](_index#point-the-sdk-at-the-https-endpoint) apply unchanged.
Because your IdP is publicly resolvable and serves the standard
`/.well-known/oauth-authorization-server` path, Flyte's `GetOAuth2Metadata` works
without the DNS and discovery-path fixes the in-cluster Dex needs.
Then `flyte.run` opens a browser to your IdP, you log in, and the SDK submits the run
with the resulting token.
## Troubleshooting
| Symptom | Cause and fix |
|---|---|
| `flyte.run` fails the upload with `Unauthorized`, **no browser opens** | The SDK is on plain HTTP (`insecure: True`) and skipped auth. Use `insecure: False` + `https://flyte.local`. |
| `InitializationError: Service is unavailable` / `EndpointUnavailable`, **no browser opens** | The SDK couldn't reach the API or discovery paths. Two common causes: the auth-discovery/API paths are still behind `oauth2-signin` (apply the two `IngressRoute`s in **Kind deployment > Set up external OIDC provider > Split the API and discovery paths off the browser middleware**); or the TLS cert is rejected (see the cert section of the [Kind deployment guide](_index#replace-the-default-cert-with-one-for-flytelocal)). |
| `Connection refused` to `https://flyte.local` | No TLS listener β Traefik's `websecure` isn't exposed, or the cluster lacks the `30443 β 443` mapping. See [step 2 of the Kind deployment guide](_index#2-create-the-kind-cluster) / the **Kind deployment > Set up external OIDC provider > Install the ingress controller**. |
| Upload still 401 *after* a successful browser login | oauth2-proxy rejects the Bearer token. Confirm `skip-jwt-bearer-tokens=true` and `oidc-extra-audience=` (**Kind deployment > Set up external OIDC provider > Deploy oauth2-proxy**); check its logs for `audience ... does not match`. |
| Browser: `Unregistered redirect_uri` | The exact callback isn't registered on the IdP app. Add `http://flyte.local/oauth2/callback` **and** `https://flyte.local/oauth2/callback` (opening `/v2` over TLS uses the `https` one). |
=== PAGE: https://www.union.ai/docs/v2/flyte/oss-deployment/aws-deployment ===
# AWS deployment
This guide installs Flyte with the `flyte-binary` Helm chart. It assumes you have
already provisioned the [external dependencies](./overview) β a Kubernetes cluster, a
PostgreSQL database, and an object-store bucket β and that you have `helm` and
`kubectl` configured against your cluster.
## 1. Add the Helm repository
```bash
helm repo add flyteorg https://flyteorg.github.io/flyte
helm repo update
```
The `helm install` commands below reference the chart as `flyteorg/flyte-binary`.
## 2. Write a values file
Create a `values.yaml` with the minimum configuration. Everything in angle brackets is
a placeholder you replace:
```yaml
# values.yaml β minimal Flyte configuration
# fullnameOverride names all resources (here: flyte, flyte-http, flyte-console).
# Give each release a distinct name if you run more than one Flyte instance.
fullnameOverride: flyte
configuration:
database:
postgres:
host: # e.g. my-db.example.com
port: 5432
dbname: flyte # database must already exist
username: flyte
password: # creates a mounted Secret (or use passwordPath)
options: "sslmode=require" # use sslmode=disable only for local/dev
storage:
metadataContainer: # object-store bucket Flyte reads and writes
provider: s3 # s3 | gcs | azure
providerConfig:
s3:
region: # e.g. us-east-1
authType: iam # iam (recommended) | accesskey
serviceAccount:
create: true
annotations: {} # IRSA role binding β see step 4
```
The required fields:
| Setting | Key | Notes |
|---|---|---|
| Database host | `configuration.database.postgres.host` | Reachable from the cluster. |
| Database name | `configuration.database.postgres.dbname` | Must already exist. |
| Database user | `configuration.database.postgres.username` | Default `postgres`. |
| Database password | `configuration.database.postgres.password` | Creates and mounts a Secret. Use `passwordPath` instead to mount your own. The two are mutually exclusive. |
| Storage bucket | `configuration.storage.metadataContainer` | The object-store bucket Flyte reads and writes. |
| Storage provider | `configuration.storage.provider` | `s3`, `gcs`, or `azure`. |
| Storage region | `configuration.storage.providerConfig.s3.region` | S3 region (S3 provider). |
| Service account | `serviceAccount.annotations` | Cloud IAM binding for object-store access (step 4). |
## 3. Install
Render the manifests first to check your values, then install for real:
```bash
# Dry run β renders templates without touching the cluster
helm install flyte flyteorg/flyte-binary -n flyte --create-namespace -f values.yaml --dry-run
# Install
helm install flyte flyteorg/flyte-binary -n flyte --create-namespace -f values.yaml
```
Watch the rollout:
```bash
kubectl -n flyte rollout status deploy/flyte
kubectl -n flyte get pods
```
A `wait-for-db` init container blocks startup until the database is reachable, so a
pod stuck in `Init` usually means the database host, credentials, or network policy
are wrong.
## 4. Grant object-store access
The Flyte pod and the task pods it launches need credentials to read and write the
bucket. Prefer cloud-native workload identity over static keys.
**IRSA (recommended).** Annotate the service account with an IAM role that can access
the bucket:
```yaml
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam:::role/
```
**Static keys** for S3-compatible stores such as MinIO β not recommended for
production:
```yaml
configuration:
storage:
providerConfig:
s3:
authType: accesskey
accessKey:
secretKey:
endpoint: # for non-AWS S3-compatible stores
disableSSL: false
v2Signing: false # set true for some MinIO setups
```
**Task pods.** Tasks run in their own pods, which need the same object-store access.
Run them under a service account that carries the IAM binding by setting it in the
executor config (merged via `configuration.inline`):
```yaml
configuration:
inline:
executor:
defaultK8sServiceAccount: flyte # IAM-annotated SA tasks run as
```
## 5. Expose Flyte with an ingress
By default the chart only creates `ClusterIP` Services. To reach Flyte from outside
the cluster, enable the ingress. A **single HTTP ingress** serves the console and the
API β there is no separate gRPC ingress (see the
[Deployment overview](./overview)).
```yaml
ingress:
create: true
host:
# Your cloud's native ingress class, e.g. alb (EKS), gce (GKE),
# azure-application-gateway (AKS). See the Deployment overview for the options.
ingressClassName:
```
The console is served under `console.basePath` (default `/v2`) on this same host. It
talks to the API same-origin, so it only works when the console and the API are behind
the **same ingress host** β always expose them together.
For provider-specific ingress annotations (TLS, ALB scheme, health checks), add them
under `ingress.httpAnnotations`. See the AWS/EKS example below and the
[Authentication and SSO](./authentication) page.
## 6. Verify the installation
**Without an ingress**, port-forward the API service and call a Connect endpoint:
```bash
kubectl -n flyte port-forward service/flyte-http 8090:8090
```
```bash
# In another terminal β list projects over the Connect (HTTP) API:
curl -s -X POST \
http://localhost:8090/flyteidl2.project.ProjectService/ListProjects \
-H 'Content-Type: application/json' -d '{}'
```
A JSON response (rather than a connection error) confirms the binary is up and talking
to its database.
**With an ingress**, open `https:///v2` in a browser to load the
console, and point the SDK/CLI at the same host.
## 7. Tear down
Uninstall the Helm release and delete the namespace:
```bash
helm uninstall flyte -n flyte
kubectl delete namespace flyte
```
Uninstalling the release removes the ingress resource, which prompts the ingress
controller (e.g. the AWS Load Balancer Controller) to delete the load balancer it
provisioned.
{{< WARNING >}}
Confirm the ALB is gone in the AWS console so it stops billing.
{{< /WARNING >}}
The external dependencies β the RDS database, the S3 bucket, and the EKS cluster
itself β are untouched. Delete those separately in the AWS console (or with the tool
you provisioned them with) if you no longer need them.
Next: secure the deployment with [Authentication and SSO](./authentication).
## Full Values File Example
A fuller values file for an AWS/EKS cluster β RDS for PostgreSQL, S3 for storage, IRSA
for credentials, and an ALB ingress. Replace every placeholder; no real account IDs,
hostnames, or ARNs are included.
```yaml
# values-eks.yaml
fullnameOverride: flyte
configuration:
database:
postgres:
host: ...rds.amazonaws.com
port: 5432
dbname: flyte
username: flyte
password: # chart stores this in a Secret, not the ConfigMap
options: "sslmode=require"
storage:
metadataContainer:
provider: s3
providerConfig:
s3:
region:
authType: iam
inline:
executor:
defaultK8sServiceAccount: flyte # task pods inherit S3 access via IRSA
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam:::role/
ingress:
create: true
host:
httpAnnotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:::certificate/
alb.ingress.kubernetes.io/healthcheck-path: /healthz
alb.ingress.kubernetes.io/healthcheck-port: "8090"
```
Install it the same way:
```bash
helm install flyte flyteorg/flyte-binary -n flyte --create-namespace -f values-eks.yaml
```
## Default task resources
Anything under `configuration.inline` is merged into the rendered Flyte config, which
is how you set options the top-level values don't expose directly.
Set the default CPU and memory **requests** for task pods that don't specify their own,
via the Kubernetes plugin config:
```yaml
configuration:
inline:
plugins:
k8s:
default-cpus: 500m
default-memory: 1Gi
```
## Default scheduling for task pods
Add tolerations, affinity / node selectors, or injected environment variables to every
task pod under `configuration.inline.plugins.k8s`:
```yaml
configuration:
inline:
plugins:
k8s:
default-tolerations:
- key: flyte.org/node-role
operator: Equal
value: worker
effect: NoSchedule
default-affinity: {} # a standard core/v1 Affinity
default-env-vars:
- MY_ENV_VAR: value # injected into every task pod
```
## OpenTelemetry
Flyte exports traces (and metrics, which reuse the trace exporter) via OpenTelemetry.
It's off by default (`otel.type: noop`). Point it at a collector under
`configuration.inline.otel`:
```yaml
configuration:
inline:
otel:
type: otlpgrpc # noop | file | jaeger | otlpgrpc | otlphttp
otlpgrpc:
endpoint: http://otel-collector.flyte.svc.cluster.local:4317
# Trace sampling β keep a fraction in production.
sampler:
parentSampler: traceid
traceIdRatio: 0.01 # sample 1% of traces
```
| `otel.type` | Where it sends | Endpoint key |
|---|---|---|
| `otlpgrpc` | OTLP collector over gRPC (recommended) | `otlpgrpc.endpoint` |
| `otlphttp` | OTLP collector over HTTP | `otlphttp.endpoint` |
| `jaeger` / `file` | Jaeger / a local file | `jaeger.*` / `file.*` |
| `noop` | disabled (default) | β |
Prefer `otlpgrpc` β the `otlphttp` metric exporter reuses the trace endpoint path.
Send to any OTLP collector (e.g. the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/),
which can fan metrics out to Prometheus and traces to Jaeger/Tempo).
## Database password from a Secret
When you set `configuration.database.postgres.password`, the chart writes it into a
Kubernetes Secret (kept out of the plaintext ConfigMap) and mounts it into the Flyte
pod β the password lives only in your values file. The same applies to S3 access keys
when `authType: accesskey`.
To keep the password out of the values file too, leave
`configuration.database.postgres.password` empty and either:
- reference an existing Kubernetes Secret with `configuration.extraInlineSecretRefs`, or
- mount the password as a file and point
`configuration.database.postgres.passwordPath` at it.
=== PAGE: https://www.union.ai/docs/v2/flyte/oss-deployment/authentication ===
# Authentication and SSO
Flyte delegates authentication to an **external OIDC identity provider** (Okta,
Google, Auth0, β¦). Two things are involved:
1. **Auth metadata** β the runs service advertises *which* IdP to use, so SDK/CLI and
browser clients can discover where to log in and get tokens. Configured under
`flyte-core-components.runs.authMetadata`.
2. **Enforcement at the ingress** β the load balancer validates those tokens (and
challenges browsers with SSO) *before* requests reach Flyte. Configured with ingress
annotations.
## Advertise your identity provider
The runs service serves OAuth2 authorization-server metadata
(`/.well-known/oauth-authorization-server` and the `AuthMetadataService` RPC) that
proxies your external IdP. Clients that discover auth at this deployment are pointed at
the IdP and obtain IdP-issued tokens, which the ingress then validates.
```yaml
flyte-core-components:
runs:
authMetadata:
# Your external OAuth2 authorization server (the IdP issuer URL).
externalAuthServerBaseUrl: https:///oauth2/default
# Public (PKCE) client advertised to the SDK/CLI via GetPublicClientConfig
# for browser login. Register this app at your IdP with the localhost redirect.
flyteClient:
clientId:
redirectUri: http://localhost:53593/callback
scopes:
- openid
- profile
- offline_access
```
The `redirectUri` (default `http://localhost:53593/callback`) is the SDK/CLI's local
PKCE callback and must be registered as a redirect URI on the IdP application.
> **Okta note.** PKCE needs an Okta **native app** client ID with the
> `http://localhost:53593/callback` redirect registered; the `client_credentials`
> (machine) flow needs a custom scope on the auth server. Adjust `clientId` / `scopes`
> for the flow you use.
## Enforce auth at the ingress
Browsers and machine clients authenticate differently, and on a controller like AWS
ALB a single ingress can't combine cookie-OIDC (browser) and JWT (token) auth β so the
chart can render up to **three ingresses**:
| Ingress (values key) | Purpose |
|---|---|
| `ingress` (`httpAnnotations`) | Serves the console (`/v2`) and API; challenges **browsers** with cookie-OIDC SSO (**Authentication and SSO > Single sign-on for the console at the ALB**). |
| `ingress.apiJwtIngress` | **JWT-validates** the `flyteidl2.*` API paths for requests carrying `Authorization: Bearer` (SDK / CLI / machine clients). Give it higher controller precedence than the http ingress so Bearer requests match it first. |
| `ingress.wellknownIngress` | Serves the **unauthenticated** auth-discovery endpoints (`/.well-known/oauth-authorization-server`, `AuthMetadataService`) β clients need these *before* they hold a token, so give it the highest precedence to bypass auth. |
Enable the JWT and discovery ingresses and supply your controller/JWT config via their
`annotations` β e.g. on ALB: the ACM `certificate-arn`, the JWT-validation config, the
`Authorization: Bearer*` match condition, and `group.order` values (lower = evaluated
first) that put `wellknownIngress` first, then `apiJwtIngress`, then the http ingress:
```yaml
ingress:
create: true
host:
# Browser SSO annotations on the main http ingress β see the walkthrough below.
httpAnnotations: { }
apiJwtIngress:
enabled: true
annotations: { } # JWT validation + the `Authorization: Bearer*` match
wellknownIngress:
enabled: true
annotations: { } # highest controller precedence; no auth
```
## Single sign-on for the console at the ALB
This is the browser cookie-OIDC SSO referenced above β it goes on the main http
ingress's `httpAnnotations`. You can put OIDC single sign-on **in front of the console**
at the ALB, so that
hitting `https:///v2` challenges the user to log in at your IdP before the
request ever reaches the console. ALB's native `authenticate-oidc` action handles the
login and manages the session cookie; the Flyte binary is unchanged. SDK/machine
clients that send an `Authorization: Bearer` token are matched by higher-precedence
API rules and bypass the cookie flow.
```
Browser ββGET /v2βββΆ ALB ββ(no session)βββΆ 302 βΆ IdP login
β² β
βββββ /oauth2/idpresponse ββββββ (ALB swaps codeβtoken,
sets session cookie, forwards to the console)
```
### Prerequisites
- The **AWS Load Balancer Controller** managing your ingress (`ingressClassName: alb`).
- An **HTTPS listener** with an ACM certificate covering your host β OIDC auth only
applies to HTTPS rules.
- An **OIDC application** at your IdP (confidential client, Authorization Code flow)
with a client ID and secret.
### 1. Configure the OIDC app
On the IdP application:
- Add the **sign-in / redirect URI** exactly (note the path β ALB's callback is fixed):
```
https:///oauth2/idpresponse
```
- Grant type **Authorization Code**; scopes at least `openid email`.
- Assign the users/groups allowed into the console.
You'll need the issuer plus the authorize/token/userinfo endpoints:
| | Okta (custom auth server) | Google |
|---|---|---|
| issuer | `https:///oauth2/` | `https://accounts.google.com` |
| authorize | `β¦/oauth2//v1/authorize` | `https://accounts.google.com/o/oauth2/v2/auth` |
| token | `β¦/oauth2//v1/token` | `https://oauth2.googleapis.com/token` |
| userinfo | `β¦/oauth2//v1/userinfo` | `https://openidconnect.googleapis.com/v1/userinfo` |
For Okta, the discovery doc
`https:///oauth2//.well-known/openid-configuration` is the source of truth.
### 2. Create the client Secret
In the **same namespace as the ingress**, with keys **exactly** `clientID` /
`clientSecret`:
```bash
kubectl create secret generic flyte-console-oidc -n flyte \
--from-literal=clientID='' \
--from-literal=clientSecret=''
```
### 3. Allow the LB controller to read the Secret
The AWS Load Balancer Controller's service account must be able to `get`/`list`/
`watch` Secrets in the ingress namespace. The upstream Helm chart usually grants this
cluster-wide, but hardened installs may not β if yours doesn't, add a namespaced Role
and RoleBinding:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: aws-lb-controller-oidc-secret
namespace: flyte
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: aws-lb-controller-oidc-secret
namespace: flyte
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: aws-lb-controller-oidc-secret
subjects:
- kind: ServiceAccount
name: aws-load-balancer-controller # adjust to your controller's SA
namespace: kube-system
```
Without this you'll see `FailedBuildModel β¦ secrets "flyte-console-oidc" is
forbidden` on the ingress and no auth rule is applied.
### 4. Add the auth annotations
Add the OIDC annotations to `ingress.httpAnnotations` in your values and
`helm upgrade`. They apply to every rule on the HTTP ingress, gating the `/v2`
console:
```yaml
ingress:
httpAnnotations:
# β¦ existing ALB annotations (certificate-arn, scheme, listen-ports, β¦) β¦
alb.ingress.kubernetes.io/auth-type: oidc
alb.ingress.kubernetes.io/auth-scope: openid email
alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate
alb.ingress.kubernetes.io/auth-session-timeout: "604800" # 7 days
alb.ingress.kubernetes.io/auth-idp-oidc: '{"issuer":"","authorizationEndpoint":"","tokenEndpoint":"","userInfoEndpoint":"","secretName":"flyte-console-oidc"}'
```
### 5. Verify
```bash
# Console redirects to the IdP:
curl -s -o /dev/null -D - https:///v2 | grep -i '^location'
# β 302 β¦ https:///v1/authorize?client_id=β¦&redirect_uri=β¦%2Foauth2%2Fidpresponseβ¦
# API path is NOT gated (a normal response, not a 302 to the IdP):
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
https:///flyteidl2.project.ProjectService/ListProjects \
-H 'Content-Type: application/json' -d '{}'
```
Then open `https:///v2` in a browser β you should be bounced through the IdP and
back into the console.
### Troubleshooting
| Symptom | Cause and fix |
|---|---|
| `FailedBuildModel β¦ secrets "β¦" is forbidden` on the ingress | The LB controller can't read the Secret. Apply the RBAC in step 3. |
| Browser: `'redirect_uri' parameter must be a Login redirect URI` | The exact callback isn't registered. Add `https:///oauth2/idpresponse` (with that path) to the IdP app's redirect URIs. |
| `401 Authorization Required` *after* a successful login | The token exchange failed β almost always a wrong client **secret** or **client_id**. A trailing `%` on a secret copied from a terminal is the shell's no-newline marker, not part of the secret; strip it. |
### Annotation reference
| Annotation | Purpose |
|---|---|
| `auth-type: oidc` | Enable OIDC auth on the ingress's HTTPS rules. |
| `auth-idp-oidc` | JSON: `issuer`, `authorizationEndpoint`, `tokenEndpoint`, `userInfoEndpoint`, `secretName` (Secret with `clientID`/`clientSecret`). |
| `auth-scope` | Space-separated scopes, e.g. `openid email`. |
| `auth-on-unauthenticated-request` | `authenticate` (challenge), `allow`, or `deny`. |
| `auth-session-timeout` | Session cookie lifetime in seconds. |
The ALB callback path is fixed at `/oauth2/idpresponse`, and auth applies only to the
annotated ingress's HTTPS listener rules.
## Run attribution (`executed_by`)
Once authentication happens at the edge, Flyte records **who created each run**
(surfaced as `executed_by` in run metadata). The runs service does not re-validate
tokens itself β it reads the identity from the headers the proxy forwards. After ALB
`authenticate-oidc` those are:
- `X-Amzn-Oidc-Data` β a signed JWT carrying the full claims (`sub`, `email`,
`given_name`, `family_name`); used on the browser/cookie path.
- `X-Amzn-Oidc-Identity` β the subject only; used when the data header is absent.
- `Authorization: Bearer ` β the SDK/CLI path (proxy-agnostic, always honored).
This token carries only the subject, so name and email are filled from the IdP's
`userinfo` endpoint when `runs.authMetadata.externalAuthServerBaseUrl` is set.
The defaults match ALB, so a standard ALB SSO deployment needs no extra configuration.
> **Trust boundary.** The forwarded JWTs are decoded but **not** signature-verified by
> the runs service. That is safe only behind a trusted proxy that validates tokens and
> strips any client-supplied copies of these headers. If the service can be reached
> directly, set `trustForwardedIdentityHeaders: false` and `executed_by` is left unset
> rather than risk a spoofed identity.
### Behind a non-ALB proxy (oauth2-proxy / Traefik)
The header names are configurable, so attribution works behind any auth proxy. For
oauth2-proxy or Traefik forward-auth, which forward plain values instead of a JWT:
```yaml
flyte-core-components:
runs:
trustForwardedIdentityHeaders: true # default; gates header-derived attribution
identityHeaders:
claimsJwtHeader: "" # no JWT header on this path
subjectHeader: X-Auth-Request-User # ALB default: X-Amzn-Oidc-Identity
emailHeader: X-Auth-Request-Email # ALB default: unset (email is in the JWT)
```
| Setting | Header read | ALB default | oauth2-proxy / Traefik |
|---|---|---|---|
| `claimsJwtHeader` | JWT with full claims | `X-Amzn-Oidc-Data` | *(empty)* |
| `subjectHeader` | subject, plain value | `X-Amzn-Oidc-Identity` | `X-Auth-Request-User` |
| `emailHeader` | email, plain value | *(unset)* | `X-Auth-Request-Email` |
The same trust boundary applies: the proxy must validate identity and strip any
client-supplied copies of these headers, with `trustForwardedIdentityHeaders` enabled.
=== PAGE: https://www.union.ai/docs/v2/flyte/oss-deployment/app-serving ===
# Enable app serving
Flyte can host long-running **apps** β web services, dashboards, model servers β next to
your workflows. Each app runs as a [Knative](https://knative.dev) Service that Flyte's
built-in app controller creates and manages for you, including scaling to zero when an
app is idle and giving every app a stable URL.
App serving is **off by default**. Because apps depend on Knative, you install Knative
Serving first, then turn app serving on in the `flyte-binary` chart.
## How it works
- You define an app (with the SDK) and Flyte's app controller creates a Knative Service
(`KService`) in your cluster.
- Knative Serving runs the app, autoscales it (including to zero), and routes traffic.
- Each app is published at a URL derived from a base domain you configure:
`{name}-{project}-{domain}.{base-domain}`.
## Prerequisites
- A running Flyte deployment (see [AWS deployment](./aws-deployment)).
- Cluster-admin access to install Knative (CRDs and controllers).
- A **wildcard DNS record** and a **TLS certificate** for the app base domain (details
in steps 2β3).
> **π Note**
>
> Your cloud's ingress/load-balancer controller (for example the AWS Load Balancer
> Controller) **cannot** act as Knative's networking layer β Knative needs one of its own
> networking layers: **Kourier**, Istio, or Contour. This guide uses **Kourier**, the
> lightweight default. You can still place your cloud load balancer *in front of* Kourier
> (see step 3).
## 1. Install Knative Serving and Kourier
Pick a Knative release that supports your Kubernetes version (Knative supports only the
most recent Kubernetes versions β check the
[Knative install docs](https://knative.dev/docs/install/)). Then install Serving and the
Kourier networking layer:
```bash
export KNATIVE_VERSION=knative-v1.17.0 # choose a release compatible with your k8s version
# Knative Serving: CRDs, then core
kubectl apply -f https://github.com/knative/serving/releases/download/$KNATIVE_VERSION/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/$KNATIVE_VERSION/serving-core.yaml
# Kourier networking layer
kubectl apply -f https://github.com/knative-extensions/net-kourier/releases/download/$KNATIVE_VERSION/kourier.yaml
# Tell Knative to route through Kourier
kubectl patch configmap/config-network -n knative-serving --type merge \
-p '{"data":{"ingress-class":"kourier.ingress.networking.knative.dev"}}'
```
Wait for the control plane and gateway to be ready:
```bash
kubectl wait --for=condition=Available deploy --all -n knative-serving --timeout=180s
kubectl wait --for=condition=Available deploy --all -n kourier-system --timeout=180s
```
> **π Note**
>
> If `kubectl apply` rejects the manifests, your cluster is probably older than this
> Knative release requires. Install an earlier Knative version (Serving and net-kourier
> must use the **same** version).
## 2. Configure the apps domain
Apps are published at `{name}-{project}-{domain}.{base-domain}`. Set the base domain
Knative uses, and make each app's hostname a **single label** under it so one wildcard
TLS certificate can cover them all:
```bash
# Base domain for app URLs
kubectl patch configmap/config-domain -n knative-serving --type merge \
-p '{"data":{"":""}}'
# Single-label hostnames: .
kubectl patch configmap/config-network -n knative-serving --type merge \
-p '{"data":{"domain-template":"{{.Name}}.{{.Domain}}"}}'
```
The single-label template matters: a wildcard TLS certificate (`*.apps.example.com`)
matches only **one** label. Knative's default template is
`{{.Name}}.{{.Namespace}}.{{.Domain}}`, which produces a two-label hostname that a
wildcard certificate would not cover, breaking TLS. Dropping the namespace keeps every
app hostname single-label.
You will provision the matching DNS record and certificate for `` in
the next step.
## 3. Expose Kourier
Traffic reaches apps through Kourier. Choose how to expose it:
### LoadBalancer (simplest)
The Kourier install creates a `kourier` Service of type `LoadBalancer`, so your cloud
provisions a network load balancer automatically. Find its address:
```bash
kubectl get svc kourier -n kourier-system
```
Then provision, for ``:
- a **wildcard DNS record** `*.` β the load balancer's hostname/IP, and
- a **TLS certificate** covering `*.` (attach it to the load balancer,
or use Knative [auto-TLS](https://knative.dev/docs/serving/encryption/)).
### AWS (ALB in front)
To reuse an existing ALB, ACM certificate, and edge auth, switch the Kourier Service to
`ClusterIP` and front it with an Ingress:
```bash
kubectl patch svc kourier -n kourier-system --type merge -p '{"spec":{"type":"ClusterIP"}}'
```
```yaml
# kourier-alb-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kourier-alb
namespace: kourier-system
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:::certificate/
# Kourier returns 404 for an unmatched host while it is healthy, so accept it.
alb.ingress.kubernetes.io/healthcheck-path: /
alb.ingress.kubernetes.io/success-codes: "200,404"
spec:
rules:
- host: "*."
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kourier
port:
number: 80
```
```bash
kubectl apply -f kourier-alb-ingress.yaml
```
Point a wildcard DNS record `*.` at the ALB. The ALB terminates TLS
with your ACM certificate and forwards to Kourier, which routes by hostname to each app.
**Require authentication (optional).** Apps are public by default β anyone who can reach
the ALB can open them. To put the same OIDC login your console uses in front of every app,
add edge authentication to the `kourier-alb` Ingress. The AWS Load Balancer Controller then
programs an `authenticate-oidc` action on the ALB, so unauthenticated requests are bounced
to your identity provider first.
Add these annotations to the Ingress above:
```yaml
alb.ingress.kubernetes.io/auth-type: oidc
alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate
alb.ingress.kubernetes.io/auth-scope: openid email profile
alb.ingress.kubernetes.io/auth-idp-oidc: |
{"issuer":"https://",
"authorizationEndpoint":"https:///v1/authorize",
"tokenEndpoint":"https:///v1/token",
"userInfoEndpoint":"https:///v1/userinfo",
"secretName":"apps-oidc"}
```
The controller reads the OIDC `clientID` and `clientSecret` from the Secret named by
`secretName`, which **must live in the same namespace as the Ingress** (`kourier-system`):
```bash
kubectl create secret generic apps-oidc -n kourier-system \
--from-literal=clientID= \
--from-literal=clientSecret=
```
Grant the controller permission to read Secrets in that namespace (its Secret access is
per-namespace). Without this the Ingress fails to reconcile with `secrets "apps-oidc" is
forbidden`, which stalls reconciliation of the **whole ALB group**, not just this Ingress:
```yaml
# kourier-oidc-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: alb-oidc-secret-reader
namespace: kourier-system
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: alb-oidc-secret-reader
namespace: kourier-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: alb-oidc-secret-reader
subjects:
- kind: ServiceAccount
name: aws-load-balancer-controller # your controller's ServiceAccount
namespace: kube-system
```
Finally, allow the ALB's callback on your OIDC client. The ALB redirects back to
`https:///oauth2/idpresponse`, and every app has a different hostname, so register
the wildcard `https://*./oauth2/idpresponse` as a sign-in redirect URI
(your IdP must permit wildcard redirect URIs). Without it, login dead-ends after the redirect
with a `redirect_uri` error.
## 4. Enable app serving in Flyte
Turn on the app controller in your Flyte values, then upgrade the release:
```yaml
configuration:
inline:
internalApps:
enabled: true
baseDomain: # must match config-domain from step 2
scheme: https
ingressAppsPort: 0 # apps sit behind the LB on 443; omit the port
```
```bash
helm upgrade flyte flyteorg/flyte-binary -n flyte -f values.yaml
kubectl -n flyte rollout status deploy/flyte
```
> **π Note**
>
> `baseDomain` must match the domain you set in `config-domain`, so the URLs Flyte
> advertises line up with the hostnames Knative actually serves β and with your wildcard
> certificate.
## 5. Verify
Confirm the Flyte ServiceAccount can manage Knative resources:
```bash
kubectl auth can-i create services.serving.knative.dev \
--as=system:serviceaccount:flyte:flyte -n flyte # -> yes
```
Port-forward the API and call the app service β it should return an empty list (`{}`),
not a 404:
```bash
kubectl -n flyte port-forward service/flyte-http 8090:8090
```
```bash
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
http://localhost:8090/flyteidl2.app.AppService/List \
-H 'Content-Type: application/json' -d '{}' # -> 200
```
The console's **Apps** section now loads. Deploy an app with the SDK and open its URL at
`https://--.`.
## Troubleshooting
- **`AppService/List` returns 404 / "unimplemented".** App serving is disabled or the
controller can't start. Check that `internalApps.enabled: true` and that the
`serving.knative.dev` RBAC rules are present on the Flyte ClusterRole.
- **An app URL shows a TLS certificate error.** The hostname has more labels than your
wildcard certificate covers. Confirm the single-label `domain-template` (step 2) and
that `baseDomain` matches `config-domain`.
- **An app URL doesn't resolve.** The wildcard DNS record `*.` isn't
pointing at the load balancer in front of Kourier.
- **Knative manifests are rejected on apply.** The Knative release is newer than your
Kubernetes version supports β install an older Knative version (step 1).