Flyte Scheduler Configuration

• admin
• auth
• catalog-cache
• cloudevents
• cluster_resources
• clusterpools
• clusters
• database
• domains
• event
• externalevents
• flyteadmin
• logger
• namespace_mapping
• notifications
• otel
• plugins
• propeller
• qualityofservice
• queues
• registration
• remotedata
• scheduler
• secrets
• server
• storage
• task_resources
• task_type_whitelist
• tasks

Section: admin

endpoint ( config.URL)

For admin types, specify where the uri of the service is located.

Default Value:

""

insecure (bool)

Use insecure connection.

Default Value:

"false"

insecureSkipVerify (bool)

InsecureSkipVerify controls whether a client verifies the server's certificate chain and host name. Caution : shouldn't be use for production usecases'

Default Value:

"false"

caCertFilePath (string)

Use specified certificate file to verify the admin server peer.

Default Value:

""

maxBackoffDelay ( config.Duration)

Max delay for grpc backoff

Default Value:

8s

perRetryTimeout ( config.Duration)

gRPC per retry timeout

Default Value:

15s

maxRetries (int)

Max number of gRPC retries

Default Value:

"4"

maxMessageSizeBytes (int)

The max size in bytes for incoming gRPC messages

Default Value:

"0"

authType (uint8)

Type of OAuth2 flow used for communicating with admin.ClientSecret,Pkce,ExternalCommand are valid values

Default Value:

ClientSecret

tokenRefreshWindow ( config.Duration)

Max duration between token refresh attempt and token expiry.

Default Value:

0s

useAuth (bool)

Deprecated: Auth will be enabled/disabled based on admin's dynamically discovered information.

Default Value:

"false"

clientId (string)

Client ID

Default Value:

flytepropeller

clientSecretLocation (string)

File containing the client secret

Default Value:

/etc/secrets/client_secret

clientSecretEnvVar (string)

Environment variable containing the client secret

Default Value:

""

scopes ([]string)

List of scopes to request

Default Value:

[]

useAudienceFromAdmin (bool)

Use Audience configured from admins public endpoint config.

Default Value:

"false"

audience (string)

Audience to use when initiating OAuth2 authorization requests.

Default Value:

""

authorizationServerUrl (string)

This is the URL to your IdP's authorization server. It'll default to Endpoint

Default Value:

""

tokenUrl (string)

OPTIONAL: Your IdP's token endpoint. It'll be discovered from flyte admin's OAuth Metadata endpoint if not provided.

Default Value:

""

authorizationHeader (string)

Custom metadata header to pass JWT

Default Value:

""

pkceConfig ( pkce.Config)

Config for Pkce authentication flow.

Default Value:

refreshTime: 5m0s
timeout: 2m0s

deviceFlowConfig ( deviceflow.Config)

Config for Device authentication flow.

Default Value:

pollInterval: 5s
refreshTime: 5m0s
timeout: 10m0s

command ([]string)

Command for external authentication token generation

Default Value:

[]

proxyCommand ([]string)

Command for external proxy-authorization token generation

Default Value:

[]

defaultServiceConfig (string)

Default Value:

""

httpProxyURL ( config.URL)

OPTIONAL: HTTP Proxy to be used for OAuth requests.

Default Value:

""

config.Duration

Duration (int64)

Default Value:

8s

config.URL

URL ( url.URL)

Default Value:

ForceQuery: false
Fragment: ""
Host: ""
OmitHost: false
Opaque: ""
Path: ""
RawFragment: ""
RawPath: ""
RawQuery: ""
Scheme: ""
User: null

url.URL

Scheme (string)

Default Value:

""

Opaque (string)

Default Value:

""

User (url.Userinfo)

Default Value:

null

Host (string)

Default Value:

""

Path (string)

Default Value:

""

RawPath (string)

Default Value:

""

OmitHost (bool)

Default Value:

"false"

ForceQuery (bool)

Default Value:

"false"

RawQuery (string)

Default Value:

""

Fragment (string)

Default Value:

""

RawFragment (string)

Default Value:

""

deviceflow.Config

refreshTime ( config.Duration)

grace period from the token expiry after which it would refresh the token.

Default Value:

5m0s

timeout ( config.Duration)

amount of time the device flow should complete or else it will be cancelled.

Default Value:

10m0s

pollInterval ( config.Duration)

amount of time the device flow would poll the token endpoint if auth server doesn't return a polling interval. Okta and google IDP do return an interval'

Default Value:

5s

pkce.Config

timeout ( config.Duration)

Amount of time the browser session would be active for authentication from client app.

Default Value:

2m0s

refreshTime ( config.Duration)

grace period from the token expiry after which it would refresh the token.

Default Value:

5m0s

Section: auth

httpAuthorizationHeader (string)

Default Value:

flyte-authorization

grpcAuthorizationHeader (string)

Default Value:

flyte-authorization

disableForHttp (bool)

Disables auth enforcement on HTTP Endpoints.

Default Value:

"false"

disableForGrpc (bool)

Disables auth enforcement on Grpc Endpoints.

Default Value:

"false"

authorizedUris ([]config.URL)

Default Value:

null

httpProxyURL ( config.URL)

OPTIONAL: HTTP Proxy to be used for OAuth requests.

Default Value:

""

userAuth ( config.UserAuthConfig)

Defines Auth options for users.

Default Value:

cookieBlockKeySecretName: cookie_block_key
cookieHashKeySecretName: cookie_hash_key
cookieSetting:
  domain: ""
  sameSitePolicy: DefaultMode
httpProxyURL: ""
idpQueryParameter: ""
openId:
  baseUrl: ""
  clientId: ""
  clientSecretFile: ""
  clientSecretName: oidc_client_secret
  scopes:
  - openid
  - profile
redirectUrl: /console

appAuth ( config.OAuth2Options)

Defines Auth options for apps. UserAuth must be enabled for AppAuth to work.

Default Value:

authServerType: Self
externalAuthServer:
  allowedAudience: []
  baseUrl: ""
  httpProxyURL: ""
  metadataUrl: ""
  retryAttempts: 5
  retryDelay: 1s
selfAuthServer:
  accessTokenLifespan: 30m0s
  authorizationCodeLifespan: 5m0s
  claimSymmetricEncryptionKeySecretName: claim_symmetric_key
  issuer: ""
  oldTokenSigningRSAKeySecretName: token_rsa_key_old.pem
  refreshTokenLifespan: 1h0m0s
  staticClients:
    flyte-cli:
      audience: null
      grant_types:
      - refresh_token
      - authorization_code
      id: flyte-cli
      public: true
      redirect_uris:
      - http://localhost:53593/callback
      - http://localhost:12345/callback
      response_types:
      - code
      - token
      scopes:
      - all
      - offline
      - access_token
    flytectl:
      audience: null
      grant_types:
      - refresh_token
      - authorization_code
      id: flytectl
      public: true
      redirect_uris:
      - http://localhost:53593/callback
      - http://localhost:12345/callback
      response_types:
      - code
      - token
      scopes:
      - all
      - offline
      - access_token
    flytepropeller:
      audience: null
      client_secret: JDJhJDA2JGQ2UFFuMlFBRlUzY0w1VjhNRGtldXVrNjN4dWJxVXhOeGp0ZlB3LkZjOU1nVjZ2cG15T0l5
      grant_types:
      - refresh_token
      - client_credentials
      id: flytepropeller
      public: false
      redirect_uris:
      - http://localhost:3846/callback
      response_types:
      - token
      scopes:
      - all
      - offline
      - access_token
  tokenSigningRSAKeySecretName: token_rsa_key.pem
thirdPartyConfig:
  flyteClient:
    audience: ""
    clientId: flytectl
    redirectUri: http://localhost:53593/callback
    scopes:
    - all
    - offline

config.OAuth2Options

authServerType (int)

Default Value:

Self

selfAuthServer ( config.AuthorizationServer)

Authorization Server config to run as a service. Use this when using an IdP that does not offer a custom OAuth2 Authorization Server.

Default Value:

accessTokenLifespan: 30m0s
authorizationCodeLifespan: 5m0s
claimSymmetricEncryptionKeySecretName: claim_symmetric_key
issuer: ""
oldTokenSigningRSAKeySecretName: token_rsa_key_old.pem
refreshTokenLifespan: 1h0m0s
staticClients:
  flyte-cli:
    audience: null
    grant_types:
    - refresh_token
    - authorization_code
    id: flyte-cli
    public: true
    redirect_uris:
    - http://localhost:53593/callback
    - http://localhost:12345/callback
    response_types:
    - code
    - token
    scopes:
    - all
    - offline
    - access_token
  flytectl:
    audience: null
    grant_types:
    - refresh_token
    - authorization_code
    id: flytectl
    public: true
    redirect_uris:
    - http://localhost:53593/callback
    - http://localhost:12345/callback
    response_types:
    - code
    - token
    scopes:
    - all
    - offline
    - access_token
  flytepropeller:
    audience: null
    client_secret: JDJhJDA2JGQ2UFFuMlFBRlUzY0w1VjhNRGtldXVrNjN4dWJxVXhOeGp0ZlB3LkZjOU1nVjZ2cG15T0l5
    grant_types:
    - refresh_token
    - client_credentials
    id: flytepropeller
    public: false
    redirect_uris:
    - http://localhost:3846/callback
    response_types:
    - token
    scopes:
    - all
    - offline
    - access_token
tokenSigningRSAKeySecretName: token_rsa_key.pem

externalAuthServer ( config.ExternalAuthorizationServer)

External Authorization Server config.

Default Value:

allowedAudience: []
baseUrl: ""
httpProxyURL: ""
metadataUrl: ""
retryAttempts: 5
retryDelay: 1s

thirdPartyConfig ( config.ThirdPartyConfigOptions)

Defines settings to instruct flyte cli tools (and optionally others) on what config to use to setup their client.

Default Value:

flyteClient:
  audience: ""
  clientId: flytectl
  redirectUri: http://localhost:53593/callback
  scopes:
  - all
  - offline

config.AuthorizationServer

issuer (string)

Defines the issuer to use when issuing and validating tokens. The default value is https://<requestUri.HostAndPort>/

Default Value:

""

accessTokenLifespan ( config.Duration)

Defines the lifespan of issued access tokens.

Default Value:

30m0s

refreshTokenLifespan ( config.Duration)

Defines the lifespan of issued access tokens.

Default Value:

1h0m0s

authorizationCodeLifespan ( config.Duration)

Defines the lifespan of issued access tokens.

Default Value:

5m0s

claimSymmetricEncryptionKeySecretName (string)

OPTIONAL: Secret name to use to encrypt claims in authcode token.

Default Value:

claim_symmetric_key

tokenSigningRSAKeySecretName (string)

OPTIONAL: Secret name to use to retrieve RSA Signing Key.

Default Value:

token_rsa_key.pem

oldTokenSigningRSAKeySecretName (string)

OPTIONAL: Secret name to use to retrieve Old RSA Signing Key. This can be useful during key rotation to continue to accept older tokens.

Default Value:

token_rsa_key_old.pem

staticClients (map[string]*fosite.DefaultClient)

Default Value:

flyte-cli:
  audience: null
  grant_types:
  - refresh_token
  - authorization_code
  id: flyte-cli
  public: true
  redirect_uris:
  - http://localhost:53593/callback
  - http://localhost:12345/callback
  response_types:
  - code
  - token
  scopes:
  - all
  - offline
  - access_token
flytectl:
  audience: null
  grant_types:
  - refresh_token
  - authorization_code
  id: flytectl
  public: true
  redirect_uris:
  - http://localhost:53593/callback
  - http://localhost:12345/callback
  response_types:
  - code
  - token
  scopes:
  - all
  - offline
  - access_token
flytepropeller:
  audience: null
  client_secret: JDJhJDA2JGQ2UFFuMlFBRlUzY0w1VjhNRGtldXVrNjN4dWJxVXhOeGp0ZlB3LkZjOU1nVjZ2cG15T0l5
  grant_types:
  - refresh_token
  - client_credentials
  id: flytepropeller
  public: false
  redirect_uris:
  - http://localhost:3846/callback
  response_types:
  - token
  scopes:
  - all
  - offline
  - access_token

config.ExternalAuthorizationServer

baseUrl ( config.URL)

This should be the base url of the authorization server that you are trying to hit. With Okta for instance, it will look something like https://company.okta.com/oauth2/abcdef123456789/

Default Value:

""

allowedAudience ([]string)

Optional: A list of allowed audiences. If not provided, the audience is expected to be the public Uri of the service.

Default Value:

[]

metadataUrl ( config.URL)

Optional: If the server doesn't support /.well-known/oauth-authorization-server, you can set a custom metadata url here.'

Default Value:

""

httpProxyURL ( config.URL)

OPTIONAL: HTTP Proxy to be used for OAuth requests.

Default Value:

""

retryAttempts (int)

Optional: The number of attempted retries on a transient failure to get the OAuth metadata

Default Value:

"5"

retryDelay ( config.Duration)

Optional, Duration to wait between retries

Default Value:

1s

config.ThirdPartyConfigOptions

flyteClient ( config.FlyteClientConfig)

Default Value:

audience: ""
clientId: flytectl
redirectUri: http://localhost:53593/callback
scopes:
- all
- offline

config.FlyteClientConfig

clientId (string)

public identifier for the app which handles authorization for a Flyte deployment

Default Value:

flytectl

redirectUri (string)

This is the callback uri registered with the app which handles authorization for a Flyte deployment

Default Value:

http://localhost:53593/callback

scopes ([]string)

Recommended scopes for the client to request.

Default Value:

- all
- offline

audience (string)

Audience to use when initiating OAuth2 authorization requests.

Default Value:

""

config.UserAuthConfig

redirectUrl ( config.URL)

Default Value:

/console

openId ( config.OpenIDOptions)

OpenID Configuration for User Auth

Default Value:

baseUrl: ""
clientId: ""
clientSecretFile: ""
clientSecretName: oidc_client_secret
scopes:
- openid
- profile

httpProxyURL ( config.URL)

OPTIONAL: HTTP Proxy to be used for OAuth requests.

Default Value:

""

cookieHashKeySecretName (string)

OPTIONAL: Secret name to use for cookie hash key.

Default Value:

cookie_hash_key

cookieBlockKeySecretName (string)

OPTIONAL: Secret name to use for cookie block key.

Default Value:

cookie_block_key

cookieSetting ( config.CookieSettings)

settings used by cookies created for user auth

Default Value:

domain: ""
sameSitePolicy: DefaultMode

idpQueryParameter (string)

idp query parameter used for selecting a particular IDP for doing user authentication. Eg: for Okta passing idp=<IDP-ID> forces the authentication to happen with IDP-ID

Default Value:

""

config.CookieSettings

sameSitePolicy (int)

OPTIONAL: Allows you to declare if your cookie should be restricted to a first-party or same-site context.Wrapper around http.SameSite.

Default Value:

DefaultMode

domain (string)

OPTIONAL: Allows you to set the domain attribute on the auth cookies.

Default Value:

""

config.OpenIDOptions

clientId (string)

Default Value:

""

clientSecretName (string)

Default Value:

oidc_client_secret

clientSecretFile (string)

Default Value:

""

baseUrl ( config.URL)

Default Value:

""

scopes ([]string)

Default Value:

- openid
- profile

Section: catalog-cache

type (string)

Catalog Implementation to use

Default Value:

noop

endpoint (string)

Endpoint for catalog service

Default Value:

""

insecure (bool)

Use insecure grpc connection

Default Value:

"false"

max-cache-age ( config.Duration)

Cache entries past this age will incur cache miss. 0 means cache never expires

Default Value:

0s

use-admin-auth (bool)

Use the same gRPC credentials option as the flyteadmin client

Default Value:

"false"

max-retries (int)

The max number of retries for event recording.

Default Value:

"5"

base-scalar (int)

The base/scalar backoff duration in milliseconds for event recording retries.

Default Value:

"100"

backoff-jitter (string)

A string representation of a floating point number between 0 and 1 specifying the jitter factor for event recording retries.

Default Value:

"0.1"

default-service-config (string)

Set the default service config for the catalog gRPC client

Default Value:

""

Section: cloudevents

enable (bool)

Default Value:

"false"

type (string)

Default Value:

local

aws ( interfaces.AWSConfig)

Default Value:

region: ""

gcp ( interfaces.GCPConfig)

Default Value:

projectId: ""

kafka ( interfaces.KafkaConfig)

Default Value:

brokers: null
saslConfig:
  enabled: false
  handshake: false
  mechanism: ""
  password: ""
  passwordPath: ""
  user: ""
tlsConfig:
  certPath: ""
  enabled: false
  insecureSkipVerify: false
  keyPath: ""
version: ""

eventsPublisher ( interfaces.EventsPublisherConfig)

Default Value:

enrichAllWorkflowEventTypes: false
eventTypes: null
topicName: ""

reconnectAttempts (int)

Default Value:

"0"

reconnectDelaySeconds (int)

Default Value:

"0"

cloudEventVersion (uint8)

Default Value:

v1

interfaces.AWSConfig

region (string)

Default Value:

""

interfaces.EventsPublisherConfig

topicName (string)

Default Value:

""

eventTypes ([]string)

Default Value:

null

enrichAllWorkflowEventTypes (bool)

Default Value:

"false"

interfaces.GCPConfig

projectId (string)

Default Value:

""

interfaces.KafkaConfig

version (string)

Default Value:

""

brokers ([]string)

Default Value:

null

saslConfig ( interfaces.SASLConfig)

Default Value:

enabled: false
handshake: false
mechanism: ""
password: ""
passwordPath: ""
user: ""

tlsConfig ( interfaces.TLSConfig)

Default Value:

certPath: ""
enabled: false
insecureSkipVerify: false
keyPath: ""

interfaces.SASLConfig

enabled (bool)

Default Value:

"false"

user (string)

Default Value:

""

password (string)

Default Value:

""

passwordPath (string)

Default Value:

""

handshake (bool)

Default Value:

"false"

mechanism (string)

Default Value:

""

interfaces.TLSConfig

enabled (bool)

Default Value:

"false"

insecureSkipVerify (bool)

Default Value:

"false"

certPath (string)

Default Value:

""

keyPath (string)

Default Value:

""

Section: cluster_resources

templatePath (string)

Default Value:

""

templateData (map[string]interfaces.DataSource)

Default Value:

{}

refreshInterval ( config.Duration)

Default Value:

1m0s

customData (map[string]map[string]interfaces.DataSource)

Default Value:

{}

standaloneDeployment (bool)

Whether the cluster resource sync is running in a standalone deployment and should call flyteadmin service endpoints

Default Value:

"false"

Section: clusterpools

clusterPoolAssignments (map[string]interfaces.ClusterPoolAssignment)

Default Value:

{}

Section: clusters

clusterConfigs ([]interfaces.ClusterConfig)

Default Value:

null

labelClusterMap (map[string][]interfaces.ClusterEntity)

Default Value:

null

defaultExecutionLabel (string)

Default Value:

""

Section: database

host (string)

Default Value:

""

port (int)

Default Value:

"0"

dbname (string)

Default Value:

""

username (string)

Default Value:

""

password (string)

Default Value:

""

passwordPath (string)

Default Value:

""

options (string)

Default Value:

""

debug (bool)

Default Value:

"false"

enableForeignKeyConstraintWhenMigrating (bool)

Whether to enable gorm foreign keys when migrating the db

Default Value:

"false"

maxIdleConnections (int)

maxIdleConnections sets the maximum number of connections in the idle connection pool.

Default Value:

"10"

maxOpenConnections (int)

maxOpenConnections sets the maximum number of open connections to the database.

Default Value:

"100"

connMaxLifeTime ( config.Duration)

sets the maximum amount of time a connection may be reused

Default Value:

1h0m0s

postgres ( database.PostgresConfig)

Default Value:

dbname: postgres
debug: false
host: localhost
options: sslmode=disable
password: postgres
passwordPath: ""
port: 30001
readReplicaHost: localhost
username: postgres

sqlite ( database.SQLiteConfig)

Default Value:

file: ""

database.PostgresConfig

host (string)

The host name of the database server

Default Value:

localhost

readReplicaHost (string)

The host name of the read replica database server

Default Value:

localhost

port (int)

The port name of the database server

Default Value:

"30001"

dbname (string)

The database name

Default Value:

postgres

username (string)

The database user who is connecting to the server.

Default Value:

postgres

password (string)

The database password.

Default Value:

postgres

passwordPath (string)

Points to the file containing the database password.

Default Value:

""

options (string)

See http://gorm.io/docs/connecting_to_the_database.html for available options passed, in addition to the above.

Default Value:

sslmode=disable

debug (bool)

Whether or not to start the database connection with debug mode enabled.

Default Value:

"false"

database.SQLiteConfig

file (string)

The path to the file (existing or new) where the DB should be created / stored. If existing, then this will be reused, else a new will be created

Default Value:

""

Section: domains

id (string)

Default Value:

development

name (string)

Default Value:

development

Section: event

type (string)

Sets the type of EventSink to configure [log/admin/file].

Default Value:

admin

file-path (string)

For file types, specify where the file should be located.

Default Value:

""

rate (int64)

Max rate at which events can be recorded per second.

Default Value:

"500"

capacity (int)

The max bucket size for event recording tokens.

Default Value:

"1000"

max-retries (int)

The max number of retries for event recording.

Default Value:

"5"

base-scalar (int)

The base/scalar backoff duration in milliseconds for event recording retries.

Default Value:

"100"

backoff-jitter (string)

A string representation of a floating point number between 0 and 1 specifying the jitter factor for event recording retries.

Default Value:

"0.1"

Section: externalevents

enable (bool)

Default Value:

"false"

type (string)

Default Value:

local

aws ( interfaces.AWSConfig)

Default Value:

region: ""

gcp ( interfaces.GCPConfig)

Default Value:

projectId: ""

eventsPublisher ( interfaces.EventsPublisherConfig)

Default Value:

enrichAllWorkflowEventTypes: false
eventTypes: null
topicName: ""

reconnectAttempts (int)

Default Value:

"0"

reconnectDelaySeconds (int)

Default Value:

"0"

Section: flyteadmin

roleNameKey (string)

Default Value:

""

metricsScope (string)

Default Value:

'flyte:'

metricsKeys ([]string)

Default Value:

- project
- domain
- wf
- task
- phase
- tasktype
- runtime_type
- runtime_version
- app_name

profilerPort (int)

Default Value:

"10254"

metadataStoragePrefix ([]string)

Default Value:

- metadata
- admin

eventVersion (int)

Default Value:

"2"

asyncEventsBufferSize (int)

Default Value:

"100"

maxParallelism (int32)

Default Value:

"25"

labels (map[string]string)

Default Value:

null

annotations (map[string]string)

Default Value:

null

interruptible (bool)

Default Value:

"false"

overwriteCache (bool)

Default Value:

"false"

assumableIamRole (string)

Default Value:

""

k8sServiceAccount (string)

Default Value:

""

outputLocationPrefix (string)

Default Value:

""

useOffloadedWorkflowClosure (bool)

Default Value:

"false"

envs (map[string]string)

Default Value:

null

featureGates ( interfaces.FeatureGates)

Enable experimental features.

Default Value:

enableArtifacts: false

consoleUrl (string)

A URL pointing to the flyteconsole instance used to hit this flyteadmin instance.

Default Value:

""

useOffloadedInputs (bool)

Use offloaded inputs for workflows.

Default Value:

"false"

interfaces.FeatureGates

enableArtifacts (bool)

Enable artifacts feature.

Default Value:

"false"

Section: logger

show-source (bool)

Includes source code location in logs.

Default Value:

"false"

mute (bool)

Mutes all logs regardless of severity. Intended for benchmarks/tests only.

Default Value:

"false"

level (int)

Sets the minimum logging level.

Default Value:

"3"

formatter ( logger.FormatterConfig)

Sets logging format.

Default Value:

type: json

logger.FormatterConfig

type (string)

Sets logging format type.

Default Value:

json

Section: namespace_mapping

mapping (string)

Default Value:

""

template (string)

Default Value:

'{{ project }}-{{ domain }}'

templateData (map[string]interfaces.DataSource)

Default Value:

null

Section: notifications

type (string)

Default Value:

local

region (string)

Default Value:

""

aws ( interfaces.AWSConfig)

Default Value:

region: ""

gcp ( interfaces.GCPConfig)

Default Value:

projectId: ""

publisher ( interfaces.NotificationsPublisherConfig)

Default Value:

topicName: ""

processor ( interfaces.NotificationsProcessorConfig)

Default Value:

accountId: ""
queueName: ""

emailer ( interfaces.NotificationsEmailerConfig)

Default Value:

body: ""
emailServerConfig:
  apiKeyEnvVar: ""
  apiKeyFilePath: ""
  serviceName: ""
  smtpPasswordSecretName: ""
  smtpPort: ""
  smtpServer: ""
  smtpSkipTLSVerify: false
  smtpUsername: ""
sender: ""
subject: ""

reconnectAttempts (int)

Default Value:

"0"

reconnectDelaySeconds (int)

Default Value:

"0"

interfaces.NotificationsEmailerConfig

emailServerConfig ( interfaces.EmailServerConfig)

Default Value:

apiKeyEnvVar: ""
apiKeyFilePath: ""
serviceName: ""
smtpPasswordSecretName: ""
smtpPort: ""
smtpServer: ""
smtpSkipTLSVerify: false
smtpUsername: ""

subject (string)

Default Value:

""

sender (string)

Default Value:

""

body (string)

Default Value:

""

interfaces.EmailServerConfig

serviceName (string)

Default Value:

""

apiKeyEnvVar (string)

Default Value:

""

apiKeyFilePath (string)

Default Value:

""

smtpServer (string)

Default Value:

""

smtpPort (string)

Default Value:

""

smtpSkipTLSVerify (bool)

Default Value:

"false"

smtpUsername (string)

Default Value:

""

smtpPasswordSecretName (string)

Default Value:

""

interfaces.NotificationsProcessorConfig

queueName (string)

Default Value:

""

accountId (string)

Default Value:

""

interfaces.NotificationsPublisherConfig

topicName (string)

Default Value:

""

Section: otel

type (string)

Sets the type of exporter to configure [noop/file/jaeger/otlpgrpc/otlphttp].

Default Value:

noop

file ( otelutils.FileConfig)

Configuration for exporting telemetry traces to a file

Default Value:

filename: /tmp/trace.txt

jaeger ( otelutils.JaegerConfig)

Configuration for exporting telemetry traces to a jaeger

Default Value:

endpoint: http://localhost:14268/api/traces

otlpgrpc ( otelutils.OtlpGrpcConfig)

Configuration for exporting telemetry traces to an OTLP gRPC collector

Default Value:

endpoint: http://localhost:4317

otlphttp ( otelutils.OtlpHttpConfig)

Configuration for exporting telemetry traces to an OTLP HTTP collector

Default Value:

endpoint: http://localhost:4318/v1/traces

sampler ( otelutils.SamplerConfig)

Configuration for the sampler to use for the tracer

Default Value:

parentSampler: always
traceIdRatio: 0.01

otelutils.FileConfig

filename (string)

Filename to store exported telemetry traces

Default Value:

/tmp/trace.txt

otelutils.JaegerConfig

endpoint (string)

Endpoint for the jaeger telemetry trace ingestor

Default Value:

http://localhost:14268/api/traces

otelutils.OtlpGrpcConfig

endpoint (string)

Endpoint for the OTLP telemetry trace collector

Default Value:

http://localhost:4317

otelutils.OtlpHttpConfig

endpoint (string)

Endpoint for the OTLP telemetry trace collector

Default Value:

http://localhost:4318/v1/traces

otelutils.SamplerConfig

parentSampler (string)

Sets the parent sampler to use for the tracer

Default Value:

always

traceIdRatio (float64)

Default Value:

"0.01"

Section: plugins

connector-service ( connector.Config)

Default Value:

connectorForTaskTypes: null
connectors: null
defaultConnector:
  defaultServiceConfig: '{"loadBalancingConfig": [{"round_robin":{}}]}'
  defaultTimeout: 3s
  endpoint: ""
  insecure: true
  timeouts: null
pollInterval: 10s
resourceConstraints:
  NamespaceScopeResourceConstraint:
    Value: 50
  ProjectScopeResourceConstraint:
    Value: 100
supportedTaskTypes:
- task_type_1
- task_type_2
webApi:
  caching:
    maxSystemFailures: 5
    resyncInterval: 30s
    size: 500000
    workers: 10
  readRateLimiter:
    burst: 100
    qps: 10
  resourceMeta: null
  resourceQuotas:
    default: 1000
  writeRateLimiter:
    burst: 100
    qps: 10

catalogcache ( catalog.Config)

Default Value:

reader:
  maxItems: 10000
  maxRetries: 3
  workers: 10
writer:
  maxItems: 10000
  maxRetries: 3
  workers: 10

connector-service ( connector.Config)

Default Value:

connectorForTaskTypes: {}
connectors: {}
defaultConnector:
  defaultServiceConfig: '{"loadBalancingConfig": [{"round_robin":{}}]}'
  defaultTimeout: 10s
  endpoint: ""
  insecure: true
  timeouts: null
pollInterval: 10s
resourceConstraints:
  NamespaceScopeResourceConstraint:
    Value: 50
  ProjectScopeResourceConstraint:
    Value: 100
supportedTaskTypes:
- task_type_3
- task_type_4
webApi:
  caching:
    maxSystemFailures: 5
    resyncInterval: 30s
    size: 500000
    workers: 10
  readRateLimiter:
    burst: 100
    qps: 10
  resourceMeta: null
  resourceQuotas:
    default: 1000
  writeRateLimiter:
    burst: 100
    qps: 10

k8s ( config.K8sPluginConfig)

Default Value:

add-tolerations-for-extended-resources: []
co-pilot:
  cpu: 500m
  default-input-path: /var/flyte/inputs
  default-output-path: /var/flyte/outputs
  image: cr.flyte.org/flyteorg/flytecopilot:v0.0.15
  input-vol-name: flyte-inputs
  memory: 128Mi
  name: flyte-copilot-
  output-vol-name: flyte-outputs
  start-timeout: 1m40s
  storage: ""
create-container-config-error-grace-period: 0s
create-container-error-grace-period: 3m0s
default-annotations:
  cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
default-cpus: "1"
default-env-from-configmaps: null
default-env-from-secrets: null
default-env-vars: null
default-env-vars-from-env: null
default-labels: null
default-memory: 1Gi
default-node-selector: null
default-pod-dns-config: null
default-pod-security-context: null
default-pod-template-name: ""
default-pod-template-resync: 30s
default-security-context: null
default-tolerations: null
delete-resource-on-finalize: false
enable-distributed-error-aggregation: false
enable-host-networking-pod: null
gpu-device-node-label: k8s.amazonaws.com/accelerator
gpu-partition-size-node-label: k8s.amazonaws.com/gpu-partition-size
gpu-resource-name: nvidia.com/gpu
gpu-unpartitioned-node-selector-requirement: null
gpu-unpartitioned-toleration: null
image-pull-backoff-grace-period: 3m0s
image-pull-policy: ""
inject-finalizer: false
interruptible-node-selector: null
interruptible-node-selector-requirement: null
interruptible-tolerations: null
non-interruptible-node-selector-requirement: null
pod-pending-timeout: 0s
resource-tolerations: null
scheduler-name: ""
send-object-events: false
update-backoff-retries: 5
update-base-backoff-duration: 10

k8s-array ( k8s.Config)

Default Value:

ErrorAssembler:
  maxItems: 100000
  maxRetries: 5
  workers: 10
OutputAssembler:
  maxItems: 100000
  maxRetries: 5
  workers: 10
logs:
  config:
    cloudwatch-enabled: false
    cloudwatch-log-group: ""
    cloudwatch-region: ""
    cloudwatch-template-uri: ""
    dynamic-log-links: null
    gcp-project: ""
    kubernetes-enabled: true
    kubernetes-template-uri: http://localhost:30082/#!/log/{{ .namespace }}/{{ .podName
      }}/pod?namespace={{ .namespace }}
    kubernetes-url: ""
    stackdriver-enabled: false
    stackdriver-logresourcename: ""
    stackdriver-template-uri: ""
    templates: null
maxArrayJobSize: 5000
maxErrorLength: 1000
namespaceTemplate: ""
node-selector: null
remoteClusterConfig:
  auth:
    certPath: ""
    tokenPath: ""
    type: ""
  enabled: false
  endpoint: ""
  name: ""
resourceConfig:
  limit: 0
  primaryLabel: ""
scheduler: ""
tolerations: null

logs ( logs.LogConfig)

Default Value:

cloudwatch-enabled: false
cloudwatch-log-group: ""
cloudwatch-region: ""
cloudwatch-template-uri: ""
dynamic-log-links: null
gcp-project: ""
kubernetes-enabled: true
kubernetes-template-uri: http://localhost:30082/#!/log/{{ .namespace }}/{{ .podName
  }}/pod?namespace={{ .namespace }}
kubernetes-url: ""
stackdriver-enabled: false
stackdriver-logresourcename: ""
stackdriver-template-uri: ""
templates: null

connector.Config

webApi ( webapi.PluginConfig)

Defines config for the base WebAPI plugin.

Default Value:

caching:
  maxSystemFailures: 5
  resyncInterval: 30s
  size: 500000
  workers: 10
readRateLimiter:
  burst: 100
  qps: 10
resourceMeta: null
resourceQuotas:
  default: 1000
writeRateLimiter:
  burst: 100
  qps: 10

resourceConstraints ( core.ResourceConstraintsSpec)

Default Value:

NamespaceScopeResourceConstraint:
  Value: 50
ProjectScopeResourceConstraint:
  Value: 100

defaultConnector ( connector.Deployment)

The default connector.

Default Value:

defaultServiceConfig: '{"loadBalancingConfig": [{"round_robin":{}}]}'
defaultTimeout: 3s
endpoint: ""
insecure: true
timeouts: null

connectors (map[string]*connector.Deployment)

The connectors.

Default Value:

null

connectorForTaskTypes (map[string]string)

Default Value:

null

supportedTaskTypes ([]string)

Default Value:

- task_type_1
- task_type_2

pollInterval ( config.Duration)

The interval at which the plugin should poll the connector for metadata updates.

Default Value:

10s

connector.Deployment

endpoint (string)

Default Value:

""

insecure (bool)

Default Value:

"true"

defaultServiceConfig (string)

Default Value:

'{"loadBalancingConfig": [{"round_robin":{}}]}'

timeouts (map[string]config.Duration)

Default Value:

null

defaultTimeout ( config.Duration)

Default Value:

3s

core.ResourceConstraintsSpec

ProjectScopeResourceConstraint ( core.ResourceConstraint)

Default Value:

Value: 100

NamespaceScopeResourceConstraint ( core.ResourceConstraint)

Default Value:

Value: 50

core.ResourceConstraint

Value (int64)

Default Value:

"100"

webapi.PluginConfig

resourceQuotas (webapi.ResourceQuotas)

Default Value:

default: 1000

readRateLimiter ( webapi.RateLimiterConfig)

Defines rate limiter properties for read actions (e.g. retrieve status).

Default Value:

burst: 100
qps: 10

writeRateLimiter ( webapi.RateLimiterConfig)

Defines rate limiter properties for write actions.

Default Value:

burst: 100
qps: 10

caching ( webapi.CachingConfig)

Defines caching characteristics.

Default Value:

maxSystemFailures: 5
resyncInterval: 30s
size: 500000
workers: 10

resourceMeta (interface)

Default Value:

<nil>

webapi.CachingConfig

size (int)

Defines the maximum number of items to cache.

Default Value:

"500000"

resyncInterval ( config.Duration)

Defines the sync interval.

Default Value:

30s

workers (int)

Defines the number of workers to start up to process items.

Default Value:

"10"

maxSystemFailures (int)

Defines the number of failures to fetch a task before failing the task.

Default Value:

"5"

webapi.RateLimiterConfig

qps (int)

Defines the max rate of calls per second.

Default Value:

"10"

burst (int)

Defines the maximum burst size.

Default Value:

"100"

catalog.Config

reader ( workqueue.Config)

Catalog reader workqueue config. Make sure the index cache must be big enough to accommodate the biggest array task allowed to run on the system.

Default Value:

maxItems: 10000
maxRetries: 3
workers: 10

writer ( workqueue.Config)

Catalog writer workqueue config. Make sure the index cache must be big enough to accommodate the biggest array task allowed to run on the system.

Default Value:

maxItems: 10000
maxRetries: 3
workers: 10

workqueue.Config

workers (int)

Number of concurrent workers to start processing the queue.

Default Value:

"10"

maxRetries (int)

Maximum number of retries per item.

Default Value:

"3"

maxItems (int)

Maximum number of entries to keep in the index.

Default Value:

"10000"

config.K8sPluginConfig

inject-finalizer (bool)

Instructs the plugin to inject a finalizer on startTask and remove it on task termination.

Default Value:

"false"

default-annotations (map[string]string)

Default Value:

cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

default-labels (map[string]string)

Default Value:

null

default-env-vars (map[string]string)

Default Value:

null

default-env-vars-from-env (map[string]string)

Default Value:

null

default-env-from-configmaps ([]string)

Default Value:

null

default-env-from-secrets ([]string)

Default Value:

null

default-cpus ( resource.Quantity)

Defines a default value for cpu for containers if not specified.

Default Value:

"1"

default-memory ( resource.Quantity)

Defines a default value for memory for containers if not specified.

Default Value:

1Gi

default-tolerations ([]v1.Toleration)

Default Value:

null

default-node-selector (map[string]string)

Default Value:

null

default-affinity (v1.Affinity)

Default Value:

null

scheduler-name (string)

Defines scheduler name.

Default Value:

""

interruptible-tolerations ([]v1.Toleration)

Default Value:

null

interruptible-node-selector (map[string]string)

Default Value:

null

interruptible-node-selector-requirement (v1.NodeSelectorRequirement)

Default Value:

null

non-interruptible-node-selector-requirement (v1.NodeSelectorRequirement)

Default Value:

null

resource-tolerations (map[v1.ResourceName][]v1.Toleration)

Default Value:

null

co-pilot ( config.FlyteCoPilotConfig)

Co-Pilot Configuration

Default Value:

cpu: 500m
default-input-path: /var/flyte/inputs
default-output-path: /var/flyte/outputs
image: cr.flyte.org/flyteorg/flytecopilot:v0.0.15
input-vol-name: flyte-inputs
memory: 128Mi
name: flyte-copilot-
output-vol-name: flyte-outputs
start-timeout: 1m40s
storage: ""

delete-resource-on-finalize (bool)

Instructs the system to delete the resource upon successful execution of a k8s pod rather than have the k8s garbage collector clean it up. This ensures that no resources are kept around (potentially consuming cluster resources). This, however, will cause k8s log links to expire as soon as the resource is finalized.

Default Value:

"false"

create-container-error-grace-period ( config.Duration)

Default Value:

3m0s

create-container-config-error-grace-period ( config.Duration)

Default Value:

0s

image-pull-backoff-grace-period ( config.Duration)

Default Value:

3m0s

image-pull-policy (string)

Default Value:

""

pod-pending-timeout ( config.Duration)

Default Value:

0s

gpu-device-node-label (string)

Default Value:

k8s.amazonaws.com/accelerator

gpu-partition-size-node-label (string)

Default Value:

k8s.amazonaws.com/gpu-partition-size

gpu-unpartitioned-node-selector-requirement (v1.NodeSelectorRequirement)

Default Value:

null

gpu-unpartitioned-toleration (v1.Toleration)

Default Value:

null

gpu-resource-name (string)

Default Value:

nvidia.com/gpu

default-pod-security-context (v1.PodSecurityContext)

Default Value:

null

default-security-context (v1.SecurityContext)

Default Value:

null

enable-host-networking-pod (bool)

Default Value:

<invalid reflect.Value>

default-pod-dns-config (v1.PodDNSConfig)

Default Value:

null

default-pod-template-name (string)

Name of the PodTemplate to use as the base for all k8s pods created by FlytePropeller.

Default Value:

""

default-pod-template-resync ( config.Duration)

Frequency of resyncing default pod templates

Default Value:

30s

send-object-events (bool)

If true, will send k8s object events in TaskExecutionEvent updates.

Default Value:

"false"

update-base-backoff-duration (int)

Initial delay in exponential backoff when updating a resource in milliseconds.

Default Value:

"10"

update-backoff-retries (int)

Number of retries for exponential backoff when updating a resource.

Default Value:

"5"

add-tolerations-for-extended-resources ([]string)

Name of the extended resources for which tolerations should be added.

Default Value:

[]

enable-distributed-error-aggregation (bool)

If true, will aggregate errors of different worker pods for distributed tasks.

Default Value:

"false"

config.FlyteCoPilotConfig

name (string)

Flyte co-pilot sidecar container name prefix. (additional bits will be added after this)

Default Value:

flyte-copilot-

image (string)

Flyte co-pilot Docker Image FQN

Default Value:

cr.flyte.org/flyteorg/flytecopilot:v0.0.15

default-input-path (string)

Default path where the volume should be mounted

Default Value:

/var/flyte/inputs

default-output-path (string)

Default path where the volume should be mounted

Default Value:

/var/flyte/outputs

input-vol-name (string)

Name of the data volume that is created for storing inputs

Default Value:

flyte-inputs

output-vol-name (string)

Name of the data volume that is created for storing outputs

Default Value:

flyte-outputs

start-timeout ( config.Duration)

Default Value:

1m40s

cpu (string)

Used to set cpu for co-pilot containers

Default Value:

500m

memory (string)

Used to set memory for co-pilot containers

Default Value:

128Mi

storage (string)

Default storage limit for individual inputs / outputs

Default Value:

""

resource.Quantity

i ( resource.int64Amount)

Default Value:

{}

d ( resource.infDecAmount)

Default Value:

<nil>

s (string)

Default Value:

"1"

Format (string)

Default Value:

DecimalSI

resource.infDecAmount

Dec (inf.Dec)

Default Value:

null

resource.int64Amount

value (int64)

Default Value:

"1"

scale (int32)

Default Value:

"0"

connector.Config

webApi ( webapi.PluginConfig)

Defines config for the base WebAPI plugin.

Default Value:

caching:
  maxSystemFailures: 5
  resyncInterval: 30s
  size: 500000
  workers: 10
readRateLimiter:
  burst: 100
  qps: 10
resourceMeta: null
resourceQuotas:
  default: 1000
writeRateLimiter:
  burst: 100
  qps: 10

resourceConstraints ( core.ResourceConstraintsSpec)

Default Value:

NamespaceScopeResourceConstraint:
  Value: 50
ProjectScopeResourceConstraint:
  Value: 100

defaultConnector ( connector.Deployment)

The default connector.

Default Value:

defaultServiceConfig: '{"loadBalancingConfig": [{"round_robin":{}}]}'
defaultTimeout: 10s
endpoint: ""
insecure: true
timeouts: null

connectors (map[string]*connector.Deployment)

The connectors.

Default Value:

{}

connectorForTaskTypes (map[string]string)

Default Value:

{}

supportedTaskTypes ([]string)

Default Value:

- task_type_3
- task_type_4

pollInterval ( config.Duration)

The interval at which the plugin should poll the connector for metadata updates.

Default Value:

10s

connector.Deployment

endpoint (string)

Default Value:

""

insecure (bool)

Default Value:

"true"

defaultServiceConfig (string)

Default Value:

'{"loadBalancingConfig": [{"round_robin":{}}]}'

timeouts (map[string]config.Duration)

Default Value:

null

defaultTimeout ( config.Duration)

Default Value:

10s

k8s.Config

scheduler (string)

Decides the scheduler to use when launching array-pods.

Default Value:

""

maxErrorLength (int)

Determines the maximum length of the error string returned for the array.

Default Value:

"1000"

maxArrayJobSize (int64)

Maximum size of array job.

Default Value:

"5000"

resourceConfig ( k8s.ResourceConfig)

Default Value:

limit: 0
primaryLabel: ""

remoteClusterConfig ( k8s.ClusterConfig)

Default Value:

auth:
  certPath: ""
  tokenPath: ""
  type: ""
enabled: false
endpoint: ""
name: ""

node-selector (map[string]string)

Default Value:

null

tolerations ([]v1.Toleration)

Default Value:

null

namespaceTemplate (string)

Default Value:

""

OutputAssembler ( workqueue.Config)

Default Value:

maxItems: 100000
maxRetries: 5
workers: 10

ErrorAssembler ( workqueue.Config)

Default Value:

maxItems: 100000
maxRetries: 5
workers: 10

logs ( k8s.LogConfig)

Config for log links for k8s array jobs.

Default Value:

config:
  cloudwatch-enabled: false
  cloudwatch-log-group: ""
  cloudwatch-region: ""
  cloudwatch-template-uri: ""
  dynamic-log-links: null
  gcp-project: ""
  kubernetes-enabled: true
  kubernetes-template-uri: http://localhost:30082/#!/log/{{ .namespace }}/{{ .podName
    }}/pod?namespace={{ .namespace }}
  kubernetes-url: ""
  stackdriver-enabled: false
  stackdriver-logresourcename: ""
  stackdriver-template-uri: ""
  templates: null

k8s.ClusterConfig

name (string)

Friendly name of the remote cluster

Default Value:

""

endpoint (string)

Remote K8s cluster endpoint

Default Value:

""

auth ( k8s.Auth)

Default Value:

certPath: ""
tokenPath: ""
type: ""

enabled (bool)