# Control Plane > This bundle contains all pages in the Control Plane section. > Source: https://www.union.ai/docs/v1/flyte/architecture/control-plane/ === PAGE: https://www.union.ai/docs/v1/flyte/architecture/control-plane === # Control Plane > **📝 Note** > > An LLM-optimized bundle of this entire section is available at [`section.md`](section.md). > This single file contains all pages in this section, optimized for AI coding agent context. This section covers key architectural and implementation details for the Flyte control plane. === PAGE: https://www.union.ai/docs/v1/flyte/architecture/control-plane/projects === # Projects A project in Flyte is a group of [workflows](https://www.union.ai/docs/v1/flyte/architecture/workflows) and [tasks](https://www.union.ai/docs/v1/flyte/architecture/tasks) tied together to achieve a goal. A Flyte project can map to an engineering project or everything that's owned by a team or an individual. There cannot be multiple projects with the same name in Flyte. Since the fully-qualified name for tasks and workflows include the project and domain name, the task/workflow names are only required to be unique within a project. The workflows in a project A can refer to tasks and workflows in other projects using the fully-qualified name. Flyte allows users to set resource limits and provides basic reports and dashboards automatically for each project. The information captured in these reports includes workflow/task level insights, resource usage, and billing information. === PAGE: https://www.union.ai/docs/v1/flyte/architecture/control-plane/domains === # Domains Domains are fixed and unique at the global level, and provide an abstraction to isolate resources and feature configuration for different deployment environments. For example: We develop and deploy Flyte workflows in development, staging, and production. We configure Flyte domains with those names, and specify lower resource limits on the development and staging domains than production domains. We also use domains to disable launch plans and schedules from development and staging domains, since those features are typically meant for production deployments. === PAGE: https://www.union.ai/docs/v1/flyte/architecture/control-plane/admin === # Flyte Admin This is an architectural overview of the Flyte Admin control plane service. ## Admin Structure FlyteAdmin serves as the main Flyte API to process all client requests to the system. Clients include the FlyteConsole, which calls: FlyteAdmin to list the workflows, get execution details, etc. Flytekit, which in turn calls FlyteAdmin to register, launch workflows, etc. Below, we'll dive into each component defined in admin in more detail. ### RPC FlyteAdmin uses the [grpc-gateway](https://github.com/grpc-ecosystem/grpc-gateway) library to serve incoming gRPC and HTTP requests with identical handlers. Refer to the admin service [definition](https://docs.flyte.org/en/latest/api/flyteidl/docs/service/service.html#ref-flyteidl-service-admin-proto) for a detailed API overview, including request and response entities. The RPC handlers are thin shims that enforce request structure validation and call out to the appropriate [manager](#manager) methods to process requests. You can find a detailed explanation of the service in the [admin service](#flyteadmin-service-background) overview. ### Manager {#manager} The Admin API is broken up into entities: - Tasks - Workflows - Launch plans - Named Entities - Executions - Node Executions - Task Executions - Projects (and their respective domains) - Each API entity has an entity manager in FlyteAdmin responsible for implementing business logic for the entity. Entity managers handle full validation of creating, updating and getting requests and data persistence in the backing store (see the [repository](#repository) section). ### Additional Components The managers utilize additional components to process requests. These additional components include: - [workflow engine](#workflow-engine): compiles workflows and launches workflow executions from launch plans. - [data](#data) (remote cloud storage): offloads data blobs to the configured cloud provider. - [runtime](#runtime) loads values from a config file to assign task resources, initialization values, execution queues, and more. - [async processes](#async-processes) provides functions to schedule and execute the workflows as well as enqueue and trigger notifications. ## Repository {#repository} Serialized entities (tasks, workflows, launch plans) and executions (workflow-, node- and task-) are stored as protos defined [here](https://github.com/flyteorg/flyte/tree/master/flyteidl/protos/flyteidl/admin). We use the excellent [gorm](https://gorm.io/docs/index.html) library to interface with our database, which currently supports a Postgres implementation. You can find the actual code for issuing queries with gorm in the [gormimpl](https://github.com/flyteorg/flyte/tree/master/flyteadmin/pkg/repositories/gormimpl) directory. ### Models Database models are defined in the models directory and correspond 1:1 with the database tables [0]. The full set of database tables includes: - executions - execution tags - execution_events - launch_plans - named entities - node_executions - node_execution_events - tasks - task_executions - workflows These database models inherit primary keys and indexes as defined in the corresponding [models](https://github.com/flyteorg/flyte/tree/master/flyteadmin/pkg/repositories/models) file. The repositories code also includes [transformers](https://github.com/flyteorg/flyte/tree/master/flyteadmin/pkg/repositories/transformers). These convert entities from the database format to a response format for the external API. If you change either of these structures, you must change the corresponding transformers too. Given the unique naming constraints, some models are redefined in [migration_models](https://github.com/flyteorg/flyte/blob/master/flyteadmin/pkg/repositories/config/migration_models.go) to guarantee unique index values. ## Component Details This section dives into the details of each top-level directory defined in `pkg/`. ### Async Processes {#async-processes} Notifications and schedules are handled by async routines that are responsible for enqueuing and subsequently processing dequeued messages. FlyteAdmin uses the [gizmo](https://github.com/nytimes/gizmo) toolkit to abstract queueing implementation. Gizmo’s [pubsub](https://github.com/nytimes/gizmo#pubsub) library offers implementations for Amazon SNS/SQS, Google Pubsub, Kafka topics, and publishing over HTTP. For the sandbox development, no-op implementations of the notifications and schedule handlers are used to remove external cloud dependencies. ### Common As the name implies, common houses shared components used across different FlyteAdmin components in a single, top-level directory to avoid cyclic dependencies. These components include execution naming and phase utils, query filter definitions, query sorting definitions, and named constants. ### Data {#data} Data interfaces are primarily handled by the [storage](https://github.com/flyteorg/flyte/tree/master/flytestdlib/storage) library implemented in `flytestdlib`. However, neither this nor the underlying [stow](https://github.com/WasabiAiR/stow) library expose [HEAD](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods/HEAD) support. Hence, the data package in admin exists as the layer responsible for additional, remote data operations. ### Errors The errors directory contains centrally defined errors that are designed for compatibility with gRPC statuses. ### Runtime {#runtime} Values specific to the FlyteAdmin application, including task, workflow registration, and execution are configured in the [runtime](https://github.com/flyteorg/flyte/tree/master/flyteadmin/pkg/runtime) directory. These interfaces expose values configured in the `flyteadmin` top-level key in the application config. ### Workflow engine {#workflow-engine} This directory contains the interfaces to build and execute workflows leveraging FlytePropeller compiler and client components. ## FlyteAdmin Service Background {#flyteadmin-service-background} ### Entities The [admin service definition](https://docs.flyte.org/en/latest/api/flyteidl/docs/service/service.html#ref-flyteidl-service-admin-proto) defines REST operations for the entities that FlyteAdmin administers. #### Static Entities These include: - Workflows - Tasks - Launch Plans Permitted operations include: - Create - Get - List The above entities are designated by an [identifier](https://docs.flyte.org/en/latest/api/flyteidl/docs/core/core.html#ref-flyteidl-core-identifier) that consists of a project, domain, name, and version specification. These entities are, for the most part, immutable. To update one of these entities, the updated version must be re-registered with a unique and new version identifier attribute. One caveat is that the launch plan can toggle between `ACTIVE` and `INACTIVE` states. At a given point in time, only one launch plan version across a shared {Project, Domain, Name} specification can be active. The state affects the scheduled launch plans only. An inactive launch plan can be used to launch individual executions. However, only an active launch plan runs on a schedule (given it has a schedule defined). #### Static entities metadata (Named Entities) A [named entity](https://docs.flyte.org/en/latest/api/flyteidl/docs/admin/admin.html#ref-flyteidl-admin-namedentity) includes metadata for one of the above entities (workflow, task or launch plan) across versions. It also includes a resource type (workflow, task or launch plan) and an [id](https://docs.flyte.org/en/latest/api/flyteidl/docs/admin/admin.html#ref-flyteidl-admin-namedentityidentifier) which is composed of project, domain and name. The named entity also includes metadata, which are mutable attributes about the referenced entity. This metadata includes: - Description: a human-readable description for the Named Entity collection. - State (workflows only): this determines whether the workflow is shown on the overview list of workflows scoped by project and domain. Permitted operations include: - Create - Update - Get - List ### Execution entities These include: - (Workflow) executions - Node executions - Task executions Permitted operations include: - Create - Get - List After an execution begins, FlytePropeller monitors the execution and sends the events which the admin uses to update the above executions. These [events](https://docs.flyte.org/en/latest/api/flyteidl/docs/event/event.html#ref-flyteidl-event-event-proto) include - WorkflowExecutionEvent - NodeExecutionEvent - TaskExecutionEvent and contain information about respective phase transitions, phase transition time and optional output data if the event concerns a terminal phase change. These events provide the **only** way to update an execution. No raw update endpoint exists. To track the lifecycle of an execution, admin and store attributes such as duration and timestamp at which an execution transitioned to running and end time are used. For debugging purposes, admin also stores Workflow and Node execution events in its database, but does not currently expose them through an API. Because array tasks can yield many executions, admin does not store TaskExecutionEvents. ### Platform entities Projects: Like named entities, projects have mutable metadata such as human-readable names and descriptions, in addition to their unique string ids. Permitted project operations include: - Register - List ## Using the Admin Service ### Adding request filters We use [gRPC Gateway](https://github.com/grpc-ecosystem/grpc-gateway) to reverse proxy HTTP requests into gRPC. While this allows for a single implementation for both HTTP and gRPC, an important limitation is that fields mapped to the path pattern cannot be repeated and must have a primitive (non-message) type. Unfortunately this means that repeated string filters cannot use a proper protobuf message. Instead, they use the internal syntax shown below: ``` func(field,value) or func(field, value) ``` For example, multiple filters would be appended to an http request like: ``` ?filters=ne(version, TheWorst)+eq(workflow.name, workflow) ``` Timestamp fields use the `RFC3339Nano` spec (For example: “2006-01-02T15:04:05.999999999Z07:00”) The fully supported set of filter functions are contains - gt (greater than) - gte (greater than or equal to) - lt (less than) - lte (less than or equal to) - eq (equal) - ne (not equal) - value_in (value in repeated sets of values) - value_not_in (value not in repeated sets of values) “value_in” and “value_not_in” are special cases where multiple values are passed to the filter expression. For example: ```sql value_in(phase, RUNNING;SUCCEEDED;FAILED) ``` > [!NOTE] > If you’re issuing your requests over http(s), be sure to URL encode the “;” semicolon using %3B like so: `value_in(phase, RUNNING%3BSUCCEEDED%3BFAILED)` Filterable fields vary based on entity types: Task - project - domain - name - version - created_at Workflow - project - domain - name - version - created_at Launch plans - project - domain - name - version - created_at - updated_at - workflows.{any workflow field above} (for example: workflow.domain) - state (you must use the integer enum, for example: 1) - States are defined in [LaunchPlanState](https://docs.flyte.org/en/latest/api/flyteidl/docs/admin/admin.html#ref-flyteidl-admin-launchplanstate). Named Entity Metadata - state (you must use the integer enum, for example: 1) - States are defined in [NamedEntityState](https://docs.flyte.org/en/latest/api/flyteidl/docs/admin/admin.html#ref-flyteidl-admin-namedentitystate). Executions (Workflow executions) - project - domain - name - workflow.{any workflow field above} (for example: workflow.domain) - launch_plan.{any launch plan field above} (for example: launch_plan.name) - phase (you must use the upper-cased string name, for example: RUNNING) - Phases are defined in [WorkflowExecution.Phase](https://docs.flyte.org/en/latest/api/flyteidl/docs/core/core.html#ref-flyteidl-core-workflowexecution-phase). - execution_created_at - execution_updated_at - duration (in seconds) - mode (you must use the integer enum, for example: 1) - Modes are defined in [ExecutionMode](https://docs.flyte.org/en/latest/api/flyteidl/docs/admin/admin.html#ref-flyteidl-admin-executionmetadata-executionmode). - user (authenticated user or role from flytekit config) Node Executions - node_id - execution.{any execution field above} (for example: execution.domain) - phase (you must use the upper-cased string name, for example: QUEUED) - Phases are defined in [NodeExecution.Phase](https://docs.flyte.org/en/latest/api/flyteidl/docs/core/core.html#ref-flyteidl-core-nodeexecution-phase). - started_at - node_execution_created_at - node_execution_updated_at - duration (in seconds) Task Executions - retry_attempt - task.{any task field above} (for example: task.version) - execution.{any execution field above} (for example: execution.domain) - node_execution.{any node execution field above} (for example: node_execution.phase) - phase (you must use the upper-cased string name, for example: SUCCEEDED) - Phases are defined in [TaskExecution.Phase](https://docs.flyte.org/en/latest/api/flyteidl/docs/core/core.html#ref-flyteidl-core-taskexecution-phase). - started_at - task_execution_created_at - task_execution_updated_at - duration (in seconds) #### Putting It All Together If you wish to query specific executions that were launched using a specific launch plan for a workflow with specific attributes, use: ``` gte(duration, 100)+value_in(phase,RUNNING;SUCCEEDED;FAILED)+eq(lauch_plan.project, foo) +eq(launch_plan.domain, bar)+eq(launch_plan.name, baz) +eq(launch_plan.version, 1234) +lte(workflow.created_at,2018-11-29T17:34:05.000000000Z07:00) ``` ### Adding sorting to requests Only a subset of fields are supported for sorting list queries. The explicit list is shown below: ListTasks - project - domain - name - version - created_at ListTaskIds - project - domain ListWorkflows - project - domain - name - version - created_at ListWorkflowIds - project - domain ListLaunchPlans - project - domain - name - version - created_at - updated_at - state (you must use the integer enum, for example: 1) - States are defined in [LaunchPlanState](https://docs.flyte.org/en/latest/api/flyteidl/docs/admin/admin.html#ref-flyteidl-admin-launchplanstate). ListWorkflowIds - project - domain ListExecutions - project - domain - name - phase (you must use the upper-cased string name, for example: RUNNING) - Phases are defined in [WorkflowExecution.Phase](https://docs.flyte.org/en/latest/api/flyteidl/docs/core/core.html#ref-flyteidl-core-workflowexecution-phase). - execution_created_at - execution_updated_at - duration (in seconds) - mode (you must use the integer enum, for example: 1) - Modes are defined in [ExecutionMode](https://docs.flyte.org/en/latest/api/flyteidl/docs/admin/admin.html#ref-flyteidl-admin-executionmetadata-executionmode). ListNodeExecutions - node_id - retry_attempt - phase (you must use the upper-cased string name, for example: QUEUED) - Phases are defined in [NodeExecution.Phase](https://docs.flyte.org/en/latest/api/flyteidl/docs/core/core.html#ref-flyteidl-core-nodeexecution-phase) - started_at - node_execution_created_at - node_execution_updated_at - duration (in seconds) ListTaskExecutions - retry_attempt - phase (you must use the upper-cased string name, for example: SUCCEEDED) - Phases are defined in [TaskExecution.Phase](https://docs.flyte.org/en/latest/api/flyteidl/docs/core/core.html#ref-flyteidl-core-taskexecution-phase). - started_at - task_execution_created_at - task_execution_updated_at - duration (in seconds) ### Sorting syntax Adding sorting to a request requires specifying the `key`. For example: The attribute you wish to sort on. Sorting can also optionally specify the direction (one of `ASCENDIN`G or `DESCENDING`) where `DESCENDING` is the default. Example sorting HTTP parameter: ``` sort_by.key=created_at&sort_by.direction=DESCENDING ``` Alternatively, since `DESCENDING` is the default sorting direction, the above could be written as ``` sort_by.key=created_at ``` === PAGE: https://www.union.ai/docs/v1/flyte/architecture/control-plane/console === # FlyteConsole FlyteConsole is the web UI for the Flyte platform. Here’s a video that dives into the graph UX: 📺 [Watch on YouTube](https://www.youtube.com/watch?v=7YSc-QHk_Ec) ## Running FlyteConsole ### Install Dependencies Running FlyteConsole locally requires [NodeJS](https://nodejs.org/en) and [yarn](https://yarnpkg.com/). Once these are installed, all of the dependencies can be installed by running yarn in the project directory. ### Environment Variables Before we can run the server, we need to set up an environment variable or two. `ADMIN_API_URL` (default: window.location.origin) FlyteConsole displays information fetched from the FlyteAdmin API. This environment variable specifies the host prefix used in constructing API requests. > [!NOTE] > This is only the host portion of the API endpoint, consisting of the protocol, domain, and port (if not using the standard 80/443). > This value will be combined with a suffix (such as /api/v1) to construct the final URL used in an API request. This value will be combined with a suffix (such as `/api/v1`) to construct the final URL used in an API request. #### Default Behavior In most cases, `FlyteConsole` is hosted in the same cluster as the Admin API, meaning that the domain used to access the console is the same as that used to access the API. For this reason, if no value is set for `ADMIN_API_URL`, the default behavior is to use the value of *window.location.origin*. ``BASE_URL`` (default: ``undefined``) This allows running the console at a prefix on the target host. This is necessary when hosting the API and console on the same domain (with prefixes of `/api/v1` and `/console` for example). For local development, this is usually not needed, so the default behavior is to run without a prefix. ``CORS_PROXY_PREFIX`` (default: ``/cors_proxy``) Sets the local endpoint for [CORS request proxying](#cors-request-proxying). ### Run the Server To start the local development server, run `yarn start`. This will spin up a Webpack development server, compile all of the code into bundles, and start the NodeJS server on the default port (3000). All requests to the NodeJS server will be stalled until the bundles have finished. The application will be accessible at http://localhost:3000 (if using the default port). ## Development ### Storybook FlyteConsole uses [Storybook](https://storybook.js.org/). Component stories live next to the components they test in the `__stories__` directory with the filename pattern `{Component}.stories.tsx`. You can run storybook with npm run storybook, and view the stories at http://localhost:9001. ### Protobuf and the Network tab Communication with the FlyteAdmin API is done using Protobuf as the request/response format. Protobuf is a binary format, which means looking at responses in the Network tab won’t be helpful. To make debugging easier, each network request is logged to the console with its URL, followed by the decoded Protobuf payload. You must have debug output enabled (on by default in development) to see these messages. ### Debug Output This application makes use of the [debug](https://github.com/debug-js/debug) library to provide namespaced debug output in the browser console. In development, all debug output is enabled. For other environments, the debug output must be enabled manually. You can do this by setting a flag in localStorage using the console: `localStorage.debug = 'flyte:*'`. Each module in the application sets its own namespace. So if you’d like to only view output for a single module, you can specify that one specifically (ex. `localStorage.debug = 'flyte:adminEntity'` to only see decoded Flyte Admin API requests). ### CORS Proxying {#cors-request-proxying} In the common hosting arrangement, all API requests are made to the same `origin` serving the client application, making CORS unnecessary. For any requests which do not share the same origin value, the client application will route requests through a special endpoint on the NodeJS server. One example would be hosting the Admin API on a different domain than the console. Another example is fetching execution data from external storage such as S3. This is done to minimize the extra configuration required for ingress to the Admin API and data storage, as well as to simplify local development of the console without the need to grant CORS access to `localhost`. The requests and responses are piped through the NodeJS server with minimal overhead. However, it is still recommended to host the Admin API and console on the same domain to prevent unnecessary load on the NodeJS server and extra latency on API requests due to the additional hop. === PAGE: https://www.union.ai/docs/v1/flyte/architecture/control-plane/dynamic-job-spec === # Dynamic Job Spec A dynamic job spec is a subset of the entire workflow spec that defines a set of tasks, workflows, nodes, and output bindings that control how the job should assemble its outputs. This spec is currently only supported as an intermediate step in running Dynamic Tasks. ```protobuf message DynamicJobSpec { repeated Node nodes = 1; int64 min_successes = 2; repeated Binding outputs = 3; repeated TaskTemplate tasks = 4; repeated WorkflowTemplate subworkflows = 5; } ``` ## Tasks Defines one or more [tasks](https://www.union.ai/docs/v1/flyte/architecture/tasks) that can then be referenced in the spec. ## Subworkflows Defines zero or more [workflows](https://www.union.ai/docs/v1/flyte/architecture/workflows) that can then be referenced in the spec. ## Nodes Defines one or more nodes that can run in parallel to produce the final outputs of the spec. ## Outputs Defines one or more binding that instructs engine on how to assemble the final outputs.