flyte.connectors

Directory

Classes

flyte.connectors.AsyncConnector

This is the base class for all async connectors, and it defines the interface that all connectors must implement. The connector service is responsible for invoking connectors. The executor will communicate with the connector service to create tasks, get the status of tasks, and delete tasks.

All the connectors should be registered in the ConnectorRegistry. Connector Service will look up the connector based on the task type and version.

Methods

create()

def create(
    task_template: flyteidl2.core.tasks_pb2.TaskTemplate,
    output_prefix: str,
    inputs: typing.Optional[typing.Dict[str, typing.Any]],
    task_execution_metadata: typing.Optional[flyteidl2.plugins.connector_pb2.TaskExecutionMetadata],
    kwargs,
) -> flyte.connectors._connector.ResourceMeta

Return a resource meta that can be used to get the status of the task.

delete()

def delete(
    resource_meta: flyte.connectors._connector.ResourceMeta,
    kwargs,
)

Delete the task. This call should be idempotent. It should raise an error if fails to delete the task.

get()

def get(
    resource_meta: flyte.connectors._connector.ResourceMeta,
    kwargs,
) -> flyte.connectors._connector.Resource

Return the status of the task, and return the outputs in some cases. For example, bigquery job can’t write the structured dataset to the output location, so it returns the output literals to the propeller, and the propeller will write the structured dataset to the blob store.

get_logs()

def get_logs(
    resource_meta: flyte.connectors._connector.ResourceMeta,
    kwargs,
) -> flyteidl2.plugins.connector_pb2.GetTaskLogsResponse

Return the metrics for the task.

get_metrics()

def get_metrics(
    resource_meta: flyte.connectors._connector.ResourceMeta,
    kwargs,
) -> flyteidl2.plugins.connector_pb2.GetTaskMetricsResponse

Return the metrics for the task.

flyte.connectors.ConnectorRegistry

This is the registry for all connectors. The connector service will look up the connector registry based on the task type and version.

Methods

get_connector()

def get_connector(
    task_type_name: str,
    task_type_version: int,
) -> flyte.connectors._connector.AsyncConnector

get_connector_metadata()

def get_connector_metadata(
    name: str,
) -> flyteidl2.plugins.connector_pb2.Connector

list_connectors()

def list_connectors()

register()

def register(
    connector: flyte.connectors._connector.AsyncConnector,
    override: bool,
)

flyte.connectors.Resource

This is the output resource of the job.

Attributes

phase : TaskExecution.Phase
    The phase of the job.
message : Optional[str]
    The return message from the job.
log_links : Optional[List[TaskLog]]
    The log links of the job. For example, the link to the BigQuery Console.
outputs : Optional[Union[LiteralMap, typing.Dict[str, Any]]]
    The outputs of the job. If return python native types, the agent will convert them to flyte literals.
custom_info : Optional[typing.Dict[str, Any]]
    The custom info of the job. For example, the job config.

class Resource(
    phase: <google.protobuf.internal.enum_type_wrapper.EnumTypeWrapper object at 0x10ca18950>,
    message: typing.Optional[str],
    log_links: typing.Optional[typing.List[flyteidl2.core.execution_pb2.TaskLog]],
    outputs: typing.Optional[typing.Dict[str, typing.Any]],
    custom_info: typing.Optional[typing.Dict[str, typing.Any]],
)

flyte.connectors.ResourceMeta

This is the metadata for the job. For example, the id of the job.

def ResourceMeta()

Methods

decode()

def decode(
    data: bytes,
) -> ResourceMeta

Decode the resource meta from bytes.

encode()

def encode()

Encode the resource meta to bytes.