flyte.types

Flyte Type System

The Flyte type system provides a way to define, transform, and manipulate types in Flyte workflows. Since the data flowing through Flyte has to often cross process, container and langauge boundaries, the type system is designed to be serializable to a universal format that can be understood across different environments. This universal format is based on Protocol Buffers. The types are called LiteralTypes and the runtime representation of data is called Literals.

The type system includes:

• TypeEngine: The core engine that manages type transformations and serialization. This is the main entry point for for all the internal type transformations and serialization logic.
• TypeTransformer: A class that defines how to transform one type to another. This is extensible allowing users to define custom types and transformations.
• Renderable: An interface for types that can be rendered as HTML, that can be outputted to a flyte.report.

It is always possible to bypass the type system and use the FlytePickle type to serialize any python object into a pickle format. The pickle format is not human-readable, but can be passed between flyte tasks that are written in python. The Pickled objects cannot be represented in the UI, and may be in-efficient for large datasets.

Directory

Classes

Protocols

Errors

Methods

Methods

guess_interface()

def guess_interface(
    interface: flyteidl.core.interface_pb2.TypedInterface,
    default_inputs: typing.Optional[typing.Iterable[workflow.common_pb2.NamedParameter]],
) -> flyte.models.NativeInterface

Returns the interface of the task with guessed types, as types may not be present in current env.

literal_string_repr()

def literal_string_repr(
    lm: typing.Union[flyteidl.core.literals_pb2.Literal, workflow.run_definition_pb2.NamedLiteral, workflow.run_definition_pb2.Inputs, workflow.run_definition_pb2.Outputs, flyteidl.core.literals_pb2.LiteralMap, typing.Dict[str, flyteidl.core.literals_pb2.Literal]],
) -> typing.Dict[str, typing.Any]

This method is used to convert a literal map to a string representation.

flyte.types.FlytePickle

This type is only used by flytekit internally. User should not use this type. Any type that flyte can’t recognize will become FlytePickle

Methods

from_pickle()

def from_pickle(
    uri: str,
) -> typing.Any

python_type()

def python_type()

to_pickle()

def to_pickle(
    python_val: typing.Any,
) -> str

flyte.types.Renderable

Base class for protocol classes.

Protocol classes are defined as::

class Proto(Protocol):
    def meth(self) -> int:
        ...

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).

For example::

class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check

See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as::

class GenProto[T](Protocol):
    def meth(self) -> T:
        ...

protocol Renderable()

Methods

to_html()

def to_html(
    python_value: typing.Any,
) -> str

Convert an object(markdown, pandas.dataframe) to HTML and return HTML as a unicode string. Returns: An HTML document as a string.

flyte.types.TypeEngine

Core Extensible TypeEngine of Flytekit. This should be used to extend the capabilities of FlyteKits type system. Users can implement their own TypeTransformers and register them with the TypeEngine. This will allow special handling of user objects

Methods

calculate_hash()

def calculate_hash(
    python_val: typing.Any,
    python_type: Type[T],
) -> Optional[str]

dict_to_literal_map()

def dict_to_literal_map(
    d: typing.Dict[str, typing.Any],
    type_hints: Optional[typing.Dict[str, type]],
) -> LiteralMap

Given a dictionary mapping string keys to python values and a dictionary containing guessed types for such string keys, convert to a LiteralMap.

get_available_transformers()

def get_available_transformers()

Returns all python types for which transformers are available

get_transformer()

def get_transformer(
    python_type: Type,
) -> TypeTransformer

Implements a recursive search for the transformer.

guess_python_type()

def guess_python_type(
    flyte_type: LiteralType,
) -> Type[T]

Transforms a flyte-specific LiteralType to a regular python value.

guess_python_types()

def guess_python_types(
    flyte_variable_dict: typing.Dict[str, interface_pb2.Variable],
) -> typing.Dict[str, Type[Any]]

Transforms a dictionary of flyte-specific Variable objects to a dictionary of regular python values.

lazy_import_transformers()

def lazy_import_transformers()

Only load the transformers if needed.

literal_map_to_kwargs()

def literal_map_to_kwargs(
    lm: LiteralMap,
    python_types: typing.Optional[typing.Dict[str, type]],
    literal_types: typing.Optional[typing.Dict[str, interface_pb2.Variable]],
) -> typing.Dict[str, typing.Any]

Given a LiteralMap (usually an input into a task - intermediate), convert to kwargs for the task

named_tuple_to_variable_map()

def named_tuple_to_variable_map(
    t: typing.NamedTuple,
) -> interface_pb2.VariableMap

Converts a python-native NamedTuple to a flyte-specific VariableMap of named literals.

register()

def register(
    transformer: TypeTransformer,
    additional_types: Optional[typing.List[Type]],
)

This should be used for all types that respond with the right type annotation when you use type(…) function

register_additional_type()

def register_additional_type(
    transformer: TypeTransformer[T],
    additional_type: Type[T],
    override,
)

register_restricted_type()

def register_restricted_type(
    name: str,
    type: Type[T],
)

to_html()

def to_html(
    python_val: typing.Any,
    expected_python_type: Type[typing.Any],
) -> str

to_literal()

def to_literal(
    python_val: typing.Any,
    python_type: Type[T],
    expected: types_pb2.LiteralType,
) -> literals_pb2.Literal

to_literal_checks()

def to_literal_checks(
    python_val: typing.Any,
    python_type: Type[T],
    expected: LiteralType,
)

to_literal_type()

def to_literal_type(
    python_type: Type[T],
) -> LiteralType

Converts a python type into a flyte specific LiteralType

to_python_value()

def to_python_value(
    lv: Literal,
    expected_python_type: Type,
) -> typing.Any

Converts a Literal value with an expected python type into a python value.

unwrap_offloaded_literal()

def unwrap_offloaded_literal(
    lv: literals_pb2.Literal,
) -> literals_pb2.Literal

flyte.types.TypeTransformer

Base transformer type that should be implemented for every python native type that can be handled by flytekit

class TypeTransformer(
    name: str,
    t: Type[T],
    enable_type_assertions: bool,
)

Methods

assert_type()

def assert_type(
    t: Type[T],
    v: T,
)

from_binary_idl()

def from_binary_idl(
    binary_idl_object: Binary,
    expected_python_type: Type[T],
) -> Optional[T]

This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access.

For untyped dict, dataclass, and pydantic basemodel: Life Cycle (Untyped Dict as example): python val -> msgpack bytes -> binary literal scalar -> msgpack bytes -> python val (to_literal) (from_binary_idl)

For attribute access: Life Cycle: python val -> msgpack bytes -> binary literal scalar -> resolved golang value -> binary literal scalar -> msgpack bytes -> python val (to_literal) (propeller attribute access) (from_binary_idl)

get_literal_type()

def get_literal_type(
    t: Type[T],
) -> LiteralType

Converts the python type to a Flyte LiteralType

guess_python_type()

def guess_python_type(
    literal_type: LiteralType,
) -> Type[T]

Converts the Flyte LiteralType to a python object type.

isinstance_generic()

def isinstance_generic(
    obj,
    generic_alias,
)

to_html()

def to_html(
    python_val: T,
    expected_python_type: Type[T],
) -> str

Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div

to_literal()

def to_literal(
    python_val: T,
    python_type: Type[T],
    expected: LiteralType,
) -> Literal

Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch

to_python_value()

def to_python_value(
    lv: Literal,
    expected_python_type: Type[T],
) -> Optional[T]

Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised

Properties

flyte.types.TypeTransformerFailedError

Inappropriate argument type.