1.16.10

flytekit.core.resources

Directory

Classes

Class Description
ResourceSpec
Resources This class is used to specify both resource requests and resource limits.

Methods

Method Description
construct_extended_resources() Convert public extended resources to idl.
convert_resources_to_resource_model() Convert flytekit Resources objects to a Resources model.
pod_spec_from_resources()

Variables

Property Type Description
SHARED_MEMORY_MOUNT_NAME str
SHARED_MEMORY_MOUNT_PATH str
TYPE_CHECKING bool

Methods

construct_extended_resources() 

def construct_extended_resources(
    accelerator: typing.Optional[flytekit.extras.accelerators.BaseAccelerator],
    shared_memory: typing.Union[typing.Literal[True], str, NoneType],
) -> typing.Optional[flyteidl.core.tasks_pb2.ExtendedResources]

Convert public extended resources to idl.

Parameter Type Description
accelerator typing.Optional[flytekit.extras.accelerators.BaseAccelerator] The accelerator to use for this task.
shared_memory typing.Union[typing.Literal[True], str, NoneType] If True, then shared memory will be attached to the container where the size is equal to the allocated memory. If str, then the shared memory is set to that size.

convert_resources_to_resource_model() 

def convert_resources_to_resource_model(
    requests: typing.Optional[flytekit.core.resources.Resources],
    limits: typing.Optional[flytekit.core.resources.Resources],
) -> flytekit.models.task.Resources

Convert flytekit Resources objects to a Resources model

Parameter Type Description
requests typing.Optional[flytekit.core.resources.Resources] Resource requests. Optional, defaults to None
limits typing.Optional[flytekit.core.resources.Resources] Resource limits. Optional, defaults to None :return: The given resources as requests and limits

pod_spec_from_resources() 

def pod_spec_from_resources(
    primary_container_name: typing.Optional[str],
    requests: typing.Optional[flytekit.core.resources.Resources],
    limits: typing.Optional[flytekit.core.resources.Resources],
    k8s_gpu_resource_key: str,
) -> V1PodSpec
Parameter Type Description
primary_container_name typing.Optional[str]
requests typing.Optional[flytekit.core.resources.Resources]
limits typing.Optional[flytekit.core.resources.Resources]
k8s_gpu_resource_key str

flytekit.core.resources.ResourceSpec 

class ResourceSpec(
    requests: flytekit.core.resources.Resources,
    limits: flytekit.core.resources.Resources,
)
Parameter Type Description
requests flytekit.core.resources.Resources
limits flytekit.core.resources.Resources

Methods

Method Description
from_dict()
from_json()
from_multiple_resource() Convert Resources that represent both a requests and limits into a ResourceSpec.
to_dict()
to_json()

from_dict() 

def from_dict(
    d,
    dialect,
)
Parameter Type Description
d
dialect

from_json() 

def from_json(
    data: typing.Union[str, bytes, bytearray],
    decoder: collections.abc.Callable[[typing.Union[str, bytes, bytearray]], dict[typing.Any, typing.Any]],
    from_dict_kwargs: typing.Any,
) -> ~T
Parameter Type Description
data typing.Union[str, bytes, bytearray]
decoder collections.abc.Callable[[typing.Union[str, bytes, bytearray]], dict[typing.Any, typing.Any]]
from_dict_kwargs typing.Any

from_multiple_resource() 

def from_multiple_resource(
    resource: flytekit.core.resources.Resources,
) -> ResourceSpec

Convert Resources that represent both a requests and limits into a ResourceSpec.

Parameter Type Description
resource flytekit.core.resources.Resources

to_dict() 

def to_dict()

to_json() 

def to_json(
    encoder: collections.abc.Callable[[typing.Any], typing.Union[str, bytes, bytearray]],
    to_dict_kwargs: typing.Any,
) -> typing.Union[str, bytes, bytearray]
Parameter Type Description
encoder collections.abc.Callable[[typing.Any], typing.Union[str, bytes, bytearray]]
to_dict_kwargs typing.Any

flytekit.core.resources.Resources

This class is used to specify both resource requests and resource limits.

Resources(cpu="1", mem="2048")  # This is 1 CPU and 2 KB of memory
Resources(cpu="100m", mem="2Gi")  # This is 1/10th of a CPU and 2 gigabytes of memory
Resources(cpu=0.5, mem=1024) # This is 500m CPU and 1 KB of memory

# For Kubernetes-based tasks, pods use ephemeral local storage for scratch space, caching, and for logs.
# This allocates 1Gi of such local storage.
Resources(ephemeral_storage="1Gi")

When used together with @task(resources=), you a specific the request and limits with one object. When the value is set to a tuple or list, the first value is the request and the second value is the limit. If the value is a single value, then both the requests and limit is set to that value. For example, the Resource(cpu=("1", "2"), mem=1024) will set the cpu request to 1, cpu limit to 2, mem limit and request to 1024.

Persistent storage is not currently supported on the Flyte backend.

Please see the :std:ref:User Guide <cookbook:customizing task resources> for detailed examples. Also refer to the K8s conventions.

class Resources(
    cpu: typing.Union[str, int, float, list, tuple, NoneType],
    mem: typing.Union[str, int, list, tuple, NoneType],
    gpu: typing.Union[str, int, list, tuple, NoneType],
    ephemeral_storage: typing.Union[str, int, NoneType],
)
Parameter Type Description
cpu typing.Union[str, int, float, list, tuple, NoneType]
mem typing.Union[str, int, list, tuple, NoneType]
gpu typing.Union[str, int, list, tuple, NoneType]
ephemeral_storage typing.Union[str, int, NoneType]

Methods

Method Description
from_dict()
from_json()
to_dict()
to_json()

from_dict() 

def from_dict(
    d,
    dialect,
)
Parameter Type Description
d
dialect

from_json() 

def from_json(
    data: typing.Union[str, bytes, bytearray],
    decoder: collections.abc.Callable[[typing.Union[str, bytes, bytearray]], dict[typing.Any, typing.Any]],
    from_dict_kwargs: typing.Any,
) -> ~T
Parameter Type Description
data typing.Union[str, bytes, bytearray]
decoder collections.abc.Callable[[typing.Union[str, bytes, bytearray]], dict[typing.Any, typing.Any]]
from_dict_kwargs typing.Any

to_dict() 

def to_dict()

to_json() 

def to_json(
    encoder: collections.abc.Callable[[typing.Any], typing.Union[str, bytes, bytearray]],
    to_dict_kwargs: typing.Any,
) -> typing.Union[str, bytes, bytearray]
Parameter Type Description
encoder collections.abc.Callable[[typing.Any], typing.Union[str, bytes, bytearray]]
to_dict_kwargs typing.Any