flytekit.tools.script_mode

Directory

Methods

Variables

Methods

add_imported_modules_from_source()

def add_imported_modules_from_source(
    source_path: str,
    destination: str,
    modules: List[ModuleType],
)

Copies modules into destination that are in modules. The module files are copied only if:

1. Not a site-packages. These are installed packages and not user files.
2. Not in the sys.base_prefix or sys.prefix. These are also installed and not user files.
3. Does not share a common path with the source_path.

compress_scripts()

def compress_scripts(
    source_path: str,
    destination: str,
    modules: List[ModuleType],
)

Compresses the single script while maintaining the folder structure for that file.

For example, given the follow file structure: . ├── flyte │   ├── init.py │   └── workflows │   ├── example.py │   ├── another_example.py │   ├── yet_another_example.py │   ├── unused_example.py │   └── init.py

Let’s say you want to compress example.py imports another_example.py. And another_example.py imports on yet_another_example.py. This will produce a tar file that contains only that file alongside with the folder structure, i.e.:

. ├── flyte │   ├── init.py │   └── workflows │   ├── example.py │   ├── another_example.py │   ├── yet_another_example.py │   └── init.py

get_all_modules()

def get_all_modules(
    source_path: str,
    module_name: Optional[str],
) -> List[ModuleType]

Import python file with module_name in source_path and return all modules.

list_all_files()

def list_all_files(
    source_path: str,
    deref_symlinks,
    ignore_group: Optional[IgnoreGroup],
) -> List[str]

list_imported_modules_as_files()

def list_imported_modules_as_files(
    source_path: str,
    modules: List[ModuleType],
) -> List[str]

Copies modules into destination that are in modules. The module files are copied only if:

1. Not a site-packages. These are installed packages and not user files.
2. Not in the sys.base_prefix or sys.prefix. These are also installed and not user files.
3. Does not share a common path with the source_path.

ls_files()

def ls_files(
    source_path: str,
    copy_file_detection: CopyFileDetection,
    deref_symlinks: bool,
    ignore_group: Optional[IgnoreGroup],
) -> Tuple[List[str], str]

user_modules_and_packages is a list of the Python modules and packages, expressed as absolute paths, that the user has run this pyflyte command with. For pyflyte run for instance, this is just a list of one. This is used for two reasons.

• Everything in this list needs to be returned. Files are returned and folders are walked.
• A common source path is derived from this is, which is just the common folder that contains everything in the list. For ex. if you do $ pyflyte –pkgs a.b,a.c package Then the common root is just the folder a/. The modules list is filtered against this root. Only files representing modules under this root are included

If the copy enum is set to loaded_modules, then the loaded sys modules will be used.

tar_strip_file_attributes()

def tar_strip_file_attributes(
    tar_info: tarfile.TarInfo,
) -> tarfile.TarInfo