Flyte 2 is available today for local execution - distributed execution coming to open source soon. Preview Flyte 2 for production, hosted on Union.ai
2.1.7

Spark

Package: flyteplugins.spark

Use this to configure a SparkContext for a your task. Task’s marked with this will automatically execute natively onto K8s as a distributed execution of spark

Parameters 

class Spark(
    spark_conf: typing.Optional[typing.Dict[str, str]],
    hadoop_conf: typing.Optional[typing.Dict[str, str]],
    executor_path: typing.Optional[str],
    applications_path: typing.Optional[str],
    driver_pod: typing.Optional[flyte._pod.PodTemplate],
    executor_pod: typing.Optional[flyte._pod.PodTemplate],
)
Parameter Type Description
spark_conf typing.Optional[typing.Dict[str, str]] Spark configuration dictionary.
hadoop_conf typing.Optional[typing.Dict[str, str]] Hadoop configuration dictionary.
executor_path typing.Optional[str] Path to the Python binary for PySpark execution.
applications_path typing.Optional[str] Path to the main application file.
driver_pod typing.Optional[flyte._pod.PodTemplate] Pod template for the driver pod.
executor_pod typing.Optional[flyte._pod.PodTemplate] Pod template for the executor pods.