Spark

Directory

Classes

Class	Description
`flytekitplugins.spark.connector.DatabricksConnector`
`flytekitplugins.spark.connector.DatabricksConnectorV2`	Add DatabricksConnectorV2 to support running the k8s spark and databricks spark together in the same workflow.
`flytekitplugins.spark.connector.DatabricksJobMetadata`
`flytekitplugins.spark.generic_task.GenericSparkConf`
`flytekitplugins.spark.generic_task.GenericSparkTask`
`flytekitplugins.spark.models.SparkJob`
`flytekitplugins.spark.models.SparkType`
`flytekitplugins.spark.pyspark_transformers.PySparkPipelineModelTransformer`
`flytekitplugins.spark.schema.ClassicSparkDataFrameSchemaReader`	Implements how Classic SparkDataFrame should be read using the `open` method of FlyteSchema.
`flytekitplugins.spark.schema.ClassicSparkDataFrameSchemaWriter`	Implements how Classic SparkDataFrame should be written using `open` method of FlyteSchema.
`flytekitplugins.spark.schema.ClassicSparkDataFrameTransformer`	Transforms Classic Spark DataFrame’s to and from a Schema (typed/untyped).
`flytekitplugins.spark.schema.SparkDataFrameSchemaReader`	Implements how SparkDataFrame should be read using the `open` method of FlyteSchema.
`flytekitplugins.spark.schema.SparkDataFrameSchemaWriter`	Implements how SparkDataFrame should be written to using `open` method of FlyteSchema.
`flytekitplugins.spark.schema.SparkDataFrameTransformer`	Transforms Spark DataFrame’s to and from a Schema (typed/untyped).
`flytekitplugins.spark.sd_transformers.ClassicSparkToParquetEncodingHandler`
`flytekitplugins.spark.sd_transformers.ParquetToClassicSparkDecodingHandler`
`flytekitplugins.spark.sd_transformers.ParquetToSparkDecodingHandler`
`flytekitplugins.spark.sd_transformers.SparkDataFrameRenderer`	Render a Spark dataframe schema as an HTML table.
`flytekitplugins.spark.sd_transformers.SparkToParquetEncodingHandler`
`flytekitplugins.spark.task.Databricks`	Deprecated.
`flytekitplugins.spark.task.DatabricksV2`	Use this to configure a Databricks task.
`flytekitplugins.spark.task.PysparkFunctionTask`	Actual Plugin that transforms the local python code for execution within a spark context.
`flytekitplugins.spark.task.Spark`	Use this to configure a SparkContext for a your task.

Functions

Function	Description
`flytekitplugins.spark.connector.get_databricks_token()`	Get the Databricks access token with multi-tenant support.
`flytekitplugins.spark.connector.get_header()`	Get the authorization header for Databricks API calls.
`flytekitplugins.spark.connector.get_secret_from_k8s()`	Read a secret from Kubernetes using the Kubernetes Python client.
`flytekitplugins.spark.connector.result_state_is_available()`
`flytekitplugins.spark.task.new_spark_session()`	Optionally creates a new spark session and returns it.
`flytekitplugins.spark.utils.is_serverless_config()`	Detect if the Databricks configuration is for serverless compute.

Packages

Package	Description
`flytekitplugins.spark.connector`
`flytekitplugins.spark.generic_task`
`flytekitplugins.spark.models`
`flytekitplugins.spark.pyspark_transformers`
`flytekitplugins.spark.schema`
`flytekitplugins.spark.sd_transformers`
`flytekitplugins.spark.task`
`flytekitplugins.spark.utils`