Spark

Directory

Classes

Class	Description
`flytekitplugins.spark.connector.DatabricksConnector`
`flytekitplugins.spark.connector.DatabricksConnectorV2`	Add DatabricksConnectorV2 to support running the k8s spark and databricks spark together in the same workflow.
`flytekitplugins.spark.connector.DatabricksJobMetadata`	Metadata persisted for a Databricks run.
`flytekitplugins.spark.databricks_auth.DatabricksAuth`	Interface for obtaining a bearer token for Databricks API calls.
`flytekitplugins.spark.databricks_auth.DatabricksAuthError`	Raised when Databricks authentication cannot be obtained.
`flytekitplugins.spark.databricks_auth.OAuthM2MAuth`	Authenticate a Databricks service principal with client credentials.
`flytekitplugins.spark.databricks_auth.OIDCConnectorAuth`	Exchange the connector workload’s projected JWT for a Databricks token.
`flytekitplugins.spark.databricks_auth.OIDCNamespaceServiceAccountAuth`	Federate as a ServiceAccount discovered in the workflow namespace.
`flytekitplugins.spark.databricks_auth.PATAuth`	Delegate to the connector’s existing multi-tenant PAT lookup.
`flytekitplugins.spark.generic_task.GenericSparkConf`
`flytekitplugins.spark.generic_task.GenericSparkTask`
`flytekitplugins.spark.models.SparkJob`
`flytekitplugins.spark.models.SparkType`
`flytekitplugins.spark.pyspark_transformers.PySparkPipelineModelTransformer`
`flytekitplugins.spark.schema.ClassicSparkDataFrameSchemaReader`	Implements how Classic SparkDataFrame should be read using the `open` method of FlyteSchema.
`flytekitplugins.spark.schema.ClassicSparkDataFrameSchemaWriter`	Implements how Classic SparkDataFrame should be written using `open` method of FlyteSchema.
`flytekitplugins.spark.schema.ClassicSparkDataFrameTransformer`	Transforms Classic Spark DataFrame’s to and from a Schema (typed/untyped).
`flytekitplugins.spark.schema.SparkDataFrameSchemaReader`	Implements how SparkDataFrame should be read using the `open` method of FlyteSchema.
`flytekitplugins.spark.schema.SparkDataFrameSchemaWriter`	Implements how SparkDataFrame should be written to using `open` method of FlyteSchema.
`flytekitplugins.spark.schema.SparkDataFrameTransformer`	Transforms Spark DataFrame’s to and from a Schema (typed/untyped).
`flytekitplugins.spark.sd_transformers.ClassicSparkToParquetEncodingHandler`
`flytekitplugins.spark.sd_transformers.ParquetToClassicSparkDecodingHandler`
`flytekitplugins.spark.sd_transformers.ParquetToSparkDecodingHandler`
`flytekitplugins.spark.sd_transformers.SparkDataFrameRenderer`	Render a Spark dataframe schema as an HTML table.
`flytekitplugins.spark.sd_transformers.SparkToParquetEncodingHandler`
`flytekitplugins.spark.task.Databricks`	Deprecated.
`flytekitplugins.spark.task.DatabricksV2`	Use this to configure a Databricks task.
`flytekitplugins.spark.task.PysparkFunctionTask`	Actual Plugin that transforms the local python code for execution within a spark context.
`flytekitplugins.spark.task.Spark`	Use this to configure a SparkContext for a your task.

Functions

Function	Description
`flytekitplugins.spark.connector.get_databricks_token()`	Get the Databricks access token with multi-tenant support.
`flytekitplugins.spark.connector.get_header()`	Get the authorization header for Databricks API calls.
`flytekitplugins.spark.connector.get_secret_from_k8s()`	Read a secret from Kubernetes using the Kubernetes Python client.
`flytekitplugins.spark.connector.list_serviceaccounts_in_k8s()`	List labelled ServiceAccounts in a workflow namespace.
`flytekitplugins.spark.connector.result_state_is_available()`
`flytekitplugins.spark.databricks_auth.build_auth()`	Rebuild an auth strategy from persisted connector metadata.
`flytekitplugins.spark.databricks_auth.select_auth()`	Select an explicitly configured strategy, defaulting to PAT.
`flytekitplugins.spark.databricks_auth.validate_connector_config()`	Validate an explicitly configured connector-wide auth type.
`flytekitplugins.spark.task.new_spark_session()`	Optionally creates a new spark session and returns it.
`flytekitplugins.spark.utils.is_serverless_config()`	Detect if the Databricks configuration is for serverless compute.

Packages

Package	Description
`flytekitplugins.spark.connector`
`flytekitplugins.spark.databricks_auth`	Authentication strategies for the Databricks connector.
`flytekitplugins.spark.generic_task`
`flytekitplugins.spark.models`
`flytekitplugins.spark.pyspark_transformers`
`flytekitplugins.spark.schema`
`flytekitplugins.spark.sd_transformers`
`flytekitplugins.spark.task`
`flytekitplugins.spark.utils`