# Spark

## Directory

### Classes

| Class | Description |
|-|-|
| [`flytekitplugins.spark.connector.DatabricksConnector`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md#flytekitpluginssparkconnectordatabricksconnector) |  |
| [`flytekitplugins.spark.connector.DatabricksConnectorV2`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md#flytekitpluginssparkconnectordatabricksconnectorv2) | Add DatabricksConnectorV2 to support running the k8s spark and databricks spark together in the same workflow. |
| [`flytekitplugins.spark.connector.DatabricksJobMetadata`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md#flytekitpluginssparkconnectordatabricksjobmetadata) |  |
| [`flytekitplugins.spark.generic_task.GenericSparkConf`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.generic_task/page.md#flytekitpluginssparkgeneric_taskgenericsparkconf) |  |
| [`flytekitplugins.spark.generic_task.GenericSparkTask`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.generic_task/page.md#flytekitpluginssparkgeneric_taskgenericsparktask) |  |
| [`flytekitplugins.spark.models.SparkJob`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.models/page.md#flytekitpluginssparkmodelssparkjob) |  |
| [`flytekitplugins.spark.models.SparkType`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.models/page.md#flytekitpluginssparkmodelssparktype) |  |
| [`flytekitplugins.spark.pyspark_transformers.PySparkPipelineModelTransformer`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.pyspark_transformers/page.md#flytekitpluginssparkpyspark_transformerspysparkpipelinemodeltransformer) |  |
| [`flytekitplugins.spark.schema.ClassicSparkDataFrameSchemaReader`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.schema/page.md#flytekitpluginssparkschemaclassicsparkdataframeschemareader) | Implements how Classic SparkDataFrame should be read using the ``open`` method of FlyteSchema. |
| [`flytekitplugins.spark.schema.ClassicSparkDataFrameSchemaWriter`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.schema/page.md#flytekitpluginssparkschemaclassicsparkdataframeschemawriter) | Implements how Classic SparkDataFrame should be written using ``open`` method of FlyteSchema. |
| [`flytekitplugins.spark.schema.ClassicSparkDataFrameTransformer`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.schema/page.md#flytekitpluginssparkschemaclassicsparkdataframetransformer) | Transforms Classic Spark DataFrame's to and from a Schema (typed/untyped). |
| [`flytekitplugins.spark.schema.SparkDataFrameSchemaReader`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.schema/page.md#flytekitpluginssparkschemasparkdataframeschemareader) | Implements how SparkDataFrame should be read using the ``open`` method of FlyteSchema. |
| [`flytekitplugins.spark.schema.SparkDataFrameSchemaWriter`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.schema/page.md#flytekitpluginssparkschemasparkdataframeschemawriter) | Implements how SparkDataFrame should be written to using ``open`` method of FlyteSchema. |
| [`flytekitplugins.spark.schema.SparkDataFrameTransformer`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.schema/page.md#flytekitpluginssparkschemasparkdataframetransformer) | Transforms Spark DataFrame's to and from a Schema (typed/untyped). |
| [`flytekitplugins.spark.sd_transformers.ClassicSparkToParquetEncodingHandler`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.sd_transformers/page.md#flytekitpluginssparksd_transformersclassicsparktoparquetencodinghandler) |  |
| [`flytekitplugins.spark.sd_transformers.ParquetToClassicSparkDecodingHandler`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.sd_transformers/page.md#flytekitpluginssparksd_transformersparquettoclassicsparkdecodinghandler) |  |
| [`flytekitplugins.spark.sd_transformers.ParquetToSparkDecodingHandler`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.sd_transformers/page.md#flytekitpluginssparksd_transformersparquettosparkdecodinghandler) |  |
| [`flytekitplugins.spark.sd_transformers.SparkDataFrameRenderer`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.sd_transformers/page.md#flytekitpluginssparksd_transformerssparkdataframerenderer) | Render a Spark dataframe schema as an HTML table. |
| [`flytekitplugins.spark.sd_transformers.SparkToParquetEncodingHandler`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.sd_transformers/page.md#flytekitpluginssparksd_transformerssparktoparquetencodinghandler) |  |
| [`flytekitplugins.spark.task.Databricks`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.task/page.md#flytekitpluginssparktaskdatabricks) | Deprecated. |
| [`flytekitplugins.spark.task.DatabricksV2`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.task/page.md#flytekitpluginssparktaskdatabricksv2) | Use this to configure a Databricks task. |
| [`flytekitplugins.spark.task.PysparkFunctionTask`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.task/page.md#flytekitpluginssparktaskpysparkfunctiontask) | Actual Plugin that transforms the local python code for execution within a spark context. |
| [`flytekitplugins.spark.task.Spark`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.task/page.md#flytekitpluginssparktaskspark) | Use this to configure a SparkContext for a your task. |

### Functions

| Function | Description |
|-|-|
| [`flytekitplugins.spark.connector.get_databricks_token()`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md#get_databricks_token) | Get the Databricks access token with multi-tenant support. |
| [`flytekitplugins.spark.connector.get_header()`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md#get_header) | Get the authorization header for Databricks API calls. |
| [`flytekitplugins.spark.connector.get_secret_from_k8s()`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md#get_secret_from_k8s) | Read a secret from Kubernetes using the Kubernetes Python client. |
| [`flytekitplugins.spark.connector.result_state_is_available()`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md#result_state_is_available) |  |
| [`flytekitplugins.spark.task.new_spark_session()`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.task/page.md#new_spark_session) | Optionally creates a new spark session and returns it. |
| [`flytekitplugins.spark.utils.is_serverless_config()`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.utils/page.md#is_serverless_config) | Detect if the Databricks configuration is for serverless compute. |

### Packages

| Package | Description |
|-|-|
| [`flytekitplugins.spark.connector`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md) |  |
| [`flytekitplugins.spark.generic_task`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.generic_task/page.md) |  |
| [`flytekitplugins.spark.models`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.models/page.md) |  |
| [`flytekitplugins.spark.pyspark_transformers`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.pyspark_transformers/page.md) |  |
| [`flytekitplugins.spark.schema`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.schema/page.md) |  |
| [`flytekitplugins.spark.sd_transformers`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.sd_transformers/page.md) |  |
| [`flytekitplugins.spark.task`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.task/page.md) |  |
| [`flytekitplugins.spark.utils`](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.utils/page.md) |  |

## Subpages

- [flytekitplugins.spark.connector](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.connector/page.md)
  - Directory
  - Classes
  - Methods
  - Variables
  - Methods
  - flytekitplugins.spark.connector.DatabricksConnector
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.connector.DatabricksConnectorV2
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.connector.DatabricksJobMetadata
  - Parameters
  - Methods
- [flytekitplugins.spark.generic_task](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.generic_task/page.md)
  - Directory
  - Classes
  - flytekitplugins.spark.generic_task.GenericSparkConf
  - Parameters
  - flytekitplugins.spark.generic_task.GenericSparkTask
  - Parameters
  - Properties
  - Methods
- [flytekitplugins.spark.models](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.models/page.md)
  - Directory
  - Classes
  - flytekitplugins.spark.models.SparkJob
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.models.SparkType
- [flytekitplugins.spark.pyspark_transformers](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.pyspark_transformers/page.md)
  - Directory
  - Classes
  - flytekitplugins.spark.pyspark_transformers.PySparkPipelineModelTransformer
  - Parameters
  - Properties
  - Methods
- [flytekitplugins.spark.schema](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.schema/page.md)
  - Directory
  - Classes
  - flytekitplugins.spark.schema.ClassicSparkDataFrameSchemaReader
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.schema.ClassicSparkDataFrameSchemaWriter
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.schema.ClassicSparkDataFrameTransformer
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.schema.SparkDataFrameSchemaReader
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.schema.SparkDataFrameSchemaWriter
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.schema.SparkDataFrameTransformer
  - Parameters
  - Properties
  - Methods
- [flytekitplugins.spark.sd_transformers](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.sd_transformers/page.md)
  - Directory
  - Classes
  - Variables
  - flytekitplugins.spark.sd_transformers.ClassicSparkToParquetEncodingHandler
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.sd_transformers.ParquetToClassicSparkDecodingHandler
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.sd_transformers.ParquetToSparkDecodingHandler
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.sd_transformers.SparkDataFrameRenderer
  - Methods
  - flytekitplugins.spark.sd_transformers.SparkToParquetEncodingHandler
  - Parameters
  - Properties
  - Methods
- [flytekitplugins.spark.task](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.task/page.md)
  - Directory
  - Classes
  - Methods
  - Variables
  - Methods
  - flytekitplugins.spark.task.Databricks
  - Parameters
  - flytekitplugins.spark.task.DatabricksV2
  - Parameters
  - flytekitplugins.spark.task.PysparkFunctionTask
  - Parameters
  - Properties
  - Methods
  - flytekitplugins.spark.task.Spark
  - Parameters
- [flytekitplugins.spark.utils](https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/flytekitplugins.spark.utils/page.md)
  - Directory
  - Methods
  - Methods

---
**Source**: https://github.com/unionai/unionai-docs/blob/main/content/api-reference/plugins/spark/_index.md
**HTML**: https://www.union.ai/docs/v1/flyte/api-reference/plugins/spark/