Databricks connector

Union.ai can be integrated with the Databricks service, enabling you to submit Spark jobs to the Databricks platform.

Installation

The Databricks connector comes bundled with the Spark plugin. To install the Spark plugin, run the following command:

$ pip install flytekitplugins-spark

Example usage

For an example query, see Databricks connector example usage

Local testing

To test the Databricks connector locally, create a class for the connector task that inherits from AsyncConnectorExecutorMixin. This mixin can handle asynchronous tasks and allows the SDK to mimic the system’s behavior in calling the connector.

For more information, see Testing connectors locally.

In some cases, you will need to store credentials in your local environment when testing locally.