Databricks Connect
Note
This article covers Databricks Connect for Databricks Runtime 13.0 and higher.
For prior Databricks Runtime versions, see Databricks Connect for Databricks Runtime 12.2 LTS and lower.
Databricks Connect allows you to connect popular IDEs such as Visual Studio Code and PyCharm, notebook servers, and other custom applications to Azure Databricks clusters.
This article explains how Databricks Connect works, walks you through the steps to get started with Databricks Connect, and explains how to troubleshoot issues that may arise when using Databricks Connect.
Overview
Databricks Connect is a client library for the Databricks Runtime. It allows you to write jobs using Spark APIs and run them remotely on an Azure Databricks cluster instead of in the local Spark session.
For example, when you run the DataFrame command spark.read.format(...).load(...).groupBy(...).agg(...).show() using Databricks Connect, the logical representation of the command is sent to the Spark server running in Azure Databricks for execution on the remote cluster.
With Databricks Connect, you can:
Run large-scale Spark jobs from any Python application. Anywhere you can
import pyspark, you can now run Spark jobs directly from your application, without needing to install any IDE plugins or use Spark submission scripts.Note
Databricks Connect for Databricks Runtime 13.0 and higher currently supports running only Python applications.
Step through and debug code in your IDE even when working with a remote cluster.
Iterate quickly when developing libraries. You do not need to restart the cluster after changing Python library dependencies in Databricks Connect, because each client session is isolated from each other in the cluster.
Shut down idle clusters without losing work. Because the client application is decoupled from the cluster, it is unaffected by cluster restarts or upgrades, which would normally cause you to lose all the variables, RDDs, and DataFrame objects defined in a notebook.
For Databricks Runtime 13.0 and higher, Databricks Connect is now built on open-source Spark Connect. Spark Connect introduces a decoupled client-server architecture for Apache Spark that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. With this “V2” architecture based on Spark Connect, Databricks Connect becomes a thin client that is simple and easy to use. Spark Connect can be embedded everywhere to connect to Azure Databricks: in IDEs, notebooks, and applications, allowing individual users and partners alike to build new (interactive) user experiences based on the Databricks Lakehouse. For more information about Spark Connect, see Introducing Spark Connect.
Requirements
This section lists the requirements for Databricks Connect.
An Azure Databricks workspace and its corresponding account that are enabled for Unity Catalog. See Get started using Unity Catalog and Enable a workspace for Unity Catalog.
A cluster with Databricks Runtime 13.0 or higher installed.
Only clusters that are compatible with Unity Catalog are supported. These include clusters with Single User or Shared cluster access modes. See What is cluster access mode?.
You must install Python 3 on your development machine, and the minor version of your client Python installation must be the same as the minor Python version of your Azure Databricks cluster. The following table shows the Python version installed with each Databricks Runtime.
Databricks Runtime version Python version 13.1 ML, 13.1 3.10 13.0 ML, 13.0 3.10 Note
If you want to use PySpark UDFs, it’s important that your development machine’s installed minor version of Python match the minor version of Python that is included with Databricks Runtime installed on the cluster.
Databricks strongly recommends that you have a Python virtual environment activated for each Python version that you use with Databricks Connect. Python virtual environments help to make sure that you are using the correct versions of Python and Databricks Connect together. This can help to reduce or shorten resolving related technical issues.
For example, if you’re using venv on your development machine and your cluster is running Python 3.10, you must create a
venvenvironment with that version. The following example command generates the scripts to activate avenvenvironment with Python 3.10, and this command then places those scripts within a hidden folder named.venvwithin the current working directory:# Linux and macOS python3.10 -m venv ./.venv # Windows python3.10 -m venv .\.venvTo use these scripts to activate this
venvenvironment, see How venvs work.The Databricks Connect major and minor package version should match your Databricks Runtime version. Databricks recommends that you always use the most recent package of Databricks Connect that matches your Databricks Runtime version. For example, when you use a Databricks Runtime 13.1 cluster, you should also use the
databricks-connect==13.1.*package.Note
See the Databricks Connect release notes for a list of available Databricks Connect releases and maintenance updates.
Using the most recent package of Databricks Connect that matches your Databricks Runtime version is not a requirement. Beginning with Databricks Runtime 13.0, you can use the Databricks Connect package against all versions of Databricks Runtime at or above the version of the Databricks Connect package. However, if you want to use features that are available in later versions of the Databricks Runtime, you must upgrade the Databricks Connect package accordingly.
Set up the client
Complete the following steps to set up the local client for Databricks Connect.
Note
Before you begin to set up the local Databricks Connect client, you must meet the requirements for Databricks Connect.
Tip
If you already have the Databricks extension for Visual Studio Code installed, you do not need to follow these setup instructions.
The Databricks extension for Visual Studio Code already has built-in support for Databricks Connect for Databricks Runtime 13.0 and higher. Skip ahead to Run or debug Python code with Databricks Connect in the documentation for the Databricks extension for Visual Studio Code.
Step 1: Install the Databricks Connect client
With your virtual environment activated, uninstall PySpark, if it is already installed, by running the
uninstallcommand. This is required because thedatabricks-connectpackage conflicts with PySpark. For details, see Conflicting PySpark installations. To check whether PySpark is already installed, run theshowcommand.# Is PySpark already installed? pip3 show pyspark # Uninstall PySpark pip3 uninstall pysparkWith your virtual environment still activated, install the Databricks Connect client by running the
installcommand. Use the--upgradeoption to upgrade any existing client installation to the specified version.pip3 install --upgrade "databricks-connect==13.1.*" # Or X.Y.* to match your cluster version.Note
Databricks recommends that you append the “dot-asterisk” notation to specify
databricks-connect==X.Y.*instead ofdatabricks-connect=X.Y, to make sure that the most recent package is installed. While this is not a requirement, it helps make sure that you can use the latest supported features for that cluster.
Step 2: Configure connection properties
In this section, you configure properties to establish a connection between Databricks Connect and your remote Azure Databricks cluster. These properties include settings to authenticate Databricks Connect with your cluster.
Beginning with Databricks Connect for Databricks Runtime 13.1 and higher, Databricks Connect includes the Databricks SDK for Python. This SDK implements the Databricks client unified authentication standard, a consolidated and consistent architectural and programmatic approach to authentication. This approach makes setting up and automating authentication with Azure Databricks more centralized and predictable. It enables you to configure Azure Databricks authentication once and then use that configuration across multiple Azure Databricks tools and SDKs without further authentication configuration changes.
Note
The Databricks SDK for Python has not yet implemented Azure MSI authentication.
Databricks Connect for Databricks Runtime 13.0 supports only Azure Databricks personal access token authentication for authentication.
Collect the following configuration properties.
- The Azure Databricks workspace instance name. This is the same as the Server Hostname value for your cluster; see Get connection details for a cluster.
- The ID of your cluster. You can obtain the cluster ID from the URL. See Cluster URL and ID.
- Any other properties that are necessary for the Databricks authentication type that you want to use.
Configure the connection within your code. Databricks Connect searches for configuration properties in the following order until it finds them. Once it finds them, it stops searching through the remaining options:
For Azure Databricks personal access token authentication only, direct configuration of connection properties, specified through the
DatabricksSessionclassFor this option, which applies to Azure Databricks personal access token authentication only, specify the workspace instance name, the Azure Databricks personal access token, and the ID of the cluster.
The following code examples demonstrate how to initialize the
DatabricksSessionclass for Azure Databricks personal access token authentication.Databricks does not recommend that you directly specify these connection properties in your code. Instead, Databricks recommends configuring properties through environment variables or configuration files, as described in later options. The following code examples assume that you provide some implementation of the proposed
retrieve_*functions yourself to get the necessary properties from the user or from some other configuration store, such as Azure KeyVault.# By setting fields in builder.remote: from databricks.connect import DatabricksSession spark = DatabricksSession.builder.remote( host = f"https://{retrieve_workspace_instance_name()}", token = retrieve_token(), cluster_id = retrieve_cluster_id() ).getOrCreate() # Or, by using the Databricks SDK's Config class: from databricks.connect import DatabricksSession from databricks.sdk.core import Config config = Config( host = f"https://{retrieve_workspace_instance_name()}", token = retrieve_token(), cluster_id = retrieve_cluster_id() ) spark = DatabricksSession.builder.sdkConfig(config).getOrCreate() # Or, specify a Databricks configuration profile and # the cluster_id field separately: from databricks.connect import DatabricksSession from databricks.sdk.core import Config config = Config( profile = "<profile-name>", cluster_id = retrieve_cluster_id() ) spark = DatabricksSession.builder.sdkConfig(config).getOrCreate() # Or, by setting the Spark Connect connection string in builder.remote: from databricks.connect import DatabricksSession workspace_instance_name = retrieve_workspace_instance_name() token = retrieve_token() cluster_id = retrieve_cluster_id() spark = DatabricksSession.builder.remote( f"sc://{workspace_instance_name}:443/;token={token};x-databricks-cluster-id={cluster_id}" ).getOrCreate()For all Azure Databricks authentication types, an Azure Databricks configuration profile name, specified using
profile()For this option, create or identify an Azure Databricks configuration profile containing the field
cluster_idand any other fields that are necessary for the Databricks authentication type that you want to use. Then set the name of this configuration profile through theConfigclass.Alternatively, you can specify
cluster_idseparately from the configuration profile. Instead of directly specifying the cluster ID in your code, the following code examples assume that you provide some implementation of the proposedretrieve_cluster_idfunction yourself to get the cluster ID from the user or from some other configuration store, such as Azure KeyVault.For example:
# Specify a Databricks configuration profile that contains the # cluster_id field: from databricks.connect import DatabricksSession spark = DatabricksSession.builder.profile("<profile-name>").getOrCreate()For Azure Databricks personal access token authentication only, the
SPARK_REMOTEenvironment variableFor this option, which applies to Azure Databricks personal access token authentication only, set the
SPARK_REMOTEenvironment variable to the following string, replacing the placeholders with the appropriate values.sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>Then initialize the
DatabricksSessionclass as follows:from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()To set environment variables, see your operating system’s documentation.
For all Azure Databricks authentication types, the
DATABRICKS_CONFIG_PROFILEenvironment variableFor this option, create or identify an Azure Databricks configuration profile containing the field
cluster_idand any other fields that are necessary for the Databricks authentication type that you want to use.Set the
DATABRICKS_CONFIG_PROFILEenvironment variable to the name of this configuration profile. Then initialize theDatabricksSessionclass as follows:from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()To set environment variables, see your operating system’s documentation.
For all Azure Databricks authentication types, an environment variable for each connection property
For this option, set the
DATABRICKS_CLUSTER_IDenvironment variable and any other environment variables that are necessary for the Databricks authentication type that you want to use. Then initialize theDatabricksSessionclass as follows:from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()To set environment variables, see your operating system’s documentation.
For all Azure Databricks authentication types, an Azure Databricks configuration profile named
DEFAULTFor this option, create or identify an Azure Databricks configuration profile containing the field
cluster_idand any other fields that are necessary for the Databricks authentication type that you want to use. Name this configuration profileDEFAULT.Then initialize the
DatabricksSessionclass as follows:from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()
If you choose to use Azure Databricks personal access token authentication authentication, you can use the included
pysparkutility to test connectivity to your Azure Databricks cluster as follows.With your virtual environment still activated, run the following command:
If you set the
SPARK_REMOTEenvironment variable earlier, run the following command:pysparkIf you did not set the
SPARK_REMOTEenvironment variable earlier, run the following command instead:pyspark --remote "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>"The Spark shell appears, for example:
Python 3.10 ... [Clang ...] on darwin Type "help", "copyright", "credits" or "license" for more information. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 13.0 /_/ Using Python version 3.10 ... Client connected to the Spark Connect server at sc://...:.../;token=...;x-databricks-cluster-id=... SparkSession available as 'spark'. >>>At the
>>>prompt, run a simple PySpark command, such asspark.range(1,10).show(). If there are no errors, you have successfully connected.If you have successfully connected, to stop the Spark shell, press
Ctrl + dorCtrl + z, or run the commandquit()orexit().
Use Databricks Connect
These sections describes how to configure many popular IDEs and notebook servers to use the Databricks Connect client. Or you can use the built-in Spark shell.
In this section:
- JupyterLab with Python
- Classic Jupyter Notebook with Python
- Visual Studio Code with Python
- PyCharm with Python
- Eclipse with PyDev
- Spark shell with Python
JupyterLab with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
To use Databricks Connect with JupyterLab and Python, follow these instructions.
To install JupyterLab, with your Python virtual environment activated, run the following command from your terminal or Command Prompt:
pip3 install jupyterlabTo start JupyterLab in your web browser, run the following command from your activated Python virtual environment:
jupyter labIf JupyterLab does not appear in your web browser, copy the URL that starts with
localhostor127.0.0.1from your virtual environment, and enter it in your web browser’s address bar.Create a new notebook: in JupyterLab, click File > New > Notebook on the main menu, select Python 3 (ipykernel) and click Select.
In the notebook’s first cell, enter either the example code or your own code. If you use your own code, at minimum you must initialize
DatabricksSessionas shown in the example code.To run the notebook, click Run > Run All Cells.
To debug the notebook, click the bug (Enable Debugger) icon next to Python 3 (ipykernel) in the notebook’s toolbar. Set one or more breakpoints, and then click Run > Run All Cells.
To shut down JupyterLab, click File > Shut Down. If the JupyterLab process is still running in your terminal or Command Prompt, stop this process by pressing
Ctrl + cand then enteringyto confirm.
For more specific debug instructions, see Debugger.
Classic Jupyter Notebook with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
To use Databricks Connect with classic Jupyter Notebook and Python, follow these instructions.
To install classic Jupyter Notebook, with your Python virtual environment activated, run the following command from your terminal or Command Prompt:
pip3 install notebookTo start classic Jupyter Notebook in your web browser, run the following command from your activated Python virtual environment:
jupyter notebookIf classic Jupyter Notebook does not appear in your web browser, copy the URL that starts with
localhostor127.0.0.1from your virtual environment, and enter it in your web browser’s address bar.Create a new notebook: in classic Jupyter Notebook, on the Files tab, click New > Python 3 (ipykernel).
In the notebook’s first cell, enter either the example code or your own code. If you use your own code, at minimum you must initialize
DatabricksSessionas shown in the example code.To run the notebook, click Cell > Run All.
To debug the notebook, add the following line of code at the beginning of your notebook:
from IPython.core.debugger import set_traceAnd then call
set_trace()to enter debug statements at that point of notebook execution.To shut down classic Jupyter Notebook, click File > Close and Halt. If the classic Jupyter Notebook process is still running in your terminal or Command Prompt, stop this process by pressing
Ctrl + cand then enteringyto confirm.
Visual Studio Code with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
Tip
The Databricks extension for Visual Studio Code already has built-in support for Databricks Connect for Databricks Runtime 13.0 and higher. See Run or debug Python code with Databricks Connect in the documentation for the Databricks extension for Visual Studio Code.
To use Databricks Connect with Visual Studio Code and Python, follow these instructions.
- Start Visual Studio Code.
- Open the folder that contains your Python virtual environment (File > Open Folder).
- In the Visual Studio Code Terminal (View > Terminal), activate the virtual environment.
- Set the current Python interpreter to be the one that is referenced from the virtual environment:
- On the Command Palette (View > Command Palette), type
Python: Select Interpreter, and then press Enter. - Select the path to the Python interpreter that is referenced from the virtual environment.
- On the Command Palette (View > Command Palette), type
- Add to the folder a Python code (
.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initializeDatabricksSessionas shown in the example code. - Run the code:
- With the Python code file open, set any breakpoints where you want your code to pause while running.
- Click the Run and Debug icon on the sidebar, or click View > Run on the main menu.
- In the Run and Debug view, click the Run and Debug button.
- Follow the on-screen instructions to start running and debugging the code.
For more specific run and debug instructions, see Configure and run the debugger and Python debugging in VS Code.
PyCharm with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
IntelliJ IDEA Ultimate provides plugin support for PyCharm with Python also. For details, see Python plug-in for IntelliJ IDEA Ultimate.
To use Databricks Connect with PyCharm and Python, follow these instructions.
- Start PyCharm.
- Create a project: click File > New Project.
- For Location, click the folder icon, and then select the path to your Python virtual environment.
- Select Previously configured interpreter.
- For Interpreter, click the ellipses.
- Click System Interpreter.
- For Interpreter, click the ellipses, and select the full path to the Python interpreter that is referenced from the virtual environment. Then click OK.
- Click OK again.
- Click Create.
- Click Create from Existing Sources.
- Add to the project a Python code (
.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initializeDatabricksSessionas shown in the example code. - With the Python code file open, set any breakpoints where you want your code to pause while running.
- Click Run > Run or Run > Debug.
- Follow the on-screen instructions to start running or debugging the code.
For more specific run and debug instructions, see Run without any previous configuring and Debug.
Eclipse with PyDev
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
To use Databricks Connect and Eclipse with PyDev, follow these instructions.
- Start Eclipse.
- Create a project: click File > New > Project > PyDev > PyDev Project, and then click Next.
- Specify a Project name.
- For Project contents, specify the path to your Python virtual environment.
- Click Please configure an interpreter before proceding.
- Click Manual config.
- Click New > Browse for python/pypy exe.
- Browse to and select select the full path to the Python interpreter that is referenced from the virtual environment, and then click Open.
- In the Select interpreter dialog, click OK.
- In the Selection needed dialog, click OK.
- In the Preferences dialog, click Apply and Close.
- In the PyDev Project dialog, click Finish.
- Click Open Perspective.
- Add to the project a Python code (
.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initializeDatabricksSessionas shown in the example code. - With the Python code file open, set any breakpoints where you want your code to pause while running.
- Click Run > Run or Run > Debug.
For more specific run and debug instructions, see Running a Program.
Spark shell with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
The Spark shell works with Azure Databricks personal access token authentication authentication only.
To use Databricks Connect with the Spark shell and Python, follow these instructions.
To start the Spark shell and to connect it to your running cluster, run one of the following commands from your activated Python virtual environment:
If you set the
SPARK_REMOTEenvironment variable earlier, run the following command:pysparkIf you did not set the
SPARK_REMOTEenvironment variable earlier, run the following command instead:pyspark --remote "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>"The Spark shell appears, for example:
Python 3.10 ... [Clang ...] on darwin Type "help", "copyright", "credits" or "license" for more information. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 13.x.dev0 /_/ Using Python version 3.10 ... Client connected to the Spark Connect server at sc://...:.../;token=...;x-databricks-cluster-id=... SparkSession available as 'spark'. >>>Refer to Interactive Analysis with the Spark Shell for information about how to use the Spark shell with Python to run commands on your cluster.
Use the built-in
sparkvariable to represent theSparkSessionon your running cluster, for example:>>> df = spark.read.table("samples.nyctaxi.trips") >>> df.show(5) +--------------------+---------------------+-------------+-----------+----------+-----------+ |tpep_pickup_datetime|tpep_dropoff_datetime|trip_distance|fare_amount|pickup_zip|dropoff_zip| +--------------------+---------------------+-------------+-----------+----------+-----------+ | 2016-02-14 16:52:13| 2016-02-14 17:16:04| 4.94| 19.0| 10282| 10171| | 2016-02-04 18:44:19| 2016-02-04 18:46:00| 0.28| 3.5| 10110| 10110| | 2016-02-17 17:13:57| 2016-02-17 17:17:55| 0.7| 5.0| 10103| 10023| | 2016-02-18 10:36:07| 2016-02-18 10:41:45| 0.8| 6.0| 10022| 10017| | 2016-02-22 14:14:41| 2016-02-22 14:31:52| 4.51| 17.0| 10110| 10282| +--------------------+---------------------+-------------+-----------+----------+-----------+ only showing top 5 rowsTo stop the Spark shell, press
Ctrl + dorCtrl + z, or run the commandquit()orexit().
Code examples
Databricks provides several example applications that show how to use Databricks Connect. See the databricks-demos/dbconnect-examples repository in GitHub.
You can also use the following simpler code examples to experiment with Databricks Connect. These examples assume that you are using default authentication for Databricks Connect.
This simple code example queries the specified table and then shows the specified table’s first 5 rows. To use a different table, adjust the call to spark.read.table.
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
df = spark.read.table("samples.nyctaxi.trips")
df.show(5)
This longer code example does the following:
- Creates an in-memory DataFrame.
- Creates a table with the name
zzz_demo_temps_tablewithin thedefaultschema. If the table with this name already exists, the table is deleted first. To use a different schema or table, adjust the calls tospark.sql,temps.write.saveAsTable, or both. - Saves the DataFrame’s contents to the table.
- Runs a
SELECTquery on the table’s contents. - Shows the query’s result.
- Deletes the table.
from databricks.connect import DatabricksSession
from pyspark.sql.types import *
from datetime import date
spark = DatabricksSession.builder.getOrCreate()
# Create a Spark DataFrame consisting of high and low temperatures
# by airport code and date.
schema = StructType([
StructField('AirportCode', StringType(), False),
StructField('Date', DateType(), False),
StructField('TempHighF', IntegerType(), False),
StructField('TempLowF', IntegerType(), False)
])
data = [
[ 'BLI', date(2021, 4, 3), 52, 43],
[ 'BLI', date(2021, 4, 2), 50, 38],
[ 'BLI', date(2021, 4, 1), 52, 41],
[ 'PDX', date(2021, 4, 3), 64, 45],
[ 'PDX', date(2021, 4, 2), 61, 41],
[ 'PDX', date(2021, 4, 1), 66, 39],
[ 'SEA', date(2021, 4, 3), 57, 43],
[ 'SEA', date(2021, 4, 2), 54, 39],
[ 'SEA', date(2021, 4, 1), 56, 41]
]
temps = spark.createDataFrame(data, schema)
# Create a table on the Databricks cluster and then fill
# the table with the DataFrame's contents.
# If the table already exists from a previous run,
# delete it first.
spark.sql('USE default')
spark.sql('DROP TABLE IF EXISTS zzz_demo_temps_table')
temps.write.saveAsTable('zzz_demo_temps_table')
# Query the table on the Databricks cluster, returning rows
# where the airport code is not BLI and the date is later
# than 2021-04-01. Group the results and order by high
# temperature in descending order.
df_temps = spark.sql("SELECT * FROM zzz_demo_temps_table " \
"WHERE AirportCode != 'BLI' AND Date > '2021-04-01' " \
"GROUP BY AirportCode, Date, TempHighF, TempLowF " \
"ORDER BY TempHighF DESC")
df_temps.show()
# Results:
#
# +-----------+----------+---------+--------+
# |AirportCode| Date|TempHighF|TempLowF|
# +-----------+----------+---------+--------+
# | PDX|2021-04-03| 64| 45|
# | PDX|2021-04-02| 61| 41|
# | SEA|2021-04-03| 57| 43|
# | SEA|2021-04-02| 54| 39|
# +-----------+----------+---------+--------+
# Clean up by deleting the table from the Databricks cluster.
spark.sql('DROP TABLE zzz_demo_temps_table')
Migrate to the latest Databricks Connect
Follow these guidelines to migrate your existing Python code project or coding environment from Databricks Connect for Databricks Runtime 12.2 LTS and lower to Databricks Connect for Databricks Runtime 13.0 and higher.
Note
Databricks Connect for Databricks Runtime 13.0 and higher currently supports only Python-based projects and coding environments.
Install the correct version of Python as listed in the requirements to match your Azure Databricks cluster, if it is not already installed locally.
Upgrade your Python virtual environment to use the correct version of Python to match your cluster, if needed. For instructions, see your virtual environment provider’s documentation.
With your virtual environment activated, uninstall PySpark from your virtual environment:
pip3 uninstall pysparkWith your virtual environment still activated, uninstall Databricks Connect for Databricks Runtime 12.2 LTS and lower:
pip3 uninstall databricks-connectWith your virtual environment still activated, install Databricks Connect for Databricks Runtime 13.0 and higher:
pip3 install --upgrade "databricks-connect==13.1.*" # Or X.Y.* to match your cluster version.Note
Databricks recommends that you append the “dot-asterisk” notation to specify
databricks-connect==X.Y.*instead ofdatabricks-connect=X.Y, to make sure that the most recent package is installed. While this is not a requirement, it helps make sure that you can use the latest supported features for that cluster.Update your Python code to initialize the
sparkvariable (which represents an instantiation of theDatabricksSessionclass, similar toSparkSessionin PySpark). For code examples, see Step 2: Configure connection properties.
Access Databricks Utilities
This section describes how to use Databricks Connect to access Databricks Utilities.
You can call Databricks File System (DBFS) functions from within an Azure Databricks workspace. To do this, you use the WorkspaceClient class’s dbfs variable. This approach is similar to calling Databricks Utilities through the dbfs variable from a notebook within a workspace. The WorkspaceClient class belongs to the Databricks SDK for Python, which is included in Databricks Connect for Databricks Runtime 13.0 and higher.
Tip
You can also use the included Databricks SDK for Python to access any available Databricks REST API, not just the DBFS API. See databricks-sdk on PyPI.
To initialize WorkspaceClient, you must provide enough information to authenticate the Databricks SDK for Python with the workspace. For example, you can:
Hard-code the workspace URL and your access token directly within your code, and then intialize
WorkspaceClientas follows. Although this option is supported, Databricks does not recommend this option, as it can expose sensitive information, such as access tokens, if your code is checked into version control or otherwise shared:w = WorkspaceClient(host = "https://<workspace-instance-name>", token = "<access-token-value")Create or specify a configuration profile that contains the fields
hostandtoken, and then intialize theWorkspaceClientas follows:w = WorkspaceClient(profile = "<profile-name>")Set the environment variables
DATABRICKS_HOSTandDATABRICKS_TOKENin the same way you set them for Databricks Connect, and then initializeWorkspaceClientas follows:w = WorkspaceClient()
The Databricks SDK for Python does not recognize the SPARK_REMOTE environment variable for Databricks Connect.
For additional Azure Databricks authentication options for the Databricks SDK for Python, as well as how to initialize AccountClient within the Databricks SDK for Python to access available Databricks REST APIs at the account level instead of at the workspace level, see databricks-sdk on PyPI.
The following example creates a file named zzz_hello.txt in the DBFS root within the workspace, writes data into the file, closes the file, reads the data from the file, and then deletes the file. This example assumes that the environment variables DATABRICKS_HOST and DATABRICKS_TOKEN have already been set:
from databricks.sdk import WorkspaceClient
import base64
w = WorkspaceClient()
file_path = "/zzz_hello.txt"
file_data = "Hello, Databricks!"
# The data must be base64-encoded before being written.
file_data_base64 = base64.b64encode(file_data.encode())
# Create the file.
file_handle = w.dbfs.create(
path = file_path,
overwrite = True
).handle
# Add the base64-encoded version of the data.
w.dbfs.add_block(
handle = file_handle,
data = file_data_base64.decode()
)
# Close the file after writing.
w.dbfs.close(handle = file_handle)
# Read the file's contents and then decode and print it.
response = w.dbfs.read(path = file_path)
print(base64.b64decode(response.data).decode())
# Delete the file.
w.dbfs.delete(path = file_path)
Tip
You can also access the Databricks Utilities secrets utility through w.secrets, the jobs utility through w.jobs, and the library utility through w.libraries.
Set Hadoop configurations
On the client you can set Hadoop configurations using the spark.conf.set API, which applies to SQL and DataFrame operations. Hadoop configurations set on the sparkContext must be set in the cluster configuration or using a notebook. This is because configurations set on sparkContext are not tied to user sessions but apply to the entire cluster.
Troubleshooting
This section describes some common issues that you might encounter with Databricks Connect and how to resolve them.
In this section:
- Error: StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status 500
- Python version mismatch
- Conflicting PySpark installations
- Conflicting or Missing
PATHentry for binaries - The filename, directory name, or volume label syntax is incorrect on Windows
Error: StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status 500
Issue: When you try to run code with Databricks Connect, you get an error messages that contains strings such as StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status: 500.
Possible cause: Databricks Connect cannot reach your cluster.
Recommended solutions:
- Check to make sure that your workspace instance name is correct. If you use environment variables, check to make sure the related environment variable is available and correct on your local development machine.
- Check to make sure that your cluster ID is correct. If you use environment variables, check to make sure the related environment variable is available and correct on your local development machine.
- Check to make sure that your cluster has the correct custom cluster version that is compatible with Databricks Connect.
Python version mismatch
Check the Python version you are using locally has at least the same minor release as the version on the cluster (for example, 3.10.11 versus 3.10.10 is OK, 3.10 versus 3.9 is not).
If you have multiple Python versions installed locally, ensure that Databricks Connect is using the right one by setting the PYSPARK_PYTHON environment variable (for example, PYSPARK_PYTHON=python3).
Conflicting PySpark installations
The databricks-connect package conflicts with PySpark. Having both installed will cause errors when initializing the Spark context in Python. This can manifest in several ways, including “stream corrupted” or “class not found” errors. If you have PySpark installed in your Python environment, ensure it is uninstalled before installing databricks-connect. After uninstalling PySpark, make sure to fully re-install the Databricks Connect package:
pip3 uninstall pyspark
pip3 uninstall databricks-connect
pip3 install --upgrade "databricks-connect==13.1.*" # or X.Y.* to match your specific cluster version.
Conflicting or Missing PATH entry for binaries
It is possible your PATH is configured so that commands like spark-shell will be running some other previously installed binary instead of the one provided with Databricks Connect. You should make sure either the Databricks Connect binaries take precedence, or remove the previously installed ones.
If you can’t run commands like spark-shell, it is also possible your PATH was not automatically set up by pip3 install and you’ll need to add the installation bin dir to your PATH manually. It’s possible to use Databricks Connect with IDEs even if this isn’t set up.
The filename, directory name, or volume label syntax is incorrect on Windows
If you are using Databricks Connect on Windows and see:
The filename, directory name, or volume label syntax is incorrect.
Databricks Connect was installed into a directory with a space in your path. You can work around this by either installing into a directory path without spaces, or configuring your path using the short name form.
Limitations
Databricks Connect does not support the following Azure Databricks features and third-party platforms.
PySpark DataFrame API limitations
- The
SparkContextclass and its methods are not available. - Creating DataFrames larger than 128 MB from local data is not supported and will fail.
- Resilient distributed datasets (RDDs) and Datasets are not supported. Only DataFrames are supported.
Azure Databricks and Databricks Connect limitations
- Queries with a duration longer than 3600 seconds are not supported and will fail.
- Synchronizing the local development environment with the remote cluster is not supported.
- Make sure that the Python version and any Python packages that you use in your local development environment match their equivalents that are installed on the cluster as much as possible, to help ensure code compatiblity and to help reduce unexpected run time errors.
- Only Python is supported. R, Scala, and Java are not supported.
- Distributed training is not supported.
- MLflow is supported, but not model inference with
mlflow.pyfunc.spark_udf(spark, ...). You can load the model locally withmlflow.pyfunc.load_model(<model>), or you can wrap it as a custom Pandas UDF. - You cannot change the Log4j log level through
SparkContext. - Mosaic is not supported.
Azure Databricks cluster limitations
The Azure Databricks workspace user who is associated with an access token that is used by Databricks Connect must have Can Attach To permissions or greater for the target cluster.
CREATE TABLE table AS SELECT ...SQL commands do not always work. Instead, usespark.sql("SELECT ...").write.saveAsTable("table").Azure Active Directory credential passthrough is supported only on standard clusters running Databricks Runtime 7.3 LTS and above, and is not compatible with service principal authentication.
Azure Active Directory (AD) tokens are not automatically refreshed and will expire one hour after they are initially generated.
The following Databricks Utilities:
Feedback
Submit and view feedback for