Databricks Connect
Note
This article covers Databricks Connect for Databricks Runtime 13.0 and higher, which is in Public Preview.
For prior Databricks Runtime versions, see Databricks Connect for Databricks Runtime 11.3 LTS and lower.
Databricks Connect allows you to connect popular IDEs such as Visual Studio Code and PyCharm, notebook servers, and other custom applications to Azure Databricks clusters.
This article explains how Databricks Connect works, walks you through the steps to get started with Databricks Connect, and explains how to troubleshoot issues that may arise when using Databricks Connect.
Overview
Databricks Connect is a client library for the Databricks Runtime. It allows you to write jobs using Spark APIs and run them remotely on an Azure Databricks cluster instead of in the local Spark session.
For example, when you run the DataFrame command spark.read.format(...).load(...).groupBy(...).agg(...).show() using Databricks Connect, the logical representation of the command is sent to the Spark server running in Azure Databricks for execution on the remote cluster.
With Databricks Connect, you can:
Run large-scale Spark jobs from any Python application. Anywhere you can
import pyspark, you can now run Spark jobs directly from your application, without needing to install any IDE plugins or use Spark submission scripts.Note
Databricks Connect for Databricks Runtime 13.0 and higher currently supports running only Python applications.
Step through and debug code in your IDE even when working with a remote cluster.
Iterate quickly when developing libraries. You do not need to restart the cluster after changing Python library dependencies in Databricks Connect, because each client session is isolated from each other in the cluster.
Shut down idle clusters without losing work. Because the client application is decoupled from the cluster, it is unaffected by cluster restarts or upgrades, which would normally cause you to lose all the variables, RDDs, and DataFrame objects defined in a notebook.
For Databricks Runtime 13.0 and higher, Databricks Connect is now built on open-source Spark Connect. Spark Connect introduces a decoupled client-server architecture for Apache Spark that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. With this “V2” architecture based on Spark Connect, Databricks Connect becomes a thin client that is simple and easy to use. Spark Connect can be embedded everywhere to connect to Azure Databricks: in IDEs, notebooks, and applications, allowing individual users and partners alike to build new (interactive) user experiences based on the Databricks Lakehouse. For more information about Spark Connect, see Introducing Spark Connect.
Requirements
This section lists the requirements for Databricks Connect.
An Azure Databricks workspace and its corresponding account that are enabled for Unity Catalog. See Get started using Unity Catalog and Enable a workspace for Unity Catalog.
Only clusters that are compatible with Unity Catalog are supported. These include clusters with Single User or Shared cluster access modes. See What is cluster access mode?.
Only Databricks Runtime 13.0 ML and Databricks Runtime 13.0 are supported.
The minor version of your client Python installation must be the same as the minor Python version of your Azure Databricks cluster. Databricks Runtime 13.0 ML and Databricks Runtime 13.0 both use Python 3.10.
Databricks strongly recommends that you have a Python virtual environment activated for each Python version that you use with Databricks Connect. Python virtual environments help to make sure that you are using the correct versions of Python and Databricks Connect together. This can help to reduce or shorten resolving related technical issues.
For example, if you’re using venv on your development machine and your cluster is running Python 3.10, you must create a
venvenvironment with that version. The following example command generates the scripts to activate avenvenvironment with Python 3.10, and this command then places those scripts within a hidden folder named.venvwithin the current working directory:# Linux and macOS python3.10 -m venv ./.venv # Windows python3.10 -m venv .\.venvTo use these scripts to activate this
venvenvironment, see How venvs work.The Databricks Connect major and minor package version should match your Databricks Runtime version. Databricks recommends that you always use the most recent package of Databricks Connect that matches your Databricks Runtime version. For example, when you use a Databricks Runtime 13.0 cluster, you should also use the
databricks-connect==13.0.*package. While this is not a requirement, it helps make sure that you can use the latest supported features for that cluster.Note
See the Databricks Connect release notes for a list of available Databricks Connect releases and maintenance updates.
Set up the client
Complete the following steps to set up the local client for Databricks Connect.
Note
Before you begin to set up the local Databricks Connect client, you must meet the requirements for Databricks Connect.
Tip
If you already have the Databricks extension for Visual Studio Code installed, you do not need to follow these setup instructions.
The Databricks extension for Visual Studio Code already has built-in support for Databricks Connect for Databricks Runtime 13.0 and higher. Skip ahead to Run or debug Python code with Databricks Connect in the documentation for the Databricks extension for Visual Studio Code.
Step 1: Install the Databricks Connect client
With your virtual environment activated, uninstall PySpark, if it is already installed, by running the
uninstallcommand. This is required because thedatabricks-connectpackage conflicts with PySpark. For details, see Conflicting PySpark installations. To check whether PySpark is already installed, run theshowcommand.# Is PySpark already installed? pip3 show pyspark # Uninstall PySpark pip3 uninstall pysparkWith your virtual environment still activated, install the Databricks Connect client by running the
installcommand. Use the--upgradeoption to upgrade any existing client installation to the specified version.pip3 install --upgrade "databricks-connect==13.0.*" # Or X.Y.* to match your cluster version.Note
Databricks recommends that you append the “dot-asterisk” notation to specify
databricks-connect==X.Y.*instead ofdatabricks-connect=X.Y, to make sure that the most recent package is installed. While this is not a requirement, it helps make sure that you can use the latest supported features for that cluster.
Step 2: Configure connection properties
Collect the following configuration properties.
The Azure Databricks workspace instance name. This is the same as the Server Hostname value for your cluster; see Get connection details for a cluster.
Your Azure Databricks personal access token.
Note
Databricks Connect for Databricks Runtime 13.0 and higher does not support Azure Active Directory (Azure AD) tokens.
The ID of your cluster. You can obtain the cluster ID from the URL. See Cluster URL and ID.
The port that Databricks Connect connects to on your cluster, which is typically
443. This is also the Port value for your cluster; see Get connection details for a cluster. This article assumes that the port is443unless otherwise specified. If your port is not443, substitute443with your port throughout this article.
Configure the connection within your code. You have four options, as follows. Databricks Connect searches for configuration properties in the following order until it finds them. Once it finds them, it stops searching through the remaining options:
- Hard-coded configuration properties specified directly within your code. Although this option is supported, Databricks does not recommend this option, as it can expose sensitive information, such as access tokens, if your code is checked into version control or otherwise shared.
- The specified configuration profile containing the fields
host,token, andcluster_id. - The
SPARK_REMOTEenvironment variable. - The
DATABRICKS_HOST,DATABRICKS_TOKEN, andDATABRICKS_CLUSTER_IDenvironment variables.
To hard-code configuration properties directly within your code (supported, but not recommended), initialize the
DatabricksSessionclass in one of two ways as follows:from databricks.connect import DatabricksSession spark = DatabricksSession.builder.remote( host = "https://<workspace-instance-name>", token = "<access-token-value>", cluster_id = "<cluster-id>" ).getOrCreate() # Or... from databricks.connect import DatabricksSession spark = DatabricksSession.builder.remote( "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>" ).getOrCreate()To use a configuration profile, manually create a configuration profile within your
.databrickscfgfile, using the following pattern:[<some-unique-profile-name>] host = https://<workspace-instance-name> token = <access-token-value> cluster_id = <cluster-id>And then initialize the
DatabricksSessionclass as follows:from databricks.connect import DatabricksSession from databricks.sdk.core import Config config = Config(profile = "<profile-name>") spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()To use the
SPARK_REMOTEenvironment variable, set the environment variable to this pattern:sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>And then initialize the
DatabricksSessionclass as follows:from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()To use the
DATABRICKS_HOST,DATABRICKS_TOKEN, andDATABRICKS_CLUSTER_IDenvironment variables, setDATABRICKS_HOSTtohttp://<workspace-instance-name>; setDATABRICKS_TOKENto your access token’s value; and setDATABRICKS_CLUSTER_IDto your cluster’s ID. Then initialize theDatabricksSessionclass as follows:from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()To set environment variables, see your operating system’s documentation.
Test connectivity to your Azure Databricks cluster as follows.
With your virtual environment still activated, run the following command:
If you set the
SPARK_REMOTEenvironment variable earlier, run the following command:pysparkIf you did not set the
SPARK_REMOTEenvironment variable earlier, run the following command instead:pyspark --remote "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>"The Spark shell appears, for example:
Python 3.10 ... [Clang ...] on darwin Type "help", "copyright", "credits" or "license" for more information. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 13.0 /_/ Using Python version 3.10 ... Client connected to the Spark Connect server at sc://...:.../;token=...;x-databricks-cluster-id=... SparkSession available as 'spark'. >>>At the
>>>prompt, run a simple PySpark command, such asspark.range(1,10).show(). If there are no errors, you have successfully connected.If you have successfully connected, to stop the Spark shell, press
Ctrl + dorCtrl + z, or run the commandquit()orexit().
Use Databricks Connect
These sections describes how to configure many popular IDEs and notebook servers to use the Databricks Connect client. Or you can use the built-in Spark shell.
In this section:
- JupyterLab with Python
- Classic Jupyter Notebook with Python
- Visual Studio Code with Python
- PyCharm with Python
- Eclipse with PyDev
- Spark shell with Python
JupyterLab with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
To use Databricks Connect with JupyterLab and Python, follow these instructions.
To install JupyterLab, with your Python virtual environment activated, run the following command from your terminal or Command Prompt:
pip3 install jupyterlabTo start JupyterLab in your web browser, run the following command from your activated Python virtual environment:
jupyter labIf JupyterLab does not appear in your web browser, copy the URL that starts with
localhostor127.0.0.1from your virtual environment, and enter it in your web browser’s address bar.Create a new notebook: in JupyterLab, click File > New > Notebook on the main menu, select Python 3 (ipykernel) and click Select.
In the notebook’s first cell, enter either the example code or your own code. If you use your own code, at minimum you must initialize
DatabricksSessionas shown in the example code.To run the notebook, click Run > Run All Cells.
To debug the notebook, click the bug (Enable Debugger) icon next to Python 3 (ipykernel) in the notebook’s toolbar. Set one or more breakpoints, and then click Run > Run All Cells.
To shut down JupyterLab, click File > Shut Down. If the JupyterLab process is still running in your terminal or Command Prompt, stop this process by pressing
Ctrl + cand then enteringyto confirm.
For more specific debug instructions, see Debugger.
Classic Jupyter Notebook with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
To use Databricks Connect with classic Jupyter Notebook and Python, follow these instructions.
To install classic Jupyter Notebook, with your Python virtual environment activated, run the following command from your terminal or Command Prompt:
pip3 install notebookTo start classic Jupyter Notebook in your web browser, run the following command from your activated Python virtual environment:
jupyter notebookIf classic Jupyter Notebook does not appear in your web browser, copy the URL that starts with
localhostor127.0.0.1from your virtual environment, and enter it in your web browser’s address bar.Create a new notebook: in classic Jupyter Notebook, on the Files tab, click New > Python 3 (ipykernel).
In the notebook’s first cell, enter either the example code or your own code. If you use your own code, at minimum you must initialize
DatabricksSessionas shown in the example code.To run the notebook, click Cell > Run All.
To debug the notebook, add the following line of code at the beginning of your notebook:
from IPython.core.debugger import set_traceAnd then call
set_trace()to enter debug statements at that point of notebook execution.To shut down classic Jupyter Notebook, click File > Close and Halt. If the classic Jupyter Notebook process is still running in your terminal or Command Prompt, stop this process by pressing
Ctrl + cand then enteringyto confirm.
Visual Studio Code with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
Tip
The Databricks extension for Visual Studio Code already has built-in support for Databricks Connect for Databricks Runtime 13.0 and higher. See Run or debug Python code with Databricks Connect in the documentation for the Databricks extension for Visual Studio Code.
To use Databricks Connect with Visual Studio Code and Python, follow these instructions.
- Start Visual Studio Code.
- Open the folder that contains your Python virtual environment (File > Open Folder).
- In the Visual Studio Code Terminal (View > Terminal), activate the virtual environment.
- Set the current Python interpreter to be the one that is referenced from the virtual environment:
- On the Command Palette (View > Command Palette), type
Python: Select Interpreter, and then press Enter. - Select the path to the Python interpreter that is referenced from the virtual environment.
- On the Command Palette (View > Command Palette), type
- Add to the folder a Python code (
.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initializeDatabricksSessionas shown in the example code. - Run the code:
- With the Python code file open, set any breakpoints where you want your code to pause while running.
- Click the Run and Debug icon on the sidebar, or click View > Run on the main menu.
- In the Run and Debug view, click the Run and Debug button.
- Follow the on-screen instructions to start running and debugging the code.
For more specific run and debug instructions, see Configure and run the debugger and Python debugging in VS Code.
PyCharm with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
IntelliJ IDEA Ultimate provides plugin support for PyCharm with Python also. For details, see Python plug-in for IntelliJ IDEA Ultimate.
To use Databricks Connect with PyCharm and Python, follow these instructions.
- Start PyCharm.
- Create a project: click File > New Project.
- For Location, click the folder icon, and then select the path to your Python virtual environment.
- Select Previously configured interpreter.
- For Interpreter, click the ellipses.
- Click System Interpreter.
- For Interpreter, click the ellipses, and select the full path to the Python interpreter that is referenced from the virtual environment. Then click OK.
- Click OK again.
- Click Create.
- Click Create from Existing Sources.
- Add to the project a Python code (
.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initializeDatabricksSessionas shown in the example code. - With the Python code file open, set any breakpoints where you want your code to pause while running.
- Click Run > Run or Run > Debug.
- Follow the on-screen instructions to start running or debugging the code.
For more specific run and debug instructions, see Run without any previous configuring and Debug.
Eclipse with PyDev
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
To use Databricks Connect and Eclipse with PyDev, follow these instructions.
- Start Eclipse.
- Create a project: click File > New > Project > PyDev > PyDev Project, and then click Next.
- Specify a Project name.
- For Project contents, specify the path to your Python virtual environment.
- Click Please configure an interpreter before proceding.
- Click Manual config.
- Click New > Browse for python/pypy exe.
- Browse to and select select the full path to the Python interpreter that is referenced from the virtual environment, and then click Open.
- In the Select interpreter dialog, click OK.
- In the Selection needed dialog, click OK.
- In the Preferences dialog, click Apply and Close.
- In the PyDev Project dialog, click Finish.
- Click Open Perspective.
- Add to the project a Python code (
.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initializeDatabricksSessionas shown in the example code. - With the Python code file open, set any breakpoints where you want your code to pause while running.
- Click Run > Run or Run > Debug.
For more specific run and debug instructions, see Running a Program.
Spark shell with Python
Note
Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.
To use Databricks Connect with the Spark shell and Python, follow these instructions.
To start the Spark shell and to connect it to your running cluster, run one of the following commands from your activated Python virtual environment:
If you set the
SPARK_REMOTEenvironment variable earlier, run the following command:pysparkIf you did not set the
SPARK_REMOTEenvironment variable earlier, run the following command instead:pyspark --remote "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>"The Spark shell appears, for example:
Python 3.10 ... [Clang ...] on darwin Type "help", "copyright", "credits" or "license" for more information. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 13.x.dev0 /_/ Using Python version 3.10 ... Client connected to the Spark Connect server at sc://...:.../;token=...;x-databricks-cluster-id=... SparkSession available as 'spark'. >>>Refer to Interactive Analysis with the Spark Shell for information about how to use the Spark shell with Python to run commands on your cluster.
Use the built-in
sparkvariable to represent theSparkSessionon your running cluster, for example:>>> df = spark.read.table("samples.nyctaxi.trips") >>> df.show(5) +--------------------+---------------------+-------------+-----------+----------+-----------+ |tpep_pickup_datetime|tpep_dropoff_datetime|trip_distance|fare_amount|pickup_zip|dropoff_zip| +--------------------+---------------------+-------------+-----------+----------+-----------+ | 2016-02-14 16:52:13| 2016-02-14 17:16:04| 4.94| 19.0| 10282| 10171| | 2016-02-04 18:44:19| 2016-02-04 18:46:00| 0.28| 3.5| 10110| 10110| | 2016-02-17 17:13:57| 2016-02-17 17:17:55| 0.7| 5.0| 10103| 10023| | 2016-02-18 10:36:07| 2016-02-18 10:41:45| 0.8| 6.0| 10022| 10017| | 2016-02-22 14:14:41| 2016-02-22 14:31:52| 4.51| 17.0| 10110| 10282| +--------------------+---------------------+-------------+-----------+----------+-----------+ only showing top 5 rowsTo stop the Spark shell, press
Ctrl + dorCtrl + z, or run the commandquit()orexit().
Code examples
You can use the following code examples to experiment with Databricks Connect. These examples assume that you have already set either the SPARK_REMOTE environment variable or the DATABRICKS_HOST, DATABRICKS_TOKEN, and DATABRICKS_CLUSTER_ID environment variables. Alternatively, you can initialize DatabricksSession as follows:
For hard-coded configuration properties directly within your code (supported, but not recommended):
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.remote(
host = "https://<workspace-instance-name>",
token = "<access-token-value>",
cluster_id = "<cluster-id>"
).getOrCreate()
# Or...
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.remote(
"sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>"
).getOrCreate()
For a configuration profile containing the fields host, token, and cluster_id:
from databricks.connect import DatabricksSession
from databricks.sdk.core import Config
config = Config(profile = "<profile-name>")
spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()
This simple code example queries the specified table and then shows the specified table’s first 5 rows. To use a different table, adjust the call to spark.read.table.
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
df = spark.read.table("samples.nyctaxi.trips")
df.show(5)
This longer code example does the following:
- Creates an in-memory DataFrame.
- Creates a table with the name
zzz_demo_temps_tablewithin thedefaultschema. If the table with this name already exists, the table is deleted first. To use a different schema or table, adjust the calls tospark.sql,temps.write.saveAsTable, or both. - Saves the DataFrame’s contents to the table.
- Runs a
SELECTquery on the table’s contents. - Shows the query’s result.
- Deletes the table.
from databricks.connect import DatabricksSession
from pyspark.sql.types import *
from datetime import date
spark = DatabricksSession.builder.getOrCreate()
# Create a Spark DataFrame consisting of high and low temperatures
# by airport code and date.
schema = StructType([
StructField('AirportCode', StringType(), False),
StructField('Date', DateType(), False),
StructField('TempHighF', IntegerType(), False),
StructField('TempLowF', IntegerType(), False)
])
data = [
[ 'BLI', date(2021, 4, 3), 52, 43],
[ 'BLI', date(2021, 4, 2), 50, 38],
[ 'BLI', date(2021, 4, 1), 52, 41],
[ 'PDX', date(2021, 4, 3), 64, 45],
[ 'PDX', date(2021, 4, 2), 61, 41],
[ 'PDX', date(2021, 4, 1), 66, 39],
[ 'SEA', date(2021, 4, 3), 57, 43],
[ 'SEA', date(2021, 4, 2), 54, 39],
[ 'SEA', date(2021, 4, 1), 56, 41]
]
temps = spark.createDataFrame(data, schema)
# Create a table on the Databricks cluster and then fill
# the table with the DataFrame's contents.
# If the table already exists from a previous run,
# delete it first.
spark.sql('USE default')
spark.sql('DROP TABLE IF EXISTS zzz_demo_temps_table')
temps.write.saveAsTable('zzz_demo_temps_table')
# Query the table on the Databricks cluster, returning rows
# where the airport code is not BLI and the date is later
# than 2021-04-01. Group the results and order by high
# temperature in descending order.
df_temps = spark.sql("SELECT * FROM zzz_demo_temps_table " \
"WHERE AirportCode != 'BLI' AND Date > '2021-04-01' " \
"GROUP BY AirportCode, Date, TempHighF, TempLowF " \
"ORDER BY TempHighF DESC")
df_temps.show()
# Results:
#
# +-----------+----------+---------+--------+
# |AirportCode| Date|TempHighF|TempLowF|
# +-----------+----------+---------+--------+
# | PDX|2021-04-03| 64| 45|
# | PDX|2021-04-02| 61| 41|
# | SEA|2021-04-03| 57| 43|
# | SEA|2021-04-02| 54| 39|
# +-----------+----------+---------+--------+
# Clean up by deleting the table from the Databricks cluster.
spark.sql('DROP TABLE zzz_demo_temps_table')
Migrate to the latest Databricks Connect
Follow these guidelines to migrate your existing Python code project or coding environment from Databricks Connect for Databricks Runtime 11.3 LTS and lower to Databricks Connect for Databricks Runtime 13.0 and higher.
Note
Databricks Connect for Databricks Runtime 13.0 and higher currently supports only Python-based projects and coding environments.
Install the correct version of Python as listed in the requirements to match your Azure Databricks cluster, if it is not already installed locally.
Upgrade your Python virtual environment to use the correct version of Python to match your cluster, if needed. For instructions, see your virtual environment provider’s documentation.
With your virtual environment activated, uninstall PySpark from your virtual environment:
pip3 uninstall pysparkWith your virtual environment still activated, uninstall Databricks Connect for Databricks Runtime 11.3 LTS and lower:
pip3 uninstall databricks-connectWith your virtual environment still activated, install Databricks Connect for Databricks Runtime 13.0 and higher:
pip3 install --upgrade "databricks-connect==13.0.*" # Or X.Y.* to match your cluster version.Note
Databricks recommends that you append the “dot-asterisk” notation to specify
databricks-connect==X.Y.*instead ofdatabricks-connect=X.Y, to make sure that the most recent package is installed. While this is not a requirement, it helps make sure that you can use the latest supported features for that cluster.Update your Python code to initialize the
sparkvariable (which represents an instantiation of theDatabricksSessionclass, similar toSparkSessionin PySpark), by using one of the following patterns:For hard-coded configuration properties directly within your code (supported, but not recommended):
from databricks.connect import DatabricksSession spark = DatabricksSession.builder.remote( host = "https://<workspace-instance-name>", token = "<access-token-value>", cluster_id = "<cluster-id>" ).getOrCreate() # Or... from databricks.connect import DatabricksSession spark = DatabricksSession.builder.remote( "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>" ).getOrCreate()For a configuration profile containing the fields
host,token, andcluster_id:from databricks.connect import DatabricksSession from databricks.sdk.core import Config config = Config(profile = "<profile-name>") spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()For either the
SPARK_REMOTEenvironment variable or theDATABRICKS_HOST,DATABRICKS_TOKEN, andDATABRICKS_CLUSTER_IDenvironment variables:from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()
Access Databricks Utilities
This section describes how to use Databricks Connect to access Databricks Utilities.
You can call Databricks File System (DBFS) functions from within an Azure Databricks workspace. To do this, you use the WorkspaceClient class’s dbfs variable. This approach is similar to calling Databricks Utilities through the dbfs variable from a notebook within a workspace. The WorkspaceClient class belongs to the Databricks SDK for Python, which is included in Databricks Connect for Databricks Runtime 13.0 and higher.
Tip
You can also use the Databricks SDK for Python to access any available Databricks REST API, not just the DBFS API. See databricks-sdk on PyPI.
To initialize WorkspaceClient, you must provide enough information to authenticate the Databricks SDK for Python with the workspace. For example, you can:
Hard-code the workspace URL and your access token directly within your code, and then intialize
WorkspaceClientas follows. Although this option is supported, Databricks does not recommend this option, as it can expose sensitive information, such as access tokens, if your code is checked into version control or otherwise shared:w = WorkspaceClient(host = "https://<workspace-instance-name>", token = "<access-token-value")Create or specify a configuration profile that contains the fields
hostandtoken, and then intialize theWorkspaceClientas follows:w = WorkspaceClient(profile = "<profile-name>")Set the environment variables
DATABRICKS_HOSTandDATABRICKS_TOKENin the same way you set them for Databricks Connect, and then initializeWorkspaceClientas follows:w = WorkspaceClient()
The Databricks SDK for Python does not recognize the SPARK_REMOTE environment variable for Databricks Connect.
For additional Azure Databricks authentication options for the Databricks SDK for Python, as well as how to initialize AccountClient within the Databricks SDK for Python to access available Databricks REST APIs at the account level instead of at the workspace level, see databricks-sdk on PyPI.
The following example creates a file named zzz_hello.txt in the DBFS root within the workspace, writes data into the file, closes the file, reads the data from the file, and then deletes the file. This example assumes that the environment variables DATABRICKS_HOST and DATABRICKS_TOKEN have already been set:
from databricks.sdk import WorkspaceClient
import base64
w = WorkspaceClient()
file_path = "/zzz_hello.txt"
file_data = "Hello, Databricks!"
# The data must be base64-encoded before being written.
file_data_base64 = base64.b64encode(file_data.encode())
# Create the file.
file_handle = w.dbfs.create(
path = file_path,
overwrite = True
).handle
# Add the base64-encoded version of the data.
w.dbfs.add_block(
handle = file_handle,
data = file_data_base64.decode()
)
# Close the file after writing.
w.dbfs.close(handle = file_handle)
# Read the file's contents and then decode and print it.
response = w.dbfs.read(path = file_path)
print(base64.b64decode(response.data).decode())
# Delete the file.
w.dbfs.delete(path = file_path)
Tip
You can also access the Databricks Utilities secrets utility through w.secrets, the jobs utility through w.jobs, and the library utility through w.libraries.
Set Hadoop configurations
On the client you can set Hadoop configurations using the spark.conf.set API, which applies to SQL and DataFrame operations. Hadoop configurations set on the sparkContext must be set in the cluster configuration or using a notebook. This is because configurations set on sparkContext are not tied to user sessions but apply to the entire cluster.
Troubleshooting
This section describes some common issues that you might encounter with Databricks Connect and how to resolve them.
In this section:
- Error: StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status 500
- Python version mismatch
- Conflicting PySpark installations
- Conflicting or Missing
PATHentry for binaries - The filename, directory name, or volume label syntax is incorrect on Windows
Error: StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status 500
Issue: When you try to run code with Databricks Connect, you get an error messages that contains strings such as StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status: 500.
Possible cause: Databricks Connect cannot reach your cluster.
Recommended solutions:
- Check to make sure that your workspace instance name is correct. If you use environment variables, check to make sure the related environment variable is available and correct on your local development machine.
- Check to make sure that your cluster ID is correct. If you use environment variables, check to make sure the related environment variable is available and correct on your local development machine.
- Check to make sure that your cluster is running.
- Check to make sure that your cluster has the correct custom cluster version that is compatible with Databricks Connect.
Python version mismatch
Check the Python version you are using locally has at least the same minor release as the version on the cluster (for example, 3.10.11 versus 3.10.10 is OK, 3.10 versus 3.9 is not).
If you have multiple Python versions installed locally, ensure that Databricks Connect is using the right one by setting the PYSPARK_PYTHON environment variable (for example, PYSPARK_PYTHON=python3).
Conflicting PySpark installations
The databricks-connect package conflicts with PySpark. Having both installed will cause errors when initializing the Spark context in Python. This can manifest in several ways, including “stream corrupted” or “class not found” errors. If you have PySpark installed in your Python environment, ensure it is uninstalled before installing databricks-connect. After uninstalling PySpark, make sure to fully re-install the Databricks Connect package:
pip3 uninstall pyspark
pip3 uninstall databricks-connect
pip3 install --upgrade "databricks-connect==13.*" # or X.Y.* to match your specific cluster version.
Conflicting or Missing PATH entry for binaries
It is possible your PATH is configured so that commands like spark-shell will be running some other previously installed binary instead of the one provided with Databricks Connect. You should make sure either the Databricks Connect binaries take precedence, or remove the previously installed ones.
If you can’t run commands like spark-shell, it is also possible your PATH was not automatically set up by pip3 install and you’ll need to add the installation bin dir to your PATH manually. It’s possible to use Databricks Connect with IDEs even if this isn’t set up.
The filename, directory name, or volume label syntax is incorrect on Windows
If you are using Databricks Connect on Windows and see:
The filename, directory name, or volume label syntax is incorrect.
Databricks Connect was installed into a directory with a space in your path. You can work around this by either installing into a directory path without spaces, or configuring your path using the short name form.
Limitations
Databricks Connect does not support the following Azure Databricks features and third-party platforms.
PySpark DataFrame API limitations
- Spark Streaming is not supported.
- Pandas API on Spark is not supported.
- The
SparkContextclass and its methods are not available. - Creating DataFrames larger than 128 MB from local data is not supported and will fail.
- Resilient distributed datasets (RDDs) and Datasets are not supported. Only DataFrames are supported.
Azure Databricks and Databricks Connect limitations
- Queries with a duration longer than 3600 seconds are not supported and will fail.
- Synchronizing the local development environment with the remote cluster is not supported.
- You can only set Spark configuration and environment variables on the cluster. You cannot set them through the
SparkSessionclass. - Make sure that the Python version and any Python packages that you use in your local development environment match their equivalents that are installed on the cluster as much as possible, to help ensure code compatiblity and to help reduce unexpected run time errors.
- OAuth is not supported.
- Only Python is supported. R, Scala, and Java are not supported.
- Distributed training is not supported.
- MLflow is supported, but not model inference with
mlflow.pyfunc.spark_udf(spark, ...). You can load the model locally withmlflow.pyfunc.load_model(<model>), or you can wrap it as a custom Pandas UDF. - The
cogroup()and.applyInPandas()functions are not supported.
Azure Databricks cluster limitations
Clusters do not automatically start when Databricks Connect attempts to connect to them. You must start the target cluster before you instruct Databricks Connect to attempt to connect to it. Otherwise, you could get an error at run time such as
StatusCode.UNAVAILABLE,StatusCode.UNKNOWN,DNS resolution failed, orReceived http2 header with status: 500.The Azure Databricks workspace user who is associated with the access token that is used by Databricks Connect must have Can Attach To permissions or greater for the target cluster.
CREATE TABLE table AS SELECT ...SQL commands do not always work. Instead, usespark.sql("SELECT ...").write.saveAsTable("table").Azure Active Directory credential passthrough is supported only on standard clusters running Databricks Runtime 7.3 LTS and above, and is not compatible with service principal authentication.
Azure Active Directory (AD) tokens are not automatically refreshed and will expire one hour after they are initially generated.
The following Databricks Utilities:
Feedback
Submit and view feedback for