Databricks Connect

Note

This article covers Databricks Connect for Databricks Runtime 13.0 and higher.

For prior Databricks Runtime versions, see Databricks Connect for Databricks Runtime 12.2 LTS and lower.

Databricks Connect allows you to connect popular IDEs such as Visual Studio Code and PyCharm, notebook servers, and other custom applications to Azure Databricks clusters.

This article explains how Databricks Connect works, walks you through the steps to get started with Databricks Connect, and explains how to troubleshoot issues that may arise when using Databricks Connect.

Overview

Databricks Connect is a client library for the Databricks Runtime. It allows you to write jobs using Spark APIs and run them remotely on an Azure Databricks cluster instead of in the local Spark session.

For example, when you run the DataFrame command spark.read.format(...).load(...).groupBy(...).agg(...).show() using Databricks Connect, the logical representation of the command is sent to the Spark server running in Azure Databricks for execution on the remote cluster.

With Databricks Connect, you can:

  • Run large-scale Spark jobs from any Python application. Anywhere you can import pyspark, you can now run Spark jobs directly from your application, without needing to install any IDE plugins or use Spark submission scripts.

    Note

    Databricks Connect for Databricks Runtime 13.0 and higher currently supports running only Python applications.

  • Step through and debug code in your IDE even when working with a remote cluster.

  • Iterate quickly when developing libraries. You do not need to restart the cluster after changing Python library dependencies in Databricks Connect, because each client session is isolated from each other in the cluster.

  • Shut down idle clusters without losing work. Because the client application is decoupled from the cluster, it is unaffected by cluster restarts or upgrades, which would normally cause you to lose all the variables, RDDs, and DataFrame objects defined in a notebook.

For Databricks Runtime 13.0 and higher, Databricks Connect is now built on open-source Spark Connect. Spark Connect introduces a decoupled client-server architecture for Apache Spark that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. With this “V2” architecture based on Spark Connect, Databricks Connect becomes a thin client that is simple and easy to use. Spark Connect can be embedded everywhere to connect to Azure Databricks: in IDEs, notebooks, and applications, allowing individual users and partners alike to build new (interactive) user experiences based on the Databricks Lakehouse. For more information about Spark Connect, see Introducing Spark Connect.

Requirements

This section lists the requirements for Databricks Connect.

  • An Azure Databricks workspace and its corresponding account that are enabled for Unity Catalog. See Get started using Unity Catalog and Enable a workspace for Unity Catalog.

  • A cluster with Databricks Runtime 13.0 or higher installed.

  • Only clusters that are compatible with Unity Catalog are supported. These include clusters with Single User or Shared cluster access modes. See What is cluster access mode?.

  • You must install Python 3 on your development machine, and the minor version of your client Python installation must be the same as the minor Python version of your Azure Databricks cluster. The following table shows the Python version installed with each Databricks Runtime.

    Databricks Runtime version Python version
    13.1 ML, 13.1 3.10
    13.0 ML, 13.0 3.10

    Note

    If you want to use PySpark UDFs, it’s important that your development machine’s installed minor version of Python match the minor version of Python that is included with Databricks Runtime installed on the cluster.

  • Databricks strongly recommends that you have a Python virtual environment activated for each Python version that you use with Databricks Connect. Python virtual environments help to make sure that you are using the correct versions of Python and Databricks Connect together. This can help to reduce or shorten resolving related technical issues.

    For example, if you’re using venv on your development machine and your cluster is running Python 3.10, you must create a venv environment with that version. The following example command generates the scripts to activate a venv environment with Python 3.10, and this command then places those scripts within a hidden folder named .venv within the current working directory:

    # Linux and macOS
    python3.10 -m venv ./.venv
    
    # Windows
    python3.10 -m venv .\.venv
    

    To use these scripts to activate this venv environment, see How venvs work.

  • The Databricks Connect major and minor package version should match your Databricks Runtime version. Databricks recommends that you always use the most recent package of Databricks Connect that matches your Databricks Runtime version. For example, when you use a Databricks Runtime 13.1 cluster, you should also use the databricks-connect==13.1.* package.

    Note

    See the Databricks Connect release notes for a list of available Databricks Connect releases and maintenance updates.

    Using the most recent package of Databricks Connect that matches your Databricks Runtime version is not a requirement. Beginning with Databricks Runtime 13.0, you can use the Databricks Connect package against all versions of Databricks Runtime at or above the version of the Databricks Connect package. However, if you want to use features that are available in later versions of the Databricks Runtime, you must upgrade the Databricks Connect package accordingly.

Set up the client

Complete the following steps to set up the local client for Databricks Connect.

Note

Before you begin to set up the local Databricks Connect client, you must meet the requirements for Databricks Connect.

Tip

If you already have the Databricks extension for Visual Studio Code installed, you do not need to follow these setup instructions.

The Databricks extension for Visual Studio Code already has built-in support for Databricks Connect for Databricks Runtime 13.0 and higher. Skip ahead to Run or debug Python code with Databricks Connect in the documentation for the Databricks extension for Visual Studio Code.

Step 1: Install the Databricks Connect client

  1. With your virtual environment activated, uninstall PySpark, if it is already installed, by running the uninstall command. This is required because the databricks-connect package conflicts with PySpark. For details, see Conflicting PySpark installations. To check whether PySpark is already installed, run the show command.

    # Is PySpark already installed?
    pip3 show pyspark
    
    # Uninstall PySpark
    pip3 uninstall pyspark
    
  2. With your virtual environment still activated, install the Databricks Connect client by running the install command. Use the --upgrade option to upgrade any existing client installation to the specified version.

    pip3 install --upgrade "databricks-connect==13.1.*"  # Or X.Y.* to match your cluster version.
    

    Note

    Databricks recommends that you append the “dot-asterisk” notation to specify databricks-connect==X.Y.* instead of databricks-connect=X.Y, to make sure that the most recent package is installed. While this is not a requirement, it helps make sure that you can use the latest supported features for that cluster.

Step 2: Configure connection properties

In this section, you configure properties to establish a connection between Databricks Connect and your remote Azure Databricks cluster. These properties include settings to authenticate Databricks Connect with your cluster.

Beginning with Databricks Connect for Databricks Runtime 13.1 and higher, Databricks Connect includes the Databricks SDK for Python. This SDK implements the Databricks client unified authentication standard, a consolidated and consistent architectural and programmatic approach to authentication. This approach makes setting up and automating authentication with Azure Databricks more centralized and predictable. It enables you to configure Azure Databricks authentication once and then use that configuration across multiple Azure Databricks tools and SDKs without further authentication configuration changes.

Note

The Databricks SDK for Python has not yet implemented Azure MSI authentication.

Databricks Connect for Databricks Runtime 13.0 supports only Azure Databricks personal access token authentication for authentication.

  1. Collect the following configuration properties.

  2. Configure the connection within your code. Databricks Connect searches for configuration properties in the following order until it finds them. Once it finds them, it stops searching through the remaining options:

    1. For Azure Databricks personal access token authentication only, direct configuration of connection properties, specified through the DatabricksSession class

      For this option, which applies to Azure Databricks personal access token authentication only, specify the workspace instance name, the Azure Databricks personal access token, and the ID of the cluster.

      The following code examples demonstrate how to initialize the DatabricksSession class for Azure Databricks personal access token authentication.

      Databricks does not recommend that you directly specify these connection properties in your code. Instead, Databricks recommends configuring properties through environment variables or configuration files, as described in later options. The following code examples assume that you provide some implementation of the proposed retrieve_* functions yourself to get the necessary properties from the user or from some other configuration store, such as Azure KeyVault.

      # By setting fields in builder.remote:
      from databricks.connect import DatabricksSession
      
      spark = DatabricksSession.builder.remote(
        host       = f"https://{retrieve_workspace_instance_name()}",
        token      = retrieve_token(),
        cluster_id = retrieve_cluster_id()
      ).getOrCreate()
      
      # Or, by using the Databricks SDK's Config class:
      from databricks.connect import DatabricksSession
      from databricks.sdk.core import Config
      
      config = Config(
        host       = f"https://{retrieve_workspace_instance_name()}",
        token      = retrieve_token(),
        cluster_id = retrieve_cluster_id()
      )
      spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()
      
      # Or, specify a Databricks configuration profile and
      # the cluster_id field separately:
      from databricks.connect import DatabricksSession
      from databricks.sdk.core import Config
      
      config = Config(
        profile    = "<profile-name>",
        cluster_id = retrieve_cluster_id()
      )
      
      spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()
      
      # Or, by setting the Spark Connect connection string in builder.remote:
      from databricks.connect import DatabricksSession
      
      workspace_instance_name = retrieve_workspace_instance_name()
      token                   = retrieve_token()
      cluster_id              = retrieve_cluster_id()
      
      spark = DatabricksSession.builder.remote(
        f"sc://{workspace_instance_name}:443/;token={token};x-databricks-cluster-id={cluster_id}"
      ).getOrCreate()
      
    2. For all Azure Databricks authentication types, an Azure Databricks configuration profile name, specified using profile()

      For this option, create or identify an Azure Databricks configuration profile containing the field cluster_id and any other fields that are necessary for the Databricks authentication type that you want to use. Then set the name of this configuration profile through the Config class.

      Alternatively, you can specify cluster_id separately from the configuration profile. Instead of directly specifying the cluster ID in your code, the following code examples assume that you provide some implementation of the proposed retrieve_cluster_id function yourself to get the cluster ID from the user or from some other configuration store, such as Azure KeyVault.

      For example:

      # Specify a Databricks configuration profile that contains the
      # cluster_id field:
      from databricks.connect import DatabricksSession
      
      spark = DatabricksSession.builder.profile("<profile-name>").getOrCreate()
      
    3. For Azure Databricks personal access token authentication only, the SPARK_REMOTE environment variable

      For this option, which applies to Azure Databricks personal access token authentication only, set the SPARK_REMOTE environment variable to the following string, replacing the placeholders with the appropriate values.

      sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>
      

      Then initialize the DatabricksSession class as follows:

      from databricks.connect import DatabricksSession
      
      spark = DatabricksSession.builder.getOrCreate()
      

      To set environment variables, see your operating system’s documentation.

    4. For all Azure Databricks authentication types, the DATABRICKS_CONFIG_PROFILE environment variable

      For this option, create or identify an Azure Databricks configuration profile containing the field cluster_id and any other fields that are necessary for the Databricks authentication type that you want to use.

      Set the DATABRICKS_CONFIG_PROFILE environment variable to the name of this configuration profile. Then initialize the DatabricksSession class as follows:

      from databricks.connect import DatabricksSession
      
      spark = DatabricksSession.builder.getOrCreate()
      

      To set environment variables, see your operating system’s documentation.

    5. For all Azure Databricks authentication types, an environment variable for each connection property

      For this option, set the DATABRICKS_CLUSTER_ID environment variable and any other environment variables that are necessary for the Databricks authentication type that you want to use. Then initialize the DatabricksSession class as follows:

      from databricks.connect import DatabricksSession
      
      spark = DatabricksSession.builder.getOrCreate()
      

      To set environment variables, see your operating system’s documentation.

    6. For all Azure Databricks authentication types, an Azure Databricks configuration profile named DEFAULT

      For this option, create or identify an Azure Databricks configuration profile containing the field cluster_id and any other fields that are necessary for the Databricks authentication type that you want to use. Name this configuration profile DEFAULT.

      Then initialize the DatabricksSession class as follows:

      from databricks.connect import DatabricksSession
      
      spark = DatabricksSession.builder.getOrCreate()
      
  3. If you choose to use Azure Databricks personal access token authentication authentication, you can use the included pyspark utility to test connectivity to your Azure Databricks cluster as follows.

    • With your virtual environment still activated, run the following command:

      If you set the SPARK_REMOTE environment variable earlier, run the following command:

      pyspark
      

      If you did not set the SPARK_REMOTE environment variable earlier, run the following command instead:

      pyspark --remote "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>"
      
    • The Spark shell appears, for example:

      Python 3.10 ...
      [Clang ...] on darwin
      Type "help", "copyright", "credits" or "license" for more information.
      Welcome to
            ____              __
           / __/__  ___ _____/ /__
          _\ \/ _ \/ _ `/ __/  '_/
         /__ / .__/\_,_/_/ /_/\_\   version 13.0
            /_/
      
      Using Python version 3.10 ...
      Client connected to the Spark Connect server at sc://...:.../;token=...;x-databricks-cluster-id=...
      SparkSession available as 'spark'.
      >>>
      
    • At the >>> prompt, run a simple PySpark command, such as spark.range(1,10).show(). If there are no errors, you have successfully connected.

    • If you have successfully connected, to stop the Spark shell, press Ctrl + d or Ctrl + z, or run the command quit() or exit().

Use Databricks Connect

These sections describes how to configure many popular IDEs and notebook servers to use the Databricks Connect client. Or you can use the built-in Spark shell.

In this section:

JupyterLab with Python

Note

Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.

To use Databricks Connect with JupyterLab and Python, follow these instructions.

  1. To install JupyterLab, with your Python virtual environment activated, run the following command from your terminal or Command Prompt:

    pip3 install jupyterlab
    
  2. To start JupyterLab in your web browser, run the following command from your activated Python virtual environment:

    jupyter lab
    

    If JupyterLab does not appear in your web browser, copy the URL that starts with localhost or 127.0.0.1 from your virtual environment, and enter it in your web browser’s address bar.

  3. Create a new notebook: in JupyterLab, click File > New > Notebook on the main menu, select Python 3 (ipykernel) and click Select.

  4. In the notebook’s first cell, enter either the example code or your own code. If you use your own code, at minimum you must initialize DatabricksSession as shown in the example code.

  5. To run the notebook, click Run > Run All Cells.

  6. To debug the notebook, click the bug (Enable Debugger) icon next to Python 3 (ipykernel) in the notebook’s toolbar. Set one or more breakpoints, and then click Run > Run All Cells.

  7. To shut down JupyterLab, click File > Shut Down. If the JupyterLab process is still running in your terminal or Command Prompt, stop this process by pressing Ctrl + c and then entering y to confirm.

For more specific debug instructions, see Debugger.

Classic Jupyter Notebook with Python

Note

Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.

To use Databricks Connect with classic Jupyter Notebook and Python, follow these instructions.

  1. To install classic Jupyter Notebook, with your Python virtual environment activated, run the following command from your terminal or Command Prompt:

    pip3 install notebook
    
  2. To start classic Jupyter Notebook in your web browser, run the following command from your activated Python virtual environment:

    jupyter notebook
    

    If classic Jupyter Notebook does not appear in your web browser, copy the URL that starts with localhost or 127.0.0.1 from your virtual environment, and enter it in your web browser’s address bar.

  3. Create a new notebook: in classic Jupyter Notebook, on the Files tab, click New > Python 3 (ipykernel).

  4. In the notebook’s first cell, enter either the example code or your own code. If you use your own code, at minimum you must initialize DatabricksSession as shown in the example code.

  5. To run the notebook, click Cell > Run All.

  6. To debug the notebook, add the following line of code at the beginning of your notebook:

    from IPython.core.debugger import set_trace

    And then call set_trace() to enter debug statements at that point of notebook execution.

  7. To shut down classic Jupyter Notebook, click File > Close and Halt. If the classic Jupyter Notebook process is still running in your terminal or Command Prompt, stop this process by pressing Ctrl + c and then entering y to confirm.

Visual Studio Code with Python

Note

Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.

Tip

The Databricks extension for Visual Studio Code already has built-in support for Databricks Connect for Databricks Runtime 13.0 and higher. See Run or debug Python code with Databricks Connect in the documentation for the Databricks extension for Visual Studio Code.

To use Databricks Connect with Visual Studio Code and Python, follow these instructions.

  1. Start Visual Studio Code.
  2. Open the folder that contains your Python virtual environment (File > Open Folder).
  3. In the Visual Studio Code Terminal (View > Terminal), activate the virtual environment.
  4. Set the current Python interpreter to be the one that is referenced from the virtual environment:
    1. On the Command Palette (View > Command Palette), type Python: Select Interpreter, and then press Enter.
    2. Select the path to the Python interpreter that is referenced from the virtual environment.
  5. Add to the folder a Python code (.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initialize DatabricksSession as shown in the example code.
  6. Run the code:
    1. With the Python code file open, set any breakpoints where you want your code to pause while running.
    2. Click the Run and Debug icon on the sidebar, or click View > Run on the main menu.
    3. In the Run and Debug view, click the Run and Debug button.
    4. Follow the on-screen instructions to start running and debugging the code.

For more specific run and debug instructions, see Configure and run the debugger and Python debugging in VS Code.

PyCharm with Python

Note

Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.

IntelliJ IDEA Ultimate provides plugin support for PyCharm with Python also. For details, see Python plug-in for IntelliJ IDEA Ultimate.

To use Databricks Connect with PyCharm and Python, follow these instructions.

  1. Start PyCharm.
  2. Create a project: click File > New Project.
  3. For Location, click the folder icon, and then select the path to your Python virtual environment.
  4. Select Previously configured interpreter.
  5. For Interpreter, click the ellipses.
  6. Click System Interpreter.
  7. For Interpreter, click the ellipses, and select the full path to the Python interpreter that is referenced from the virtual environment. Then click OK.
  8. Click OK again.
  9. Click Create.
  10. Click Create from Existing Sources.
  11. Add to the project a Python code (.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initialize DatabricksSession as shown in the example code.
  12. With the Python code file open, set any breakpoints where you want your code to pause while running.
  13. Click Run > Run or Run > Debug.
  14. Follow the on-screen instructions to start running or debugging the code.

For more specific run and debug instructions, see Run without any previous configuring and Debug.

Eclipse with PyDev

Note

Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.

To use Databricks Connect and Eclipse with PyDev, follow these instructions.

  1. Start Eclipse.
  2. Create a project: click File > New > Project > PyDev > PyDev Project, and then click Next.
  3. Specify a Project name.
  4. For Project contents, specify the path to your Python virtual environment.
  5. Click Please configure an interpreter before proceding.
  6. Click Manual config.
  7. Click New > Browse for python/pypy exe.
  8. Browse to and select select the full path to the Python interpreter that is referenced from the virtual environment, and then click Open.
  9. In the Select interpreter dialog, click OK.
  10. In the Selection needed dialog, click OK.
  11. In the Preferences dialog, click Apply and Close.
  12. In the PyDev Project dialog, click Finish.
  13. Click Open Perspective.
  14. Add to the project a Python code (.py) file that contains either the example code or your own code. If you use your own code, at minimum you must initialize DatabricksSession as shown in the example code.
  15. With the Python code file open, set any breakpoints where you want your code to pause while running.
  16. Click Run > Run or Run > Debug.

For more specific run and debug instructions, see Running a Program.

Spark shell with Python

Note

Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.

The Spark shell works with Azure Databricks personal access token authentication authentication only.

To use Databricks Connect with the Spark shell and Python, follow these instructions.

  1. To start the Spark shell and to connect it to your running cluster, run one of the following commands from your activated Python virtual environment:

    If you set the SPARK_REMOTE environment variable earlier, run the following command:

    pyspark
    

    If you did not set the SPARK_REMOTE environment variable earlier, run the following command instead:

    pyspark --remote "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>"
    

    The Spark shell appears, for example:

    Python 3.10 ...
    [Clang ...] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    Welcome to
         ____              __
        / __/__  ___ _____/ /__
       _\ \/ _ \/ _ `/ __/  '_/
      /__ / .__/\_,_/_/ /_/\_\   version 13.x.dev0
         /_/
    
    Using Python version 3.10 ...
    Client connected to the Spark Connect server at sc://...:.../;token=...;x-databricks-cluster-id=...
    SparkSession available as 'spark'.
    >>>
    
  2. Refer to Interactive Analysis with the Spark Shell for information about how to use the Spark shell with Python to run commands on your cluster.

    Use the built-in spark variable to represent the SparkSession on your running cluster, for example:

    >>> df = spark.read.table("samples.nyctaxi.trips")
    >>> df.show(5)
    +--------------------+---------------------+-------------+-----------+----------+-----------+
    |tpep_pickup_datetime|tpep_dropoff_datetime|trip_distance|fare_amount|pickup_zip|dropoff_zip|
    +--------------------+---------------------+-------------+-----------+----------+-----------+
    | 2016-02-14 16:52:13|  2016-02-14 17:16:04|         4.94|       19.0|     10282|      10171|
    | 2016-02-04 18:44:19|  2016-02-04 18:46:00|         0.28|        3.5|     10110|      10110|
    | 2016-02-17 17:13:57|  2016-02-17 17:17:55|          0.7|        5.0|     10103|      10023|
    | 2016-02-18 10:36:07|  2016-02-18 10:41:45|          0.8|        6.0|     10022|      10017|
    | 2016-02-22 14:14:41|  2016-02-22 14:31:52|         4.51|       17.0|     10110|      10282|
    +--------------------+---------------------+-------------+-----------+----------+-----------+
    only showing top 5 rows
    
  3. To stop the Spark shell, press Ctrl + d or Ctrl + z, or run the command quit() or exit().

Code examples

Databricks provides several example applications that show how to use Databricks Connect. See the databricks-demos/dbconnect-examples repository in GitHub.

You can also use the following simpler code examples to experiment with Databricks Connect. These examples assume that you are using default authentication for Databricks Connect.

This simple code example queries the specified table and then shows the specified table’s first 5 rows. To use a different table, adjust the call to spark.read.table.

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.getOrCreate()

df = spark.read.table("samples.nyctaxi.trips")
df.show(5)

This longer code example does the following:

  1. Creates an in-memory DataFrame.
  2. Creates a table with the name zzz_demo_temps_table within the default schema. If the table with this name already exists, the table is deleted first. To use a different schema or table, adjust the calls to spark.sql, temps.write.saveAsTable, or both.
  3. Saves the DataFrame’s contents to the table.
  4. Runs a SELECT query on the table’s contents.
  5. Shows the query’s result.
  6. Deletes the table.
from databricks.connect import DatabricksSession
from pyspark.sql.types import *
from datetime import date

spark = DatabricksSession.builder.getOrCreate()

# Create a Spark DataFrame consisting of high and low temperatures
# by airport code and date.
schema = StructType([
  StructField('AirportCode', StringType(), False),
  StructField('Date', DateType(), False),
  StructField('TempHighF', IntegerType(), False),
  StructField('TempLowF', IntegerType(), False)
])

data = [
  [ 'BLI', date(2021, 4, 3), 52, 43],
  [ 'BLI', date(2021, 4, 2), 50, 38],
  [ 'BLI', date(2021, 4, 1), 52, 41],
  [ 'PDX', date(2021, 4, 3), 64, 45],
  [ 'PDX', date(2021, 4, 2), 61, 41],
  [ 'PDX', date(2021, 4, 1), 66, 39],
  [ 'SEA', date(2021, 4, 3), 57, 43],
  [ 'SEA', date(2021, 4, 2), 54, 39],
  [ 'SEA', date(2021, 4, 1), 56, 41]
]

temps = spark.createDataFrame(data, schema)

# Create a table on the Databricks cluster and then fill
# the table with the DataFrame's contents.
# If the table already exists from a previous run,
# delete it first.
spark.sql('USE default')
spark.sql('DROP TABLE IF EXISTS zzz_demo_temps_table')
temps.write.saveAsTable('zzz_demo_temps_table')

# Query the table on the Databricks cluster, returning rows
# where the airport code is not BLI and the date is later
# than 2021-04-01. Group the results and order by high
# temperature in descending order.
df_temps = spark.sql("SELECT * FROM zzz_demo_temps_table " \
  "WHERE AirportCode != 'BLI' AND Date > '2021-04-01' " \
  "GROUP BY AirportCode, Date, TempHighF, TempLowF " \
  "ORDER BY TempHighF DESC")
df_temps.show()

# Results:
#
# +-----------+----------+---------+--------+
# |AirportCode|      Date|TempHighF|TempLowF|
# +-----------+----------+---------+--------+
# |        PDX|2021-04-03|       64|      45|
# |        PDX|2021-04-02|       61|      41|
# |        SEA|2021-04-03|       57|      43|
# |        SEA|2021-04-02|       54|      39|
# +-----------+----------+---------+--------+

# Clean up by deleting the table from the Databricks cluster.
spark.sql('DROP TABLE zzz_demo_temps_table')

Migrate to the latest Databricks Connect

Follow these guidelines to migrate your existing Python code project or coding environment from Databricks Connect for Databricks Runtime 12.2 LTS and lower to Databricks Connect for Databricks Runtime 13.0 and higher.

Note

Databricks Connect for Databricks Runtime 13.0 and higher currently supports only Python-based projects and coding environments.

  1. Install the correct version of Python as listed in the requirements to match your Azure Databricks cluster, if it is not already installed locally.

  2. Upgrade your Python virtual environment to use the correct version of Python to match your cluster, if needed. For instructions, see your virtual environment provider’s documentation.

  3. With your virtual environment activated, uninstall PySpark from your virtual environment:

    pip3 uninstall pyspark
    
  4. With your virtual environment still activated, uninstall Databricks Connect for Databricks Runtime 12.2 LTS and lower:

    pip3 uninstall databricks-connect
    
  5. With your virtual environment still activated, install Databricks Connect for Databricks Runtime 13.0 and higher:

    pip3 install --upgrade "databricks-connect==13.1.*"  # Or X.Y.* to match your cluster version.
    

    Note

    Databricks recommends that you append the “dot-asterisk” notation to specify databricks-connect==X.Y.* instead of databricks-connect=X.Y, to make sure that the most recent package is installed. While this is not a requirement, it helps make sure that you can use the latest supported features for that cluster.

  6. Update your Python code to initialize the spark variable (which represents an instantiation of the DatabricksSession class, similar to SparkSession in PySpark). For code examples, see Step 2: Configure connection properties.

Access Databricks Utilities

This section describes how to use Databricks Connect to access Databricks Utilities.

You can call Databricks File System (DBFS) functions from within an Azure Databricks workspace. To do this, you use the WorkspaceClient class’s dbfs variable. This approach is similar to calling Databricks Utilities through the dbfs variable from a notebook within a workspace. The WorkspaceClient class belongs to the Databricks SDK for Python, which is included in Databricks Connect for Databricks Runtime 13.0 and higher.

Tip

You can also use the included Databricks SDK for Python to access any available Databricks REST API, not just the DBFS API. See databricks-sdk on PyPI.

To initialize WorkspaceClient, you must provide enough information to authenticate the Databricks SDK for Python with the workspace. For example, you can:

  • Hard-code the workspace URL and your access token directly within your code, and then intialize WorkspaceClient as follows. Although this option is supported, Databricks does not recommend this option, as it can expose sensitive information, such as access tokens, if your code is checked into version control or otherwise shared:

    w = WorkspaceClient(host  = "https://<workspace-instance-name>",
                        token = "<access-token-value")
    
  • Create or specify a configuration profile that contains the fields host and token, and then intialize the WorkspaceClient as follows:

    w = WorkspaceClient(profile = "<profile-name>")
    
  • Set the environment variables DATABRICKS_HOST and DATABRICKS_TOKEN in the same way you set them for Databricks Connect, and then initialize WorkspaceClient as follows:

    w = WorkspaceClient()
    

The Databricks SDK for Python does not recognize the SPARK_REMOTE environment variable for Databricks Connect.

For additional Azure Databricks authentication options for the Databricks SDK for Python, as well as how to initialize AccountClient within the Databricks SDK for Python to access available Databricks REST APIs at the account level instead of at the workspace level, see databricks-sdk on PyPI.

The following example creates a file named zzz_hello.txt in the DBFS root within the workspace, writes data into the file, closes the file, reads the data from the file, and then deletes the file. This example assumes that the environment variables DATABRICKS_HOST and DATABRICKS_TOKEN have already been set:

from databricks.sdk import WorkspaceClient
import base64

w = WorkspaceClient()

file_path  = "/zzz_hello.txt"
file_data  = "Hello, Databricks!"

# The data must be base64-encoded before being written.
file_data_base64 = base64.b64encode(file_data.encode())

# Create the file.
file_handle = w.dbfs.create(
  path      = file_path,
  overwrite = True
).handle

# Add the base64-encoded version of the data.
w.dbfs.add_block(
  handle = file_handle,
  data   = file_data_base64.decode()
)

# Close the file after writing.
w.dbfs.close(handle = file_handle)

# Read the file's contents and then decode and print it.
response = w.dbfs.read(path = file_path)
print(base64.b64decode(response.data).decode())

# Delete the file.
w.dbfs.delete(path = file_path)

Tip

You can also access the Databricks Utilities secrets utility through w.secrets, the jobs utility through w.jobs, and the library utility through w.libraries.

Set Hadoop configurations

On the client you can set Hadoop configurations using the spark.conf.set API, which applies to SQL and DataFrame operations. Hadoop configurations set on the sparkContext must be set in the cluster configuration or using a notebook. This is because configurations set on sparkContext are not tied to user sessions but apply to the entire cluster.

Troubleshooting

This section describes some common issues that you might encounter with Databricks Connect and how to resolve them.

In this section:

Error: StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status 500

Issue: When you try to run code with Databricks Connect, you get an error messages that contains strings such as StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status: 500.

Possible cause: Databricks Connect cannot reach your cluster.

Recommended solutions:

  • Check to make sure that your workspace instance name is correct. If you use environment variables, check to make sure the related environment variable is available and correct on your local development machine.
  • Check to make sure that your cluster ID is correct. If you use environment variables, check to make sure the related environment variable is available and correct on your local development machine.
  • Check to make sure that your cluster has the correct custom cluster version that is compatible with Databricks Connect.

Python version mismatch

Check the Python version you are using locally has at least the same minor release as the version on the cluster (for example, 3.10.11 versus 3.10.10 is OK, 3.10 versus 3.9 is not).

If you have multiple Python versions installed locally, ensure that Databricks Connect is using the right one by setting the PYSPARK_PYTHON environment variable (for example, PYSPARK_PYTHON=python3).

Conflicting PySpark installations

The databricks-connect package conflicts with PySpark. Having both installed will cause errors when initializing the Spark context in Python. This can manifest in several ways, including “stream corrupted” or “class not found” errors. If you have PySpark installed in your Python environment, ensure it is uninstalled before installing databricks-connect. After uninstalling PySpark, make sure to fully re-install the Databricks Connect package:

pip3 uninstall pyspark
pip3 uninstall databricks-connect
pip3 install --upgrade "databricks-connect==13.1.*"  # or X.Y.* to match your specific cluster version.

Conflicting or Missing PATH entry for binaries

It is possible your PATH is configured so that commands like spark-shell will be running some other previously installed binary instead of the one provided with Databricks Connect. You should make sure either the Databricks Connect binaries take precedence, or remove the previously installed ones.

If you can’t run commands like spark-shell, it is also possible your PATH was not automatically set up by pip3 install and you’ll need to add the installation bin dir to your PATH manually. It’s possible to use Databricks Connect with IDEs even if this isn’t set up.

The filename, directory name, or volume label syntax is incorrect on Windows

If you are using Databricks Connect on Windows and see:

The filename, directory name, or volume label syntax is incorrect.

Databricks Connect was installed into a directory with a space in your path. You can work around this by either installing into a directory path without spaces, or configuring your path using the short name form.

Limitations

Databricks Connect does not support the following Azure Databricks features and third-party platforms.

PySpark DataFrame API limitations

  • The SparkContext class and its methods are not available.
  • Creating DataFrames larger than 128 MB from local data is not supported and will fail.
  • Resilient distributed datasets (RDDs) and Datasets are not supported. Only DataFrames are supported.

Azure Databricks and Databricks Connect limitations

  • Queries with a duration longer than 3600 seconds are not supported and will fail.
  • Synchronizing the local development environment with the remote cluster is not supported.
  • Make sure that the Python version and any Python packages that you use in your local development environment match their equivalents that are installed on the cluster as much as possible, to help ensure code compatiblity and to help reduce unexpected run time errors.
  • Only Python is supported. R, Scala, and Java are not supported.
  • Distributed training is not supported.
  • MLflow is supported, but not model inference with mlflow.pyfunc.spark_udf(spark, ...). You can load the model locally with mlflow.pyfunc.load_model(<model>), or you can wrap it as a custom Pandas UDF.
  • You cannot change the Log4j log level through SparkContext.
  • Mosaic is not supported.

Azure Databricks cluster limitations

  • The Azure Databricks workspace user who is associated with an access token that is used by Databricks Connect must have Can Attach To permissions or greater for the target cluster.

  • CREATE TABLE table AS SELECT ... SQL commands do not always work. Instead, use spark.sql("SELECT ...").write.saveAsTable("table").

  • Azure Active Directory credential passthrough is supported only on standard clusters running Databricks Runtime 7.3 LTS and above, and is not compatible with service principal authentication.

  • Azure Active Directory (AD) tokens are not automatically refreshed and will expire one hour after they are initially generated.

  • The following Databricks Utilities: