Enable serverless SQL warehouses

Important

Serverless SQL warehouses are available in Public Preview.

With the serverless compute version of the Databricks platform architecture, the compute layer exists in the Databricks cloud subscription rather than the customer’s cloud subscription. Serverless compute is supported for use with Databricks SQL. Admins can create serverless SQL warehouses that enable instant compute and are managed by Databricks. Serverless SQL warehouses use compute clusters in the customer’s Databricks account. Use serverless SQL warehouses with Databricks SQL queries the same way you would with the original customer-hosted SQL warehouses, which are now called classic SQL warehouses.

If serverless SQL warehouses are enabled for your workspace:

This feature only affects Databricks SQL. It does not affect how Databricks Runtime clusters work with notebooks and jobs in the Data Science & Engineering or Databricks Machine Learning workspace environments.

Databricks Runtime clusters always run in the classic data plane in your Azure subscription.

Serverless SQL warehouses do not have public IP addresses. For more architectural information, see Serverless compute.

Important

The following procedure requires that you have owner or contributor permissions on the Azure Databricks workspace.

Databricks changed the name from SQL endpoint to SQL warehouse because, in the industry, endpoint refers to either a remote computing device that communicates with a network that it’s connected to, or an entry point to a cloud service. A data warehouse is a data management system that stores current and historical data from multiple sources in a business friendly manner for easier insights and reporting. SQL warehouse accurately describes the full capabilities of this compute resource.

Requirements

For a list of regions that support serverless SQL warehouses, see Azure Databricks regions.

Also note that the Azure Databricks documentation on cluster size instance types and CPU quotas applies only to pro and classic SQL warehouses, not to serverless SQL warehouses.

Step 1: Enable serverless SQL warehouses for your workspace

  1. As an Azure Databricks workspace administrator, go to the admin console.

  2. Click the SQL Warehouse Settings tab.

  3. Select Serverless SQL Warehouses.

    Enable Serverless Compute

  4. Click Save changes.

Step 2 (optional): Configure Azure storage firewalls

If you use an Azure Storage firewall to protect access to Azure storage data sources, you must configure your firewall to allow access from the serverless compute nodes. See Configure Azure storage firewalls to allow access from serverless SQL warehouses.

Step 3: Test usage of serverless SQL warehouses

  1. Create or convert a warehouse:
  2. Run a query with your new serverless SQL warehouse.