Serverless compute

Important

Serverless SQL warehouses are available in Public Preview. To start using Databricks SQL Serverless on Azure, please submit your request using this form.

With the Serverless compute version of the Databricks platform architecture, the compute layer exists in the Azure subscription of Azure Databricks rather than the customer’s Azure subscription.

As of the current release, Serverless compute is supported for use with Databricks SQL. Admins can create Serverless SQL warehouses (formerly SQL endpoints) that enable instant compute and are managed by Azure Databricks. Serverless SQL warehouses use compute clusters in the Azure subscription of Azure Databricks. Use them with Databricks SQL queries just like you normally would with the original customer-hosted SQL warehouses, which are now called Classic SQL warehouses.

Databricks changed the name from SQL endpoint to SQL warehouse because, in the industry, endpoint refers to either a remote computing device that communicates with a network that it’s connected to, or an entry point to a cloud service. A data warehouse is a data management system that stores current and historical data from multiple sources in a business friendly manner for easier insights and reporting. SQL warehouse accurately describes the full capabilities of this compute resource.

Before you can create Serverless SQL warehouses, you must enable Serverless Databricks SQL warehouses for your workspace. If Serverless SQL warehouses are enabled for your workspace:

Azure Databricks supports Serverless SQL warehouses in Azure region East US (eastus), East US 2 (eastus2), and West Europe (westeurope).

Compare Serverless compute to other Databricks architectures

Azure Databricks operates out of a control plane and a data plane:

  • The control plane includes the backend services that Azure Databricks manages in its own Azure subscription. Databricks SQL queries, notebook commands, and many other workspace configurations are stored in the control plane and encrypted at rest.
  • The data plane is where data is processed by clusters of compute resources.

There are important differences between the Classic data plane (the original Azure Databricks platform architecture) and the Serverless data plane:

  • For a Classic data plane, Azure Databricks compute resources run in the customer’s Azure subscription. Clusters perform distributed data analysis using queries (in Databricks SQL) or notebooks (in the Data Science & Engineering or Databricks Machine Learning environments):
    • New clusters are created within each workspace’s virtual network in the customer’s Azure subscription.
    • A Classic data plane has natural isolation because it runs in each customer’s own Azure subscription. A Classic data plane is not a shared resource for multiple customers.
  • For a Serverless data plane, Azure Databricks compute resources run in a special compute layer within the Azure subscription for Azure Databricks:
    • As of this release, the Serverless data plane is used only for Serverless SQL warehouses. Enabling this feature does not change how Databricks Runtime clusters work in the Data Science & Engineering or Databricks Machine Learning environments.
    • The Serverless data plane is a shared resource in the Azure subscription of Azure Databricks for multiple Azure Databricks customers.
    • To protect customer data within the Serverless data plane, Serverless compute runs within a network boundary for the workspace, with various layers of security to isolate different Azure Databricks customer workspaces and additional network controls between clusters of the same customer.

Azure Databricks creates a Serverless data plane in the same Azure region as your workspace’s Classic data plane.

The following diagram shows important differences between the Serverless data plane and Classic data plane.

Compare Classic and Serverless data plane

For more information about secure cluster connectivity, which is mentioned in the diagram, see Secure cluster connectivity (No Public IP / NPIP).

The table below summarizes differences between Serverless compute and the Classic data plane architecture of Azure Databricks, focusing on product security. It is not a complete explanation of those security features or a detailed comparison. For more details about Serverless compute security, or if you have questions about items in this table, contact your Azure Databricks representative.

Item Serverless data plane for SQL warehouses on Azure Databricks Classic data plane on Azure Databricks
Virtual private network for data plane The VNet in the Azure Databricks subscription is shared among customers, with additional network boundaries between clusters. The VNet is in customer’s subscription
Customize virtual network and firewall settings No. Yes, if you use VNet injection
Customize CIDR ranges No. Yes, if you use VNet injection
Container-level network isolation for clusters and SQL warehouses Databricks managed software firewall. Databricks managed software firewall
Communication between control plane and data plane Direct Databricks-managed TLS encrypted communication using private connectivity, with connection initiated from the control plane. When secure cluster connectivity (SCC) is enabled, which is the default for Azure, individual VMs connect to the SCC relay in the control plane during cluster creation. When SCC is disabled, the control plane connects to individual VMs using public IPs.