Use serverless SQL warehouses
This article explains how to use serverless SQL warehouses and how to enable them for your account.
With serverless compute, the compute layer exists in your Azure Databricks account rather than your Azure subscription. This gives users in your account instant access to fully managed and elastic compute resources. Serverless compute is supported for use with Databricks SQL. Serverless compute does not affect how Databricks Runtime clusters work with notebooks and jobs.
Serverless SQL warehouses do not have public IP addresses. For more architectural information, see Serverless compute.
Important
Tasks on this page require that you have owner or contributor permissions on the Azure Databricks workspace.
Requirements
- Your Azure Databricks workspace must be on the Premium tier.
- Your workspace must be in a region that supports Databricks SQL Serverless.
Serverless quotas
Serverless quotas are a safety measure for serverless compute. Serverless quotas restrict how many serverless compute resources a customer can have at any given time. The quota is enforced at the regional level for all workspaces in your account. Quotas are enforced only for serverless SQL warehouses. See Serverless quotas.
Configure Azure storage firewalls, if applicable
If you use an Azure Storage firewall to protect access to Azure storage data sources, you must configure your firewall to allow access from the serverless compute nodes. See Configure Azure storage firewalls to allow access from serverless SQL warehouses.
Create a serverless SQL warehouse
If your workspace is enabled for Databricks SQL Serverless, new SQL warehouses are serverless by default. The workspace must be in a supported region for serverless SQL warehouses and on the Premium pricing tier. For other requirements, see Requirements.
You can also upgrade an existing pro or classic SQL warehouse to serverless.
If you use the SQL Warehouse API to create the warehouse, serverless is not the default. Databricks recommends always explicitly setting enable_serverless_compute to true and warehouse_type to pro.
Auto-restart of serverless SQL warehouses
A serverless SQL warehouse auto-restarts in the following conditions:
- You attempt to run a query that uses a stopped warehouse.
- A job assigned to a stopped warehouse is scheduled to run.
- You connect to a stopped warehouse from a JDBC/ODBC interface.
- You open a query in the SQL editor that is saved to a stopped warehouse.
- A dashboard is opened that is saved with a dashboard-level warehouse assigned.
Limitations
Serverless warehouses have the following limitations:
- External Hive legacy metastores are not supported.
- Cluster policies, including spot instance policies are not supported.
- The serverless data plane for serverless SQL warehouses does not use the customer-configurable back-end Azure Private Link connectivity. The Azure Databricks control plane connects to the serverless data plane with mTLS with IP access allowed only for the control plane IP address.
- Although the serverless data plane does not use the secure cluster connectivity relay that is used for the classic data plane, serverless SQL warehouses do not have public IP addresses.
- VNet Injection is not applicable.
- Serverless SQL warehouses do not use customer-managed keys for managed disks. The disks used for serverless compute resources are short-lived and tied to the lifecycle of the serverless workload. For example, when serverless SQL warehouses are stopped or scaled down, the VMs and their storage are destroyed. See Serverless compute and customer-managed keys.
Feedback
Submit and view feedback for