What is Databricks SQL?

Databricks SQL describes the enterprise data warehouse built into the Azure Databricks Lakehouse Platform. The core offering of Databricks SQL is optimized compute called a SQL warehouse.

Azure Databricks provides a colllection of UI tools known as the SQL persona to compose and execute SQL queries, visualizations, and dashboards.

SQL warehouses provide general compute for SQL queries executed from many environments, including third party BI and visualization tools. Databricks SQL also provides a robust API.

Note

Databricks SQL is not available in Azure China regions.

To learn about the latest Databricks SQL features, see Databricks SQL release notes.

Spark SQL vs. Databricks SQL

Databricks SQL is an optimized compute environment, while Spark SQL describes a collection of Apache Spark APIs. Azure Databricks provides numerous tools and technologies that allow SQL developers to do work like ETL, analytics, and create dashboards. Databricks recommends Databricks SQL for an enterprise data warehouse experience.

What is Spark SQL?

The term Spark SQL technically applies to all operations that use Spark DataFrames. Spark SQL replaced the Spark RDD API in Spark 2.x, introducing support for SQL queries and the DataFrame API for Python, Scala, R, and Java.

The Azure Databricks documentation often uses Spark SQL to describe SQL queries and functionality. SQL queries run against Databricks Runtime compute map closely to functionality of open source Apache Spark, with some built-extension protocols for Delta Lake and proprietary Azure Databricks features. You can review all available functions and learn differences between execution contexts using the SQL reference.

How does SQL differ on Azure Databricks and open source Apache Spark?

Regardless of the execution envrionment, SQL on Azure Databricks does not differ greatly from Apache Spark SQL executed elsewhere. You can use SQL on Azure Databricks to execute common queries and statements against three different options for compute. All of these environments add a number of commands not available in open source Apache Spark and come pre-configured to run all the SQL queries associated with Delta Lake.

In Unity Catalog-enabled workspaces, a number of additional keywords and commands are present. Note that some SQL commands have different execution when run against a Unity Catalog catalog instead of the built-in Hive metastore.

Each of these compute options provides slightly different semantics and syntax. Generally speaking:

  • SQL on Databricks Runtime compute most closely resembles execution in open source Apache Spark.
  • SQL executed with Delta Live Tables generally inherits syntax and semantics from the Databricks Runtime, but adds a number of exclusive keywords and functions for Delta Live Tables.
  • SQL run on SQL warehouses abides most closely to ANSI standards. Many features supported by SQL run in Databricks notebooks against Databricks Runtime compute do not work, including HiveQL syntax, variable declaration and reference, and use of DButils widgets.

Databricks SQL administrative tasks

If you’re looking for information on enabling Databricks SQL, managing users and data access, and other administrative tasks, see Databricks SQL administration.