Clusters
An Azure Databricks cluster is a set of computation resources and configurations on which you run data engineering, data science, and data analytics workloads, such as production ETL pipelines, streaming analytics, ad-hoc analytics, and machine learning.
You run these workloads as a set of commands in a notebook or as an automated job. Azure Databricks makes a distinction between all-purpose clusters and job clusters. You use all-purpose clusters to analyze data collaboratively using interactive notebooks. You use job clusters to run fast and robust automated jobs.
- You can create an all-purpose cluster using the UI, CLI, or REST API. You can manually terminate and restart an all-purpose cluster. Multiple users can share such clusters to do collaborative interactive analysis.
- The Azure Databricks job scheduler creates a job cluster when you run a job on a new job cluster and terminates the cluster when the job is complete. You cannot restart a job cluster.
This section describes how to work with clusters using the UI. For other methods, see Clusters CLI and Clusters API 2.0.
This section also focuses more on all-purpose than job clusters, although many of the configurations and management tools described apply equally to both cluster types. To learn more about creating job clusters, see Create, run, and manage Azure Databricks Jobs.
Important
Azure Databricks retains cluster configuration information for up to 200 all-purpose clusters terminated in the last 30 days and up to 30 job clusters recently terminated by the job scheduler. To keep an all-purpose cluster configuration even after it has been terminated for more than 30 days, an administrator can pin a cluster to the cluster list.
In this section:
- Databricks Clusters UI changes
- Create a cluster
- Manage clusters
- Display clusters
- Pin a cluster
- View a cluster configuration as a JSON file
- Edit a cluster
- Clone a cluster
- Control access to clusters
- Start a cluster
- Terminate a cluster
- Delete a cluster
- Restart a cluster to update it with the latest images
- View cluster information in the Apache Spark UI
- View cluster logs
- Monitor performance
- Decommission spot instances
- Configure clusters
- Cluster policy
- Cluster mode
- Data lineage
- Pools
- Databricks Runtime
- Cluster node type
- Cluster size and autoscaling
- Autoscaling local storage
- Local disk encryption
- Security mode
- Spark configuration
- Retrieve a Spark configuration property from a secret
- Environment variables
- Cluster tags
- SSH access to clusters
- Cluster log delivery
- Init scripts
- Best practices: Cluster configuration
- Task preemption
- Customize containers with Databricks Container Services
- Cluster node initialization scripts
- GPU-enabled clusters
- Single Node clusters
- Pools
- Web terminal
- Debugging with the Apache Spark UI
Feedback
Submit and view feedback for