Enable the compliance security profile
If an Azure Databricks workspace has the compliance security profile enabled, the workspace has additional monitoring, a hardened compute image, and other features and controls. For details, see Features and technical controls.
The compliance security profile includes controls that help meet the applicable security requirements of PCI-DSS.
Enabling the compliance security profile is required to use Azure Databricks to process data that is regulated under the PCI-DSS compliance standard, but you can choose to enable the compliance security profile for its enhanced security features without the need to conform to PCI-DSS.
Important
Contact your Azure Databricks representative to enable the compliance security profile. You must specify the workspaces on which you want to enable the compliance security profile. If you create additional workspaces later, it is your responsibility to contact your Azure Databricks representative to request enablement of the compliance security profile on your new workspaces if needed.
If you enable this feature on any workspace, you’ll be charged for the Enhanced Security and Compliance add-on as described on the pricing page.
Which compute resources get enhanced security
The compliance security profile enhancements apply to compute resources in the classic data plane, such as clusters and non-serverless SQL warehouses. This applies in all regions.
For serverless SQL warehouses, when the compliance security profile is enabled, Azure Databricks does not allow starting serverless SQL warehouses.
Requirements
- Your Azure Databricks workspace is on the Premium pricing tier.
- Most Azure instance types are supported, but generation 2 (Gen2) and Arm64-based virtual machines are unsupported. If you try to launch a cluster with those instance types, the cluster fails to start if the compliance security profile is enabled.
Enable the compliance security profile
Prepare any existing workspaces that will use the compliance security profile. See Prepare a workspace for the compliance security profile.
Contact your Azure Databricks representative and request enabling the compliance security profile.
Send the list of workspace IDs for the workspaces on which to enable the profile. Get a workspace ID from the URL when you are using the workspace. The URL format is
adb-<workspace-id>.<random-number>.azuredatabricks.net, your workspace ID is the part that says<workspace-id>.Wait for confirmation from Azure Databricks that the profile is now enabled.
Important
After you receive notification that the profile is enabled, settings may take up to an additional six hours to propagate to all environments, including global regions and to downstream systems like billing. Workloads that are actively running continue with the settings that were active at the time of starting the cluster or other compute resource, and new settings will start applying the next time these workloads are started. This means that if a change is made late in the day or on the last day of the month, you might still see usage reported with the old settings early the next day or month. Please plan accordingly.
If any clusters or SQL warehouses were running, restart them. If you have many clusters running and only want to restart the ones that were started before enablement, you can use a script that Azure Databricks provides to determine if the start time was before the enablement date.
Now that profile setup is complete, create or use Azure Databricks compute resources as desired.
Long-running clusters
If you enable the compliance security profile for your account or your workspace, long-running clusters automatically restart after 25 days by default. Databricks recommends that workspace admins regularly restart clusters before they run for 25 days and do so during a scheduled maintenance window. This reduces the risk of an auto-restart disrupting a scheduled job.
If you want to restart long running clusters manually, you can use a script that Azure Databricks provides that can determine how long your clusters have been running, and optionally restart them. See Notebook example: Find long-running clusters.
Prepare a workspace for the compliance security profile
Some steps are necessary to prepare a workspace for the compliance security profile. If you have not yet enabled the security profile, follow these steps before requesting to enable the security profile.
If the security profile is already enabled for some workspaces and you create new workspaces for which you at a later time want to enable the compliance security profile, you must follow these steps after creating the workspaces.
- If the workspace is configured to restrict outbound network access, you must configure your network to allow traffic to specific ports. Configure your firewall, security group, or network appliance to allow ports outbound connections on port 443, 2443, and 6666. For related information, see Deploy Azure Databricks in your Azure virtual network (VNet injection).
- For every workspace that uses the profile, run the following tests to verify that the changes were correctly applied:
- Launch a Databricks cluster with one driver, one worker, any DBR version, and any supported instance type.
- Confirm that starting the cluster eventually enters the Running state.
Features and technical controls
The main enhancements of the compliance security profile affect the Azure Databricks compute resources in your workspace’s classic data plane.
Enhancements include:
- An enhanced disk image (a CIS-hardened Ubuntu Advantage worker image).
- Clusters automatically restart after 25 days and get the latest image with the latest security updates. If you enable the compliance security profile for your account or your workspace, long-running clusters are automatically restarted after 25 days.
- Security monitoring agents that generate logs that you can review. Two monitor agents run on compute resources (cluster workers) in your workspace’s classic data plane in your Azure subscription. This applies to clusters for notebooks and jobs, as well disk images that are used for pro or classic SQL warehouses, but not those used for serverless SQL warehouses.
- Communications within the cluster and for egress use TLS 1.2 encryption or higher, including connecting to the metastore.
- Clusters are limited to the versions that the compliance security profile supports. Azure Databricks limits the Databricks Runtime versions in the UI, and does not allow API requests for unsupported Databricks Runtime versions. Supported versions are Databricks Runtime 7.3 LTS and above.
- A shield logo appears in the top-right of the page, just to the left of the workspace name. See Confirm that the compliance security profile is enabled for a workspace.
Azure Databricks runs two monitoring agents in the data plane:
- Antivirus
- File integrity monitoring
See Monitoring agents in Azure Databricks compute images.
Disk image with enhanced hardening
While the compliance security profile is enabled, Databricks compute resources (cluster worker images) in your classic data plane use an enhanced hardened operating system image based on Ubuntu Advantage.
Ubuntu Advantage is a package of enterprise security and support for open source infrastructure and applications that includes a CIS Level 1 hardened image.
Monitoring agents in Azure Databricks compute images
While the compliance security profile is enabled, there are additional security monitoring agents, including two agents that are pre-installed in the images that are used for Azure Databricks compute resource VMs. You cannot disable the monitoring agents that are in the enhanced disk image.
| Monitoring agent | Description | How to get output |
|---|---|---|
| File integrity monitoring | Monitors for file integrity and security boundary violations. This monitor agent runs on the worker VM in your cluster. | Configure diagnostic logs and review logs for new rows. |
| Antivirus and malware detection | Scans the filesystem for viruses daily. This monitor agent runs on the VMs in your compute resources such as clusters and pro or classic SQL warehouses. The antivirus and malware detection agent scans the entire host OS filesystem and the Databricks Runtime container filesystem. Anything outside the cluster VMs is outside of its scanning scope. | Configure diagnostic logs and review logs for new rows. |
| Vulnerability scanning | Scans the container host (VM) for certain known vulnerabilities and CVEs. The scanning happens in representative images in the Azure Databricks environments. | Request scan reports on the image from your Azure Databricks representative. |
File integrity monitoring
The data plane image includes a file integrity monitoring service that provides runtime visibility and threat detection for compute resources (cluster workers) in the classic data plane in your account.
The file integrity monitor output is generated within diagnostic logs. Configure diagnostic logs. For the JSON schema for new auditable events that are specific to file integrity monitoring, see Audit log schemas for security monitoring.
Important
It is your responsibility to review antivirus monitor logs. Databricks may, in its sole discretion, review these logs but does not make a commitment to do so. If the agent detects a malicious activity, it is your responsibility to triage these events and open a support ticket with Databricks if the resolution or remediation requires an action by Databricks. Databricks may take action on the basis of these logs, including suspending or terminating the resources, but does not make any commitment to do so.
Antivirus and malware detection
The enhanced data plane image includes an antivirus engine for detecting trojans, viruses, malware, and other malicious threats. The antivirus monitor scans the entire host OS filesystem and the Databricks Runtime container filesystem. Anything outside the cluster VMs is outside of its scanning scope.
The antivirus monitor output is generated within diagnostic logs. For the JSON schema for new auditable events that are specific to antivirus monitoring, see Schema for antivirus monitoring.
When a new virtual machine image is built, updated signature files are included within it.
Important
It is your responsibility to review antivirus monitor logs. Databricks may, in its sole discretion, review these logs but does not make a commitment to do so. If the agent detects a malicious activity, it is your responsibility to triage these events and open a support ticket with Databricks if the resolution or remediation requires an action by Databricks. Databricks may take action on the basis of these logs, including suspending or terminating the resources, but does not make any commitment to do so.
When a new AMI image is built, updated signature files are included within the new AMI image.
Vulnerability scanning
A vulnerability monitor agent performs vulnerability scans of the container host (VM) for certain known CVEs.
Important
The scanning happens in representative images in the Azure Databricks environments.
You can request the vulnerability scan reports from your Azure Databricks representative.
When vulnerabilities are found with this agent, Databricks tracks them against its Vulnerability Management SLA and releases an updated image when available. It is your responsibility to restart all compute resources regularly to keep the image up-to-date with the latest image version.
Management and upgrade of monitoring agents
The additional monitoring agents that are on the disk images used for the compute resources in the Classic data plane are part of the standard Azure Databricks process for upgrading systems:
- The classic data plane base disk image (AMI) is owned, managed, and patched by Databricks.
- Databricks delivers and applies security patches by releasing new AMI disk images. The delivery schedule depends on new functionality and the SLA for discovered vulnerabilities. Typical delivery is every two to four weeks.
- The base operating system for the data plane is Ubuntu Advantage.
- Azure Databricks clusters and pro or classic SQL warehouses are ephemeral by default. Upon launch, clusters and pro or classic SQL warehouses use the latest available base image. Older versions that may have security vulnerabilities are unavailable for new clusters.
- You are responsible for restarting clusters (using the UI or API) regularly to ensure they use the latest patched host VM images.
- Databricks can, upon request, share an Azure Databricks notebook to identify your workspace’s running clusters and hosts older than a specified number of days and optionally, restart a cluster.
Monitor agent termination
If a monitor agent on the worker VM is found to be not running due to crash or other termination, the system will attempt to restart the agent.
Data retention policy for monitor agent data
Monitoring logs are sent to your own storage in your Azure subscription that you provided when you configured diagnostic logs. Retention, ingestion, and analysis of these logs is your responsibility.
Vulnerability scanning reports and logs are retained for at least one year by Databricks. You can request the vulnerability reports from your Azure Databricks representative.
Confirm that the compliance security profile is enabled for a workspace
To confirm that a workspace is using the compliance security profile, check that it has the yellow shield logo displayed in the user interface.
A shield logo appears in the top-right of the page, to the left of the workspace name:

If you click the workspace name, a menu shows a list of the workspaces that you have access to. The workspaces that enable the compliance security profile have a shield icon followed by the text “Compliance security profile”.

Important
If the shield icons are missing for a workspace with the compliance security profile enabled, contact your Azure Databricks representative.
Check whether any existing clusters need to be restarted after enablement
After a workspace is enabled with the compliance security profile, you need to restart any clusters that were created before the time of enablement to ensure it is using the compliance security profile enhancements and controls.
If you have many clusters running and only want to restart the ones that were started before enablement, you can use this script to determine if the start time was before the enablement date. Given a workspace URL, a personal access token for access REST APIs on this workspace, and an enablement date/time, this script returns a list of clusters that were started and/or restarted before the enablement timestamp. The script prints the cluster ID and the cluster name.
import requests
import json
# This notebook requires a user-level Personal Access Token. This should be stored
# in the Databricks Secrets API (or similar) and shouldn't be hardcoded in a notebook.
# Add a secret using the Databricks CLI or API. CLI example:
# $ databricks secrets create-scope --scope YOUR_SCOPE_NAME
# $ databricks secrets put --scope splunk_env --key YOUR_KEY_NAME
# Configure your scope and key name below.
#====== UPDATE THE FOLLOWING BELOW
WORKSPACE_URL="<WORKSPACE-URL-HERE>"
TOKEN=dbutils.secrets.get(scope="YOUR_SCOPE_NAME", key="YOUR_KEY_NAME")
# One of the below should be configured:
WORKSPACE_ENABLEMENT_TIME_UTC_MILLIS=<TIME-IN-UTC> # note millis, e.g. 1651366230000
WORKSPACE_ENABLEMENT_TIME_FORMATTED=None # Format YYYY-MM-DD HH:MM:SS -0000
# Example "2022-06-01 15:01:01 -0700"
#====== UPDATE THE ABOVE
if WORKSPACE_ENABLEMENT_TIME_FORMATTED != None:
WORKSPACE_ENABLEMENT_TIME_UTC_MILLIS=datetime.strptime(
WORKSPACE_ENABLEMENT_TIME_FORMATTED,
"%Y-%m-%d %H:%M:%S %z").timestamp()*1000
headers = {
'Authorization': 'Bearer ' + TOKEN
}
url = WORKSPACE_URL + "/api/2.0/clusters/list"
response = requests.request("GET", url, headers=headers, data={})
clusters = json.loads(response.text)["clusters"]
need_restart = []
for c in clusters:
start_time = c["start_time"]
last_start = start_time
if "last_restarted_time" in c:
last_start = max(start_time, c["last_restarted_time"])
if last_start <= WORKSPACE_ENABLEMENT_TIME_UTC_MILLIS:
need_restart.append((c["cluster_id"], c["cluster_name"]))
if (len(need_restart) == 0):
print("All clusters have been restarted since {}".format(WORKSPACE_ENABLEMENT_TIME_UTC_MILLIS))
else:
print("The following clusters still need to be restarted to remain in compliance")
for (id, name) in need_restart:
print("Cluster {}, {}".format(id, name))
Feedback
Submit and view feedback for