Secure cluster connectivity (No Public IP / NPIP)

With secure cluster connectivity enabled, customer virtual networks have no open ports and Databricks Runtime cluster nodes have no public IP addresses. Secure cluster connectivity is also known as No Public IP (NPIP).

  • At a network level, each cluster initiates a connection to the control plane secure cluster connectivity relay during cluster creation. The cluster establishes this connection using port 443 (HTTPS) and uses a different IP address than is used for the Web application and REST API.
  • When the control plane logically starts new Databricks Runtime jobs or performs other cluster administration tasks, these requests are sent to the cluster through this tunnel.
  • The data plane (the VNet) has no open ports, and Databricks Runtime cluster nodes have no public IP addresses.

Benefits:

  • Easy network administration, with no need to configure ports on security groups or to configure network peering.
  • With enhanced security and simple network administration, information security teams can expedite approval of Databricks as a PaaS provider.

Note

All Azure Databricks network traffic between the data plane VNet and the Azure Databricks control plane goes across the Microsoft network backbone, not the public Internet. This is true even if secure cluster connectivity is disabled.

Important

This article mentions the term data plane, which is the compute layer of the Azure Databricks platform. In the context of this article, data plane refers to the Classic data plane in your Azure subscription. By contrast, the serverless data plane that supports serverless SQL warehouses runs in the Azure subscription of Azure Databricks. To learn more, see Serverless compute.

Secure cluster connectivity

Use secure cluster connectivity

To use secure cluster connectivity with a new Azure Databricks workspace, use any of the following options.

  • Azure Portal: When you provision the workspace, go to the Networking tab and set the option Deploy Azure Databricks workspace with Secure Cluster Connectivity (No Public IP) to Yes.
  • ARM Templates: For the Microsoft.Databricks/workspaces resource that creates your new workspace, set the enableNoPublicIp Boolean parameter to true.

Important

In either case, you must register the Azure Resource Provider Microsoft.ManagedIdentity in the Azure subscription that is used to launch workspaces with secure cluster connectivity. This is a one-time operation per subscription. For instructions, see Azure resource providers and types.

You cannot add secure cluster connectivity to an existing workspace. For information about migrating your resources to the new workspaces, contact your Microsoft or Databricks account team for details.

If you’re using ARM templates, add the parameter to one of the following templates, based on whether you want Azure Databricks to create a default (managed) virtual network for the workspace, or if you want to use your own virtual network, also known as VNet injection. VNet injection is an optional feature that allows you to provide your own VNet to host new Azure Databricks clusters.

Egress from workspace subnets

When you enable secure cluster connectivity, both of your workspace subnets are private subnets, since cluster nodes do not have public IP addresses.

The implementation details of network egress vary based on whether you use the default (managed) VNet or whether you use the optional VNet injection feature to provide your own VNet in which to deploy your workspace. See the following sections for details.

Important

Additional costs may be incurred due to increased egress traffic when you use secure cluster connectivity. For a smaller organization that needs a cost-optimized solution, it may be acceptable to disable secure cluster connectivity when you deploy your workspace. However, for the most secure deployment, Microsoft and Databricks strongly recommend that you enable secure cluster connectivity.

Egress with default (managed) VNet

If you use secure cluster connectivity with the default VNet that Azure Databricks creates, Azure Databricks automatically creates a NAT gateway for outbound traffic from your workspace’s subnets to the Azure backbone and public network. The NAT gateway is created within the managed resource group managed by Azure Databricks. You cannot modify this resource group or any resources provisioned within it.

The automatically-created NAT gateway incurs additional cost.

Egress with VNet injection

If you use secure cluster connectivity with optional VNet injection to provide your own VNet, ensure that your workspace has a stable egress public IP and choose one of the following options:

  • For simple deployments, choose an egress load balancer, also called an outbound load balancer. The load balancer’s configuration is managed by Azure Databricks. Clusters have a stable public IP, but you cannot modify the configuration for custom egress needs. This Azure template-only solution has the following requirements:
    • Azure Databricks expects additional fields to the ARM template that creates the workspace: loadBalancerName (load balancer name), loadBalancerBackendPoolName (load balancer backend pool name), loadBalancerFrontendConfigName (load balancer frontend configuration name) and loadBalancerPublicIpName (load balancer public IP name).
    • Azure Databricks expects the Microsoft.Databricks/workspaces resource to have parameters loadBalancerId (load balancer ID) and loadBalancerBackendPoolName (load balancer backend pool name).
    • Azure Databricks does not support changing the configuration of the load balancer.
  • For deployments that need some customization, choose an Azure NAT gateway. Configure the gateway on both of the workspace’s subnets to ensure that all outbound traffic to the Azure backbone and public network transits through it. Clusters have a stable egress public IP, and you can modify the configuration for custom egress needs. You can implement this solution using either an Azure template or from the Azure portal.
  • For deployments with complex routing requirements or deployments that use VNet injection with an egress firewall such as Azure Firewall or other custom networking architectures, you can use custom routes called user-defined routes (UDRs). UDRs ensure that network traffic is routed correctly for your workspace, either directly to the required endpoints or through an egress firewall. If you use such a solution, you must add direct routes or allowed firewall rules for the Azure Databricks secure cluster connectivity relay and other required endpoints listed at User-defined route settings for Azure Databricks.

Add secure cluster connectivity to an existing workspace

You can enable secure cluster connectivity on an existing workspace. The upgrade requires that the workspace uses VNet Injection.

You can use an ARM template or azurerm Terraform provider version 3.41.0+. You can use the Azure portal to apply a custom template and modify the parameter in the UI. However, there is no Azure Portal user interface support for this upgrade on the Azure Databricks workspace instance itself.

If something goes wrong with the upgrade and you need to temporarily rollback the change, see Temporary rollback of upgrading to secure cluster connectivity.

Step 1: Stop all compute resources

Before attempting this upgrade, you must stop all compute resources such as clusters, pools, or classic SQL warehouses. No workspace compute resources can be running or the upgrade attempt fails. Databricks recommends planning the timing of the upgrade for down time.

Step 2: Update the workspace

You must update the enableNoPublicIp parameter to the workspace with the value true.

Use one of these methods:

Apply an update ARM template using Azure portal

Note

If your managed resource group has a custom name, you must modify the template accordingly. Contact your Azure Databricks representative for more information.

  1. Copy the following upgrade ARM template JSON:

      {
        "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
        "contentVersion": "1.0.0.0",
        "parameters": {
            "location": {
                "defaultValue": "[resourceGroup().location]",
                "type": "String",
                "metadata": {
                    "description": "Location for all resources."
                }
            },
            "workspaceName": {
                "type": "String",
                "metadata": {
                    "description": "The name of the Azure Databricks workspace to create."
                }
            },
            "apiVersion": {
                "defaultValue": "2023-02-01",
                "allowedValues": [
                   "2018-04-01",
                   "2020-02-15",
                   "2022-04-01-preview",
                   "2023-02-01"
                ],
                "type": "String",
                "metadata": {
                    "description": "2018-03-15 for 'full region isolation control plane' and 2020-02-15 for 'FedRAMP certified' regions"
                }
            },
            "enableNoPublicIp": {
                "defaultValue": true,
                "type": "Bool"
            },
            "pricingTier": {
                "defaultValue": "premium",
                "allowedValues": [
                    "premium",
                    "standard",
                    "trial"
                ],
                "type": "String",
                "metadata": {
                    "description": "The pricing tier of workspace."
                }
            }
        },
        "variables": {
            "managedResourceGroupName": "[concat('databricks-rg-', parameters('workspaceName'), '-', uniqueString(parameters('workspaceName'), resourceGroup().id))]",
            "managedResourceGroupId": "[subscriptionResourceId('Microsoft.Resources/resourceGroups', variables('managedResourceGroupName'))]"
        },
        "resources": [
            {
                "type": "Microsoft.Databricks/workspaces",
                "apiVersion": "[parameters('apiVersion')]",
                "name": "[parameters('workspaceName')]",
                "location": "[parameters('location')]",
                "sku": {
                    "name": "[parameters('pricingTier')]"
                },
                "properties": {
                    "ManagedResourceGroupId": "[variables('managedResourceGroupId')]",
                    "publicNetworkAccess": "[parameters('publicNetworkAccess')]",
                    "requiredNsgRules": "[parameters('requiredNsgRules')]",
                    "parameters": {
                        "enableNoPublicIp": {
                            "value": "[parameters('enableNoPublicIp')]"
                        }
                    }
                }
            }
        ]
    }
    
    1. Go to the Azure portal Custom deployment page.

    2. Click Build your own template in the editor.

    3. Paste in the JSON for the template that you copied.

    4. Click Save.

    5. Fill in the parameters.

    6. To update an existing workspace, use the same parameters that you used to create the workspace other than enableNoPublicIp which you must set to true. Set the subscription, region, workspace name, subnet names, resource ID of the existing VNet.

      Important

      The resource group name, workspace name, and subnet names are identical to your existing workspace so that this command updates the existing workspace rather than creating a new workspace.

    7. Click Review + Create.

    8. If there are no validation issues, click Create.

      Note

      The network update may take over 10 minutes to complete.

Apply an update using Terraform

For workspaces created with Terraform, you can update the workspace without recreating the workspace.

Important

You must use terraform-provider-azurerm version 3.41.0 or later, so upgrade your Terraform provider version as needed. Earlier versions attempt to recreate the workspace if you change any of these settings.

  1. Change the following workspace settings:

    • no_public_ip in the custom_parameters block can be changed from false to true.

    Note

    The network update may take over 10 minutes to complete.

Step 3: Validate the update

Once the workspace is in active state, the update job is completed. Verify that the update was applied:

  1. Open Azure Databricks in your web browser.

  2. Start one of the workspace’s clusters and wait until the cluster is fully started.

  3. Go to your workspace instance in the Azure portal.

  4. Click the blue ID next to the field label Managed Resource Group.

  5. In that group, find the VMs for the cluster and click on one of them.

  6. In the VM settings, within Properties, look for the fields in the Networking area.

  7. Confirm that the Public IP address field is empty.

    If it’s populated, the VM has a public IP address, which means the update failed.

Failure recovery

If a workspace update fails, the workspace might be marked as a Failed state, which means that the workspace is unable to perform compute operations. To restore a failed workspace back to Active state, review the instructions in the status message of the update operation. Once you fix any issues, redo the update on the failed workspace. Repeat the steps until the update successfully completes. If you have questions, contact your Azure Databricks representative.

Temporary rollback of upgrading to secure cluster connectivity

If something goes wrong during deployment you can reverse the process as a temporary rollback, but disabling SCC on a workspace is unsupported other than for temporary rollback before continuing the upgrade later. If this is necessary temporarily, you can follow the instructions above for upgrade but set enableNoPublicIp to false instead of true. Contact your Azure Databricks representative if you have questions.