Microsoft Q&A

Azure Databricks

1,455 questions

An Apache Spark-based analytics platform optimized for Azure.

Browse all Azure tags

1,455 questions with Azure Databricks tags

Sort by: Updated
0 answers

Databricks cluster failure : OutOfMemoryError

Getting a continous failures after a couple of days of stream runs with below error message, what could be potential permanent solution for the same ? Driver restarted - possibly due to an OutOfMemoryError - and this stream has been stopped. Please…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-11T17:03:22.0533333+00:00
NIKHIL KUMAR 81 Reputation points
edited a comment 2023-07-11T21:06:36.6666667+00:00
BhargavaGunnam-MSFT 15,561 Reputation points Microsoft Employee
2 answers One of the answers was accepted by the question author.

Tenant to Tenant cloning - Azure data components

Hello team, There is a requirement for us to clone Azure data components (Azure SQL DB, ADF, Azure Databricks, Azure Synapse Analytics) from one tenant to another tenant. There are a lot of resources that needs migration. Checking here to see if there…

Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,258 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-05T05:35:25.34+00:00
Nagesh CL 476 Reputation points
commented 2023-07-11T15:40:07.8466667+00:00
BhargavaGunnam-MSFT 15,561 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Do we need two Azure databricks workspace to configure private endpoints

According to the Documentation for Enable Azure Private Link as a simplified deployment for Databricks, we need two workspace. One for the actual workload and another just for browser authentication. The Actual Workspace has databricks_ui_api endpoint…

Azure Private Link
Azure Private Link
An Azure service that provides private connectivity from a virtual network to Azure platform as a service, customer-owned, or Microsoft partner services.
345 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-06-21T12:13:24.21+00:00
Vinayak Pote 45 Reputation points
accepted 2023-07-11T14:27:41.1233333+00:00
Vinayak Pote 45 Reputation points
1 answer

Azure Databricks -Any pointers for cost optimization ?

Hello, We would like to do cost optimization for Azure Databricks resources. Is there any pointer/white paper/ reference article /blog post that might be useful ? Applicate your Response. Thank you!!

Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
1,253 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-11T12:13:28.5066667+00:00
anil kumar 1,546 Reputation points
answered 2023-07-11T12:34:43.6933333+00:00
Rishikesh Singh 160 Reputation points
1 answer

Configured privacy settings disallow access for workspace <workspace-id> over your current network.

Hi All, We have a couple of Databricks workspaces that are deployed with secure access - basically you can only log in when you are within the network, However, I am running into a problem where I sometimes get this error: “Configured privacy settings…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-06-11T20:26:26.69+00:00
John Aherne 111 Reputation points
commented 2023-07-11T00:38:25.9133333+00:00
John Aherne 111 Reputation points
1 answer

I can't scan Databricks Delta lake and Notebook when using Purview.

When using purview, only Databricks' Workspace and Hive metastores are scanned, but Notebooks and Delta Lake are not scanned. What could be the cause of this problem? What should I do to solve this problem?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
Microsoft Purview
Microsoft Purview
A Microsoft data governance service that helps manage and govern on-premises, multicloud, and software-as-a-service data. Previously known as Azure Purview.
582 questions
asked 2023-07-09T22:46:23.8533333+00:00
김기만 0 Reputation points
commented 2023-07-11T00:24:04.7133333+00:00
김기만 0 Reputation points
1 answer

multiple json files load through api

Hi Expert, How to load multiple json files using API URL in blob or databricks

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,028 questions
Azure Database for MySQL
Azure Database for MySQL
An Azure managed MySQL database service for app development and deployment.
528 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,454 questions
asked 2023-07-07T08:16:47.0866667+00:00
Shambhu Rai 866 Reputation points
commented 2023-07-10T21:09:58.2633333+00:00
Shambhu Rai 866 Reputation points
0 answers

How to fix Azure databricks Server Error: Cache is Stale

Good morning, I have been working for several months on azure databricks without any issue, using notebooks stored in databricks. This morning, while working with Databricks notebooks stored within the platform, I suddenly experienced a complete…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-05T11:52:18.0533333+00:00
Gracia Espelt, Pol 0 Reputation points
commented 2023-07-10T20:13:12.7666667+00:00
BhargavaGunnam-MSFT 15,561 Reputation points Microsoft Employee
1 answer

Bringing SAP data into Databricks

HI, I am wondering if there is a way to bring SAP data into databricks? Does databricks provides connector for this purpose? Regards Rajaniesh

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2020-07-20T19:39:38.57+00:00
Rajaniesh Kaushikk 426 Reputation points
commented 2023-07-10T10:45:08.1966667+00:00
Nirav Murarka 1 Reputation point
2 answers One of the answers was accepted by the question author.

Support of Multiselect List in Azure Data Factory parameters

Hello Team, We are working on a scenario where we want to provide a capability to user to select some values (Multiple) from a predefined list of values. And as a result, the data factory should run only for the selected values and not for all the values…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,454 questions
asked 2023-07-07T05:36:31.2866667+00:00
Ravindra Raghuvanshi (Consultant) 20 Reputation points
accepted 2023-07-10T07:36:55.12+00:00
Ravindra Raghuvanshi (Consultant) 20 Reputation points
1 answer

How to access Azure Devops from Databricks using a Service Principal

Hi. I am trying to install a package (Flask==2.0.2) from my Azure Devops portal using Databricks. For this purpose I am using a PAT and passing this in the %pip install statement in databricks. Now I have created a Service principal and used the same in…

Azure Active Directory
Azure Active Directory
An Azure enterprise identity service that provides single sign-on and multi-factor authentication.
15,286 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-03T16:18:06.66+00:00
Kunal Chowdhury 0 Reputation points
commented 2023-07-10T06:07:55.38+00:00
Kunal Chowdhury 0 Reputation points
0 answers

How do we get the folder path of the file inside the workspace - Azure Databricks

I'm using the Databricks REST API to create the folder under the workspace, so that the folder will be accessed controlled as shown below URL - {{baseUrl}}/2.0/workspace/mkdirs Body - {"path":"/dde9540f/cert"} It's getting…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-06-29T21:36:03.5333333+00:00
Sharath 20 Reputation points
commented 2023-07-10T06:05:27.6733333+00:00
PRADEEPCHEEKATLA-MSFT 61,701 Reputation points Microsoft Employee
0 answers

Azure databricks unity catalog service principal

I am trying to assign permissions (Account admin) to service principal in unity catalog . but it's appearing WITHOUT name in Unity catalog--> User management . Only id and status is showing. name is blank. I am not able to edit or delete this from…

Azure Data Catalog
Azure Data Catalog
An Azure service that serves as a system of registration and system of discovery for enterprise data assets.
81 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-05T17:02:55.9266667+00:00
Azuretech 80 Reputation points
commented 2023-07-10T05:50:35.9233333+00:00
PRADEEPCHEEKATLA-MSFT 61,701 Reputation points Microsoft Employee
1 answer

Reading avro file in Databricks

How to read an .avro files stored in data lake using databricks.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-06T08:39:12.52+00:00
NIKHIL KUMAR 81 Reputation points
commented 2023-07-10T05:42:31.9033333+00:00
PRADEEPCHEEKATLA-MSFT 61,701 Reputation points Microsoft Employee
1 answer

suspecting wrong info by azure

In the document page ,it mentioned i can access databrciks in community https://docs.databricks.com/getting-started/community-edition.html, This article describes how to sign up for Databricks Community Edition. Unlike the Databricks Free Trial,…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-06T21:18:15.5866667+00:00
Amit Das 0 Reputation points
commented 2023-07-10T05:38:08.4933333+00:00
PRADEEPCHEEKATLA-MSFT 61,701 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Transition Azure Data Factory Mapping Data Flows to Azure Databricks (Scala?)

As a result of some infrastructure changes that are outside of my control, my organization is shifting from the use of Azure Data Factory (including Mapping Data Flows) to Azure Databricks. It is my understanding that ADF Mapping Data Flows functionally…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,454 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-06-13T17:23:24.64+00:00
Todd Lazure 46 Reputation points
accepted 2023-07-08T11:09:15.5333333+00:00
Todd Lazure 46 Reputation points
0 answers

CNAME records for westus.pl-auth.azuredatabricks.net for multiple Azure databricks workspace usign custom DNS

How to set up CNAME records for multiple Azure Databricks workspaces for Azure Active Directory OAuth flow reply URL, using Custom DNS and my databricks workspaces has backend private endpoint configured but not browser authentication private…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-06T09:39:42.8933333+00:00
Gaikwad, Ashwini 0 Reputation points
edited the question 2023-07-07T15:08:41.42+00:00
KranthiPakala-MSFT 39,907 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

azcopy upload file failed with error 403 using service principal

when i run azcopy upload files to storage account(subscription A) failed with error:403 using service principal from azure databrick's VM(subscription B, vent B, subnet B). i has granted storage blob contributor role to that service principal, also add…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
1,923 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-07-06T03:10:44.9333333+00:00
Winston Huang 20 Reputation points
accepted 2023-07-07T02:44:23.57+00:00
Winston Huang 20 Reputation points
1 answer

I am trying to connect using databricks sql but I am getting error using node js

I am trying to connect using databricks but I am getting error. Pls help to resolved it. Here is Error Screenshot. Here is Code Screenshot

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-01-02T06:15:25.683+00:00
Nisha Singh 1 Reputation point
commented 2023-07-06T14:39:59.3566667+00:00
Ed T Guy 0 Reputation points
3 answers

How to read append blob in (Azure Blob or ADLS) from Synapse or Databricks

Hi All I am trying to read a telemetry log , which is stored currently in blob storage without the hierarchical namespace enabled , when I try to read , I am getting the following error. All my files in storage account are currently saved as append blob,…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
1,923 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,258 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,455 questions
asked 2023-05-13T16:01:41.36+00:00
Bloody Data 0 Reputation points
answered 2023-07-06T13:42:31.1433333+00:00
Paul Grardel 0 Reputation points