Microsoft Q&A

Azure Data Lake Storage

1,103 questions

An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.

Browse all Azure tags

1,103 questions with Azure Data Lake Storage tags

Sort by: Updated
1 answer One of the answers was accepted by the question author.

I can't view Data Lake Storage Gen1 resource in Storage Explorer

I can't view Data Lake Storage Gen1 resource in Storage Explorer

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Storage Explorer
Azure Storage Explorer
An Azure tool that is used to manage cloud storage resources on Windows, macOS, and Linux.
187 questions
asked 2023-09-06T03:25:08.0766667+00:00
Yuming Xie 20 Reputation points Microsoft Employee
accepted 2023-09-07T12:44:22.1133333+00:00
Yuming Xie 20 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

How do I extract only specific files from a compressed file using ADF?

I am working on a File Extraction pipeline. I have one compressed file (i.e., CompressedA) stored in one container which contains 5 separate files (i.e., File1-File5). In a scenario where user only requires File1, File2 and File5, how do I extract them…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-09-06T07:37:24.6033333+00:00
Alfi Khair 20 Reputation points
commented 2023-09-07T06:44:56.5666667+00:00
Subashri Vasudevan 7,871 Reputation points
0 answers

Unable to access external tables in lake database with Synapse SQL admin user credentials from SSMS.

Using Azure Synapse Link, we were able to integrate Projects Operations environment with Azure Data Lake (https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse). Lake database is created in synapse workpsace. We can…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
asked 2023-09-06T05:34:07.3966667+00:00
Sivaram Talluri 0 Reputation points
commented 2023-09-06T07:55:20.2133333+00:00
AnnuKumari-MSFT 26,476 Reputation points Microsoft Employee
1 answer

I'm attempting to use Data Factory to transfer a CSV file from my computer to Azure Data Lake. I'm unable to select the path to the file I want to migrate from my local computer when I try to link the file system in the data factory's source tab.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-03-09T19:43:27.6566667+00:00
Abhijith Yamsani 0 Reputation points
edited a comment 2023-09-06T05:16:52.17+00:00
Gustavo Henrique Bavia 0 Reputation points
1 answer

How do I read an xlsx in Azure Data Lake from Powershell

Hi, What would be the best way to read in an xlsx to Powershell from the Azure data lake. I have tried the ImportExcel module however it does not like the https path? Any help appreciated

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Windows Server PowerShell
Windows Server PowerShell
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.PowerShell: A family of Microsoft task automation and configuration management frameworks consisting of a command-line shell and associated scripting language.
5,061 questions
PowerShell
PowerShell
A family of Microsoft task automation and configuration management frameworks consisting of a command-line shell and associated scripting language.
1,222 questions
asked 2023-02-07T12:02:34.6133333+00:00
Christopher Jack 0 Reputation points
commented 2023-09-04T09:07:35.7766667+00:00
Mugileeshwaran J S 0 Reputation points
1 answer

Save Model Keras AZ Synapse

How to save a keras file using python Notebook in Azure Synapse?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
asked 2023-08-29T17:14:55.3+00:00
Londono Quintero Ferney De Jesus 0 Reputation points
commented 2023-09-01T14:40:42.3166667+00:00
Londono Quintero Ferney De Jesus 0 Reputation points
2 answers One of the answers was accepted by the question author.

How to use linked service in Notebook with pyspark

I have pyspark script in Notebook to read and write data in ADLS Gen2. Below is an sample of the pyspark script. But in the Synapse I only have a linked service created with Service Principle could connect to the ADLS Gen2, so I need to specify in…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
asked 2023-03-31T04:24:40.7333333+00:00
Shao Peng Sun 81 Reputation points
commented 2023-09-01T11:22:29.34+00:00
Dom Sadie 0 Reputation points
1 answer

Extracting Data with SOAP API using ADF

Hi, I am using ADF Copy Activity to copy data from SOAP API to data lake gen 2 storage in parquet format. The problem I am having is with copying some columns which have multiple values in the API response. for example here is an example of my API…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-08-15T03:37:36.6466667+00:00
Asad Abid 0 Reputation points
commented 2023-09-01T06:11:44.2933333+00:00
KranthiPakala-MSFT 42,682 Reputation points Microsoft Employee
1 answer

query to databricks cluster using postman

Hi Expert, can you help me with hostend point to URL to call Databricks compute cluster I used PAT and below URL but not working ad.azuredatabricks.net/api/2.0/clusters/pat_token

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,565 questions
Azure Database for MySQL
Azure Database for MySQL
An Azure managed MySQL database service for app development and deployment.
560 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-08-14T13:30:43.0233333+00:00
Shambhu Rai 956 Reputation points
commented 2023-09-01T05:15:03.9733333+00:00
KranthiPakala-MSFT 42,682 Reputation points Microsoft Employee
2 answers One of the answers was accepted by the question author.

data ingestion architectural approach

What is the sound technical strategy and approach for data ingestion design? What are the important considerations?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-08-23T13:02:25.8233333+00:00
Anshal 1,161 Reputation points
accepted 2023-08-31T11:53:43.8933333+00:00
Anshal 1,161 Reputation points
2 answers One of the answers was accepted by the question author.

azure synapse and datalake storage

Hi friends, in some of our projects I have seen data is stored in Azure Data Lake storage and then again in Azure Synapse analytics( gold layer). This is a duplication of data and is costly. Is it not correct and architecture is wrong as I think of it?…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
asked 2023-08-24T16:33:46.1133333+00:00
Anshal 1,161 Reputation points
accepted 2023-08-30T15:25:50.55+00:00
Anshal 1,161 Reputation points
2 answers One of the answers was accepted by the question author.

Can we have blob storage only as central storage instead of ADLS

Hi friends, can we only have blob storage only as central storage instead of ADLS , did anybody have only blob storage in place of ADLS ? What are the challenges of having this architecture?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,907 questions
asked 2023-08-23T12:58:57.48+00:00
Anshal 1,161 Reputation points
accepted 2023-08-30T15:25:24.54+00:00
Anshal 1,161 Reputation points
2 answers One of the answers was accepted by the question author.

Azure Synapse Copy Error

We have Synapse ETL pipelines that just began failing this morning on a copy step in all of our environments and for all of our pipelines.  The step worked previously and just began failing today.  The copy step that is failing is just copying a parquet…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
542 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
asked 2023-08-16T15:10:47.4433333+00:00
Kevin Nowell 90 Reputation points
accepted 2023-08-29T11:09:42.52+00:00
Kevin Nowell 90 Reputation points
1 answer

Creating database with location ADLS Gen 2 using abfss generates exception

Hello, I am creating a database in Azure Databricks using the abfss location in the create table statement and it throws an exception. %sql CREATE DATABASE IF NOT EXISTS test COMMENT "Database for Test Area" LOCATION…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,565 questions
asked 2023-08-08T12:34:56.7033333+00:00
Mihai Cosmin 0 Reputation points
commented 2023-08-28T04:47:58.8266667+00:00
PRADEEPCHEEKATLA-MSFT 64,801 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Get MetaData Activity unable to read from Blob container

Greetings, I'm trying to read file names and content within multiple CSV files stored in a Blob container viz. 'past-members-contacts-csv', which is fetched via 'Get Metadata' activity in a Data Factory pipeline. Error Message: Field 'childItems'…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-08-26T07:36:40.4566667+00:00
Chandresh Johnson 20 Reputation points
commented 2023-08-27T21:58:48.6666667+00:00
Chandresh Johnson 20 Reputation points
1 answer One of the answers was accepted by the question author.

Compute Cluster cache

Hi Expert, How to cache compute cluster in databricks tried but not working as expected spark.databricks.io.cache.maxDiskUsage 50g spark.databricks.io.cache.maxMetaDataCache 1g spark.databricks.io.cache.compression.enabled false

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
357 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,565 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-08-24T12:42:49.7933333+00:00
Shambhu Rai 956 Reputation points
commented 2023-08-25T16:23:18.92+00:00
Shambhu Rai 956 Reputation points
2 answers

Encryption

Hello, I have a requirement to bring the data from SQL database to Azure Datalake for Analytics and Reporting in Power BI. Source :Microsoft SQL database host externally. Destination : Azure Data Lake. I want to ingest data from the SQL table into Azure…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
5,835 questions
SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
10,758 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-08-09T17:48:27.93+00:00
Sourav 0 Reputation points
edited an answer 2023-08-25T11:06:43.8466667+00:00
maither23 5 Reputation points
1 answer One of the answers was accepted by the question author.

Data ingestion using copy data activity (from Oracle to ADLS GEN2)

Hello , I am using a query to ingest the data from Oracle to ADLS GEN2 (as a parquet file) using copy activity. The Query which I am using has a row count of 5.2B records. When I use the copy activity to run the script , it is running for more than…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-07-28T07:47:47.67+00:00
Anandhakumar Cholendran 45 Reputation points
accepted 2023-08-25T08:00:11.9766667+00:00
Anandhakumar Cholendran 45 Reputation points
1 answer One of the answers was accepted by the question author.

How to Sync Data from Azure Data Lake Gen 2 to On prem Sql Data base

Hi All Experts, We configured Azure Data Lake Gen 2, It is syncing properly from D365. Now business user want view data in SQL or any other media using filter. We have a on prem Sql Server. Is it possible to sync Azure Data Lake data to on prem Sql? If…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
10,758 questions
asked 2023-08-24T07:24:29.23+00:00
Anshuman Chowdhury (NVS EXT) 20 Reputation points
accepted 2023-08-24T10:14:25.68+00:00
Anshuman Chowdhury (NVS EXT) 20 Reputation points
2 answers One of the answers was accepted by the question author.

ADF can access External Table in Synapse

I am trying to do a copy activity in ADF that reference an external table in Synape (which references a parquet file in Storage accountV2). This runs fine in Synapse however I get the following error which I run it through an ADF pipeline. Any help on…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,103 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,476 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,888 questions
asked 2023-08-17T05:18:40.9233333+00:00
Shobna Lata 20 Reputation points
accepted 2023-08-23T23:12:29.71+00:00
Shobna Lata 20 Reputation points