Content
I can't view Data Lake Storage Gen1 resource in Storage Explorer
I can't view Data Lake Storage Gen1 resource in Storage Explorer
How do I extract only specific files from a compressed file using ADF?
I am working on a File Extraction pipeline. I have one compressed file (i.e., CompressedA) stored in one container which contains 5 separate files (i.e., File1-File5). In a scenario where user only requires File1, File2 and File5, how do I extract them…
Unable to access external tables in lake database with Synapse SQL admin user credentials from SSMS.
Using Azure Synapse Link, we were able to integrate Projects Operations environment with Azure Data Lake (https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse). Lake database is created in synapse workpsace. We can…
How do I read an xlsx in Azure Data Lake from Powershell
Hi, What would be the best way to read in an xlsx to Powershell from the Azure data lake. I have tried the ImportExcel module however it does not like the https path? Any help appreciated
Save Model Keras AZ Synapse
How to save a keras file using python Notebook in Azure Synapse?
How to use linked service in Notebook with pyspark
I have pyspark script in Notebook to read and write data in ADLS Gen2. Below is an sample of the pyspark script. But in the Synapse I only have a linked service created with Service Principle could connect to the ADLS Gen2, so I need to specify in…
Extracting Data with SOAP API using ADF
Hi, I am using ADF Copy Activity to copy data from SOAP API to data lake gen 2 storage in parquet format. The problem I am having is with copying some columns which have multiple values in the API response. for example here is an example of my API…
query to databricks cluster using postman
Hi Expert, can you help me with hostend point to URL to call Databricks compute cluster I used PAT and below URL but not working ad.azuredatabricks.net/api/2.0/clusters/pat_token
data ingestion architectural approach
What is the sound technical strategy and approach for data ingestion design? What are the important considerations?
azure synapse and datalake storage
Hi friends, in some of our projects I have seen data is stored in Azure Data Lake storage and then again in Azure Synapse analytics( gold layer). This is a duplication of data and is costly. Is it not correct and architecture is wrong as I think of it?…
Can we have blob storage only as central storage instead of ADLS
Hi friends, can we only have blob storage only as central storage instead of ADLS , did anybody have only blob storage in place of ADLS ? What are the challenges of having this architecture?
Azure Synapse Copy Error
We have Synapse ETL pipelines that just began failing this morning on a copy step in all of our environments and for all of our pipelines. The step worked previously and just began failing today. The copy step that is failing is just copying a parquet…
Creating database with location ADLS Gen 2 using abfss generates exception
Hello, I am creating a database in Azure Databricks using the abfss location in the create table statement and it throws an exception. %sql CREATE DATABASE IF NOT EXISTS test COMMENT "Database for Test Area" LOCATION…
Get MetaData Activity unable to read from Blob container
Greetings, I'm trying to read file names and content within multiple CSV files stored in a Blob container viz. 'past-members-contacts-csv', which is fetched via 'Get Metadata' activity in a Data Factory pipeline. Error Message: Field 'childItems'…
Compute Cluster cache
Hi Expert, How to cache compute cluster in databricks tried but not working as expected spark.databricks.io.cache.maxDiskUsage 50g spark.databricks.io.cache.maxMetaDataCache 1g spark.databricks.io.cache.compression.enabled false
Encryption
Hello, I have a requirement to bring the data from SQL database to Azure Datalake for Analytics and Reporting in Power BI. Source :Microsoft SQL database host externally. Destination : Azure Data Lake. I want to ingest data from the SQL table into Azure…
Data ingestion using copy data activity (from Oracle to ADLS GEN2)
Hello , I am using a query to ingest the data from Oracle to ADLS GEN2 (as a parquet file) using copy activity. The Query which I am using has a row count of 5.2B records. When I use the copy activity to run the script , it is running for more than…
How to Sync Data from Azure Data Lake Gen 2 to On prem Sql Data base
Hi All Experts, We configured Azure Data Lake Gen 2, It is syncing properly from D365. Now business user want view data in SQL or any other media using filter. We have a on prem Sql Server. Is it possible to sync Azure Data Lake data to on prem Sql? If…
ADF can access External Table in Synapse
I am trying to do a copy activity in ADF that reference an external table in Synape (which references a parquet file in Storage accountV2). This runs fine in Synapse however I get the following error which I run it through an ADF pipeline. Any help on…