11,021 questions with Azure Data Factory tags

Sort by: Updated
4 answers

Read WebApp connection string credentials from Keyvault

I have hosted an ASP.Net web form webapp in Azure. Right now the password of the backend DB is being exposed in the connection string. Is there a way to read it through Azure Keyvault ? Please share your recommendations. Thanks in advance.

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2023-03-28T18:19:17.81+00:00
Viswanathan Madhavan 0 Reputation points
answered 2024-12-13T14:09:09.9366667+00:00
Eric Hill 0 Reputation points
0 answers

Incomplete data output from GET API ADF copy transformation

I am trying to copy data using Bing GET API in ADF, during the preview data stage the data looks fine. However when pushing it to sink as JSON, the output data is incomplete. Is there any steps i missed/ misconfigured? Configurations: Data…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-13T14:06:27.59+00:00
Chun Hanssen 25 Reputation points
2 answers One of the answers was accepted by the question author.

ADF - run an activity if any one of a set of activities fail

Is there a way in which I can run one generic activity (say log error), when any of the activities in the pipeline fail ? Thanks, Kothai

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2020-12-23T12:19:00.21+00:00
Kothai Ramanathan 941 Reputation points Microsoft Employee
commented 2024-12-13T13:53:10.4533333+00:00
Barry Smith 0 Reputation points
1 answer One of the answers was accepted by the question author.

How to Resolve Missing or Null Columns When Copying Data from Dataverse to Azure SQL Using Dataverse Connector?

I am currently working on a project where I need to copy data from a Dataverse table to an Azure SQL database using the Dataverse connector in Azure Data Factory (ADF). However, I am encountering the following issues: Some columns from the Dataverse…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-12T07:29:40.3233333+00:00
Akshay Patel 65 Reputation points
edited a comment 2024-12-13T12:53:35.87+00:00
Smaran Thoomu 18,385 Reputation points Microsoft Vendor
1 answer

Intermittent Issue with Lookup Activity in ADF Fetching Data from Snowflake

I'm encountering an intermittent issue with the Lookup activity in Azure Data Factory (ADF) when working with a Snowflake dataset. My setup involves an admin table in Snowflake where I store details of multiple tables. When I try to fetch all rows using…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-09T14:10:44.4766667+00:00
Ankit Mishra 0 Reputation points
commented 2024-12-13T12:39:16.2633333+00:00
Chandra Boorla 4,845 Reputation points Microsoft Vendor
0 answers

Dataflow not using the Base and Relative URLs in Dataset

I've set Base and Relative URLs in REST Linked Service and Dataset. These are read correctly by the dataset, which connects successfully. However, Dataflows do not read the Base and Relative URLs correctly from Dataset / Linked Service (only reads the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-13T12:16:37.6866667+00:00
Chris Jones 0 Reputation points
edited the question 2024-12-13T12:21:47.0166667+00:00
Chris Jones 0 Reputation points
2 answers

Synchronizing Tables from Multiple Azure SQL Databases Using Azure Data Factory

Currently facing a challenge involving two different Azure SQL Servers, each containing 4-5 databases. The goal is to select specific tables from each database and synchronize them into a single Azure SQL Database to serve as a data warehouse for…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-11T11:12:32.5166667+00:00
Rajat Lalzare 0 Reputation points
commented 2024-12-13T12:07:47.2266667+00:00
Chandra Boorla 4,845 Reputation points Microsoft Vendor
0 answers

ADF data load

Hi friends, I have datalake by structure subject area ABC 2024 12 13 It is YYYY mm and dd partitioned created data lake, now I am trying to load the file in date 13 but I can only point it to the subject area,…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-13T10:56:20.6466667+00:00
azure_learner 525 Reputation points
1 answer

Is it possible to get User Assigned Managed Identity bearer token via ADF on MSIT tenant?

Can you please confirm if it is possible to do UAMI token generation via ADF on MSIT tenant? We do not have a PME subscription currently. Stackoverflow questions: https://stackoverflow.microsoft.com/questions/435481 …

Microsoft Identity Manager
Microsoft Identity Manager
A family of Microsoft products that manage a user's digital identity using identity synchronization, certificate management, and user provisioning.
723 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
22,535 questions
asked 2024-12-09T10:23:47.76+00:00
Mohammad Sheraj 0 Reputation points
answered 2024-12-13T08:50:10.4433333+00:00
Coder 0 Reputation points Microsoft Employee
1 answer

Data factory permissions to storage account

I can copy data from on premise database into data lake but when I try to add a dataflow to the pipeline I get this error I think the IAM is set up correctly I only have one resource group so it's pointing to the correct one I've also made data…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-12T18:03:40.0766667+00:00
jwilck 0 Reputation points
edited the question 2024-12-13T08:47:30.8166667+00:00
GitaraniSharma-MSFT 49,651 Reputation points Microsoft Employee
1 answer

How to write to two tables using pipelines

Hello I'm trying to achieve the following: Get data from one table using a stored procedure Compare this data to another table Write to 2 tables using a stored procedure It seems i can use dataflow to do 1&2 but not 3, and I can use pipeline to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-09T08:37:06.83+00:00
V 0 Reputation points
commented 2024-12-13T08:00:35.98+00:00
phemanth 12,325 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Analysis of Cost Saved

Hi, Current Solution: Delta table from HiveMetastore -> Staging Table with Business Logic in Databricks -> PowerBI Current Solution in Detailed: Main responsibilities for me is to consume the data the delta tables from hive_metastore in…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,272 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-11T05:47:30.34+00:00
Jayaswaroop Adabala 0 Reputation points
accepted 2024-12-13T06:50:15.1566667+00:00
Jayaswaroop Adabala 0 Reputation points
1 answer

How to transfer azure domain name one account to another account

Hello Teams, I want to transfer domain names from one account to another. How can I process this? I have been trying to figure this out since last week but haven’t found any proper documentation on Google or Microsoft. I found a document suggesting…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-09T18:14:44.0233333+00:00
Sk Nowlak 0 Reputation points
commented 2024-12-13T06:50:00.3533333+00:00
Chandra Boorla 4,845 Reputation points Microsoft Vendor
0 answers

ADF is unable to place logs inside Log analytics workspace.

Hello, I would like to bring your attention to an issue we are currently experiencing: We have an Azure Data Factory (ADF) setup and are trying to configure logging for it. A Log Analytics workspace has been created and attached to ADF. We set up…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-06T10:32:46.7033333+00:00
Sai Kumar Thadepalli 0 Reputation points
commented 2024-12-13T06:25:10.8666667+00:00
Sai Kumar Thadepalli 0 Reputation points
1 answer

Need to execute the java jar from ADF

I have jar file which perform certain operation. I need to call that from ADF , Is there any solution or way to execute the jar file from ADF?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-12T08:23:42.3066667+00:00
Banoth anil 0 Reputation points
commented 2024-12-13T06:04:15.32+00:00
Smaran Thoomu 18,385 Reputation points Microsoft Vendor
1 answer

Copy data from oracle to azure sql server

I have a use case where I need to copy the data from the Oracle database (on-prem) to the Azure SQL server. I need to upsert the data along with the timestamp column to the destination table which indicates the date a new record has been inserted or an…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-13T01:08:48.56+00:00
Keerthi Reddy Jagapathi 20 Reputation points
commented 2024-12-13T04:35:58.3266667+00:00
Keerthi Reddy Jagapathi 20 Reputation points
1 answer

The remote server returned an error: (400) Bad

Our PIPELINE file process was running perfectly, but couple of days ago it started failing below it is copying sql data to .parquet format on Hot Storage, please let me know what to check. Failure happened on 'Sink' side.…

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,988 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-13T00:16:11.2366667+00:00
Zaeemul Haque 45 Reputation points
answered 2024-12-13T03:38:27.8033333+00:00
Amrinder Singh 5,310 Reputation points Microsoft Employee
0 answers

Encountering 'Bad Request' Error Whenever Debugging Azure Simple Pipeline

I am trying to debug a simple Azure pipeline, but I consistently encounter the following error: "code": "BadRequest", "message": null, "target": "pipeline//runid/94168fdf-d49b-4ca8-9d01-dc17298b4b5d",…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,068 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-12-12T22:26:24.4766667+00:00
PHẠM NGỌC THÙY TRANG 0 Reputation points
edited the question 2024-12-13T01:37:20.9766667+00:00
Shireesha Eeraboina (Quadrant Resource LLC) 235 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

Azure Data Factory / Synapse: paging method for copy activity and data flow source work diffrently

Hi everyone, me again fighting with the REST sources in synapse. I finally discovered how paging methods work in a copy activity. This is the projection of the payload coming from the graph api: If more than 100 records in the value…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,068 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2022-02-11T15:34:48.583+00:00
Paul Hernandez 676 Reputation points Microsoft Employee
commented 2024-12-13T01:28:04.5766667+00:00
Seth Coleman 46 Reputation points
1 answer

Resultsets returned by script activity after switching to new SNF connector for ADF are not backward compatible - breaking pipelines

Hi MS support team, I am trying to upgrade Snowflake connector to V2 in my ADF pipelines. I have script activity which has multiple SQL commands (eg. set query_tag, call a stored procedure). Although these get executed successfully now, my subsequent IF…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,021 questions
asked 2024-09-24T11:31:00.4666667+00:00
Manjari Walia 5 Reputation points
commented 2024-12-13T01:12:02.9566667+00:00
Chandra Boorla 4,845 Reputation points Microsoft Vendor