11,358 questions with Azure Data Factory tags

Sort by: Updated
0 answers

Is it possible to pass an array (of objects) as an input payload to External Call REST API Transformation in Dataflow?

I am trying to send an array of objects in below format to externalcall to a…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-19T17:38:18.1733333+00:00
Anantharaman Rajagopal 0 Reputation points
commented 2025-03-20T23:49:16.2766667+00:00
J N S S Kasyap 945 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

What is the difference between Data Factory in Microsoft Fabric and ADF ?

What is the difference between Data Factory in Microsoft Fabric and ADF ? When to use each one ? Which one requires strong technical skills?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-20T21:58:42.54+00:00
WeirdMan 300 Reputation points
accepted 2025-03-20T22:15:38.82+00:00
WeirdMan 300 Reputation points
1 answer One of the answers was accepted by the question author.

ADF Copy Activity using Google Ads Dataset - Preview Data error

I recently configured a Google Ads linked service that successfully connects to the Google Ads API. I created a dataset of type Google Ads, and it uses the new linked service that connects to the Google Ads API as seen here: I created a Copy Activity…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-17T16:51:20.2+00:00
Terry Shelton 40 Reputation points
commented 2025-03-20T21:45:29.7033333+00:00
Terry Shelton 40 Reputation points
2 answers

Adf copy activity source file json column missing

I am using copy activity under FOREACH activity to ingest multiple json files which contain col1, col2, col3, col4, col5, col6. Each json file contain only one row. I am inserting records from json files to azure sql table. Now some json file will not…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-19T18:30:46.53+00:00
Pankaj Joshi 371 Reputation points
commented 2025-03-20T19:25:11.9+00:00
Chandra Boorla 10,250 Reputation points Microsoft External Staff
2 answers

DataFactory Transform ResultSet into a flat array

Hi, i have a pipeline and a script activity. I use a linked database and and issued a select The result set is "resultSets": [ { "rowCount": 2, "rows": [ {…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-19T14:55:50.5433333+00:00
Jan Schmidt 0 Reputation points
commented 2025-03-20T19:22:09.6666667+00:00
Chandra Boorla 10,250 Reputation points Microsoft External Staff
2 answers

Connectivity from Azure Data Factory to SFTP via NAT Gateway fails

I have to reach to the client sftp through my adf linked service. Client needs a single static ip to whitelist on their end. I created a SHIR within adf in a subnet that further is associated with a NAT gateway public ip. I requested the client to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-17T12:31:21.68+00:00
Pratima Patil 30 Reputation points
commented 2025-03-20T19:16:30.4233333+00:00
Chandra Boorla 10,250 Reputation points Microsoft External Staff
1 answer

Need some information adding parameters to the Connection string for MYSQL link Service

Hi , I am using the ADF on Connecting MySQL database using Link Service. When executing the Copy Activities from MySQL as Source and SQLServer as the Sink, I am running in to the exception after couple of runs. Failure happened on 'Source' side.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-18T16:33:23.2066667+00:00
Senthil Avanasi 0 Reputation points
commented 2025-03-20T19:04:46.3466667+00:00
Senthil Avanasi 0 Reputation points
0 answers

'json' parameter is not valid after updating secret key in ADF web activity

Hello, We have a pipeline with two web activities: The first retrieves a secret key from the key vault. The second uses that secret key to obtain a bearer token. Recently, we updated the expired secret key retrieved in the first web activity. The…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-20T15:33:29.6633333+00:00
Anneka Osmun 0 Reputation points
commented 2025-03-20T18:05:03.51+00:00
J N S S Kasyap 945 Reputation points Microsoft External Staff
1 answer

ADF pipeline to Salesforce fails in Trigger mode, but runs in Debug mode

I have an ADF Pipeline that is connected to Salesforce. It runs fine in debug mode. But when I trigger it, it fails with user credential error: Failure happened on 'Source' side.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-20T13:05:06.4766667+00:00
SW 0 Reputation points
edited a comment 2025-03-20T17:24:18.0966667+00:00
SW 0 Reputation points
1 answer

how to add ingestion time column in csv getting created by calling rest api from azure data factory copy activity

Hi, I am getting some data from Rest API and copying it in a csv file. I want to add a additional column ingestion_timestamp, which is time at which pipeline ran, to each row of csv . How do I achieve this, as additional column is only available on…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-18T14:57:56.8066667+00:00
vijayshankar 0 Reputation points
commented 2025-03-20T16:17:41.6633333+00:00
J N S S Kasyap 945 Reputation points Microsoft External Staff
1 answer

connector for crunchbase

Is there a built-in Azure Data Factory connector for Crunchbase? I am looking to extract data from Crunchbase, which provides comprehensive private-company insights, and integrate it into Azure for further processing. If a native connector for Crunchbase…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-20T11:26:58.4466667+00:00
Gabriel-2005 465 Reputation points
commented 2025-03-20T11:58:52.0366667+00:00
Smaran Thoomu 21,510 Reputation points Microsoft External Staff
2 answers One of the answers was accepted by the question author.

Have there been changes to how ADF web activities access blob storage while authorized via SAMI?

We have an Azure Data Factory pipeline that includes a Web Activity, which writes data directly to an Azure Blob Storage container. This Web Activity is configured to use System-Assigned Managed Identity for authentication against the Blob Storage…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-13T07:33:42.9733333+00:00
Juho Paavola 20 Reputation points
accepted 2025-03-20T11:29:28.72+00:00
Juho Paavola 20 Reputation points
1 answer

Suggest strategy/architecture for API Data Ingestion - Data factory or Function apps or something else?

Hi! Description and Goal I have 50 external APIs and 2 on Prem APIs right now. So, I want to pull data from these APIs (some Apis can have pagination too) and put into some storage. I might pull one API every 2-minute, one API every 10-minute,…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-19T12:59:12.7166667+00:00
Amrale, Siddhesh 20 Reputation points
commented 2025-03-20T10:49:13.9066667+00:00
Nandan Hegde 34,431 Reputation points MVP
0 answers

Export to CSV in ADF is copying the text to multiple lines

The preview on ADF dataflow shows the string in single line, but the sink which is my CSV file is showing multi line in the CSV which is causing error while running the next job.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-19T08:09:00.3766667+00:00
Sowjanya M 0 Reputation points
commented 2025-03-20T06:20:13.92+00:00
phemanth 14,795 Reputation points Microsoft External Staff
1 answer

ADF Dataflow issue with source API query

am trying to setup an Azure Data Factory Dataflow. The source is a REST API from Monday.com which uses GraphQL. The API requires POST method, with 3 headers Content-Type "application/json", API-Version "2025-01 " and Authorization:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-02-22T03:15:01.8166667+00:00
Daniel Baird 0 Reputation points
commented 2025-03-20T06:13:16.0733333+00:00
Rakesh Govindula 260 Reputation points Microsoft External Staff
0 answers

Delta version upgrade in ADF

Hi, I am facing the issue to read file in Blob (delta file) using dataflows in ADF. you can see the error message below. **"**Job failed due to reason: at Source 'ecadl': Delta table "XXXXXXXXXXXXXXXXXX" with minReaderVersion of 3 and…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-17T18:41:04.3633333+00:00
Venkateswarlu Pottapalli 0 Reputation points Microsoft External Staff
commented 2025-03-19T16:16:27.89+00:00
J N S S Kasyap 945 Reputation points Microsoft External Staff
2 answers One of the answers was accepted by the question author.

Incomplete Files from Copy Data Command in Azure Data Factory pipeline when uploading data from Snowflake

I am experiencing an issue where the file-sink of the Copy Data command (SnowflakeExportCopyCommand) is producing incomplete files when uploading data from Snowflake to Azure Blob Storage in our Azure Data Factory pipeline. Observations: The number of…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-12T19:07:37.29+00:00
Susan Rakers 20 Reputation points
accepted 2025-03-19T10:36:13.6833333+00:00
Susan Rakers 20 Reputation points
3 answers

ADF Power Query Sink Settings UI Missing Options

Why would my data factory not have the same UI as I see in training videos and QA answers on the Power Query data wrangling part of ADF? I can only see the delete button next to the dataset in the Power Query activity Sink…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-01-23T15:50:49.8533333+00:00
Elisa Schrader 20 Reputation points
commented 2025-03-19T10:27:05.43+00:00
AnnuKumari-MSFT 34,361 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Using RFC table options to filter rows based on date column in SAP data

I have an ADF pipeline consisting of a copy data activity which pulls SAP data using an SAP table connector. I want to filter rows where the date column has value equal to current date. It should dynamically fetch the current date and filter rows based…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-18T09:47:58.4466667+00:00
Joshi, Anirudh 30 Reputation points
commented 2025-03-19T05:07:09.0266667+00:00
Ganesh Gurram 5,135 Reputation points Microsoft External Staff
1 answer

I am using a copy activity in ADF to copy 4million dicoms from one storage account to another. It throws 409 error :

I am trying to copy 4 million images from one storage account to another. It throws this error: Failure happened on 'Sink' side. ErrorCode=AzureBlobWriteOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Blob…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,358 questions
asked 2025-03-14T13:34:56.1966667+00:00
Jyothi Chaganty 6 Reputation points
commented 2025-03-19T04:51:57.1566667+00:00
Ganesh Gurram 5,135 Reputation points Microsoft External Staff