Is it possible to pass an array (of objects) as an input payload to External Call REST API Transformation in Dataflow?
I am trying to send an array of objects in below format to externalcall to a…
What is the difference between Data Factory in Microsoft Fabric and ADF ?
What is the difference between Data Factory in Microsoft Fabric and ADF ? When to use each one ? Which one requires strong technical skills?
ADF Copy Activity using Google Ads Dataset - Preview Data error
I recently configured a Google Ads linked service that successfully connects to the Google Ads API. I created a dataset of type Google Ads, and it uses the new linked service that connects to the Google Ads API as seen here: I created a Copy Activity…
Adf copy activity source file json column missing
I am using copy activity under FOREACH activity to ingest multiple json files which contain col1, col2, col3, col4, col5, col6. Each json file contain only one row. I am inserting records from json files to azure sql table. Now some json file will not…
DataFactory Transform ResultSet into a flat array
Hi, i have a pipeline and a script activity. I use a linked database and and issued a select The result set is "resultSets": [ { "rowCount": 2, "rows": [ {…
Connectivity from Azure Data Factory to SFTP via NAT Gateway fails
I have to reach to the client sftp through my adf linked service. Client needs a single static ip to whitelist on their end. I created a SHIR within adf in a subnet that further is associated with a NAT gateway public ip. I requested the client to…
Need some information adding parameters to the Connection string for MYSQL link Service
Hi , I am using the ADF on Connecting MySQL database using Link Service. When executing the Copy Activities from MySQL as Source and SQLServer as the Sink, I am running in to the exception after couple of runs. Failure happened on 'Source' side.…
'json' parameter is not valid after updating secret key in ADF web activity
Hello, We have a pipeline with two web activities: The first retrieves a secret key from the key vault. The second uses that secret key to obtain a bearer token. Recently, we updated the expired secret key retrieved in the first web activity. The…
ADF pipeline to Salesforce fails in Trigger mode, but runs in Debug mode
I have an ADF Pipeline that is connected to Salesforce. It runs fine in debug mode. But when I trigger it, it fails with user credential error: Failure happened on 'Source' side.…
how to add ingestion time column in csv getting created by calling rest api from azure data factory copy activity
Hi, I am getting some data from Rest API and copying it in a csv file. I want to add a additional column ingestion_timestamp, which is time at which pipeline ran, to each row of csv . How do I achieve this, as additional column is only available on…
connector for crunchbase
Is there a built-in Azure Data Factory connector for Crunchbase? I am looking to extract data from Crunchbase, which provides comprehensive private-company insights, and integrate it into Azure for further processing. If a native connector for Crunchbase…
Have there been changes to how ADF web activities access blob storage while authorized via SAMI?
We have an Azure Data Factory pipeline that includes a Web Activity, which writes data directly to an Azure Blob Storage container. This Web Activity is configured to use System-Assigned Managed Identity for authentication against the Blob Storage…
Suggest strategy/architecture for API Data Ingestion - Data factory or Function apps or something else?
Hi! Description and Goal I have 50 external APIs and 2 on Prem APIs right now. So, I want to pull data from these APIs (some Apis can have pagination too) and put into some storage. I might pull one API every 2-minute, one API every 10-minute,…
Export to CSV in ADF is copying the text to multiple lines
The preview on ADF dataflow shows the string in single line, but the sink which is my CSV file is showing multi line in the CSV which is causing error while running the next job.
ADF Dataflow issue with source API query
am trying to setup an Azure Data Factory Dataflow. The source is a REST API from Monday.com which uses GraphQL. The API requires POST method, with 3 headers Content-Type "application/json", API-Version "2025-01 " and Authorization:…
Delta version upgrade in ADF
Hi, I am facing the issue to read file in Blob (delta file) using dataflows in ADF. you can see the error message below. **"**Job failed due to reason: at Source 'ecadl': Delta table "XXXXXXXXXXXXXXXXXX" with minReaderVersion of 3 and…
Incomplete Files from Copy Data Command in Azure Data Factory pipeline when uploading data from Snowflake
I am experiencing an issue where the file-sink of the Copy Data command (SnowflakeExportCopyCommand) is producing incomplete files when uploading data from Snowflake to Azure Blob Storage in our Azure Data Factory pipeline. Observations: The number of…
ADF Power Query Sink Settings UI Missing Options
Why would my data factory not have the same UI as I see in training videos and QA answers on the Power Query data wrangling part of ADF? I can only see the delete button next to the dataset in the Power Query activity Sink…
Using RFC table options to filter rows based on date column in SAP data
I have an ADF pipeline consisting of a copy data activity which pulls SAP data using an SAP table connector. I want to filter rows where the date column has value equal to current date. It should dynamically fetch the current date and filter rows based…
I am using a copy activity in ADF to copy 4million dicoms from one storage account to another. It throws 409 error :
I am trying to copy 4 million images from one storage account to another. It throws this error: Failure happened on 'Sink' side. ErrorCode=AzureBlobWriteOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Blob…