11,780 questions with Azure Data Factory tags

Sort by: Updated
1 answer

Update Oracle Connection version 2.0 from 1.0

This is connected to another question I have. https://learn.microsoft.com/en-us/answers/questions/5490043/adf-oracle-connection-deprecated-version-1-0 Since the version 1 will be decommissioned soon. We need to update our Oracle connections to version 2.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-22T14:59:35.27+00:00
Colinn Calaguas 70 Reputation points
answered 2025-08-24T05:00:38.8766667+00:00
Vinodh247 36,681 Reputation points MVP Volunteer Moderator
1 answer

Delete records from a salesforce object using Azure data factory

Hi there, I am working on a ADF pipeline that transforms data from salesforce and finally i need to delete some records from salesforce object. adf does not have a built in feature to delete and i was wondering if we can utilize lookup activity to run …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2024-05-06T16:57:11.73+00:00
Jiby Jacob 90 Reputation points
edited a comment 2025-08-23T17:12:46.5133333+00:00
Kleber Rebello 0 Reputation points
1 answer

Monitoring Shows Lower Inserted Row Count After Switching Azure SQL DB to UTF-8 Collation

Hello, I'm encountering an unusual monitoring issue after changing my Azure SQL Database collation to UTF-8 (e.g., Latin1_General_100_CI_AS_SC_UTF8). Here’s the scenario: I load 40679 records from a Parquet file (UTF-8 encoded data) into the database…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-23T06:15:46.8733333+00:00
sudadigitus az 20 Reputation points
answered 2025-08-23T17:05:29.3533333+00:00
Vinodh247 36,681 Reputation points MVP Volunteer Moderator
2 answers One of the answers was accepted by the question author.

ADF Logging Strategy – Expert Insights Wanted!

ADF Logging Strategy – Expert Insights Wanted! I'm currently building a reusable logging framework in Azure Data Factory (ADF) to capture execution metadata for key activities like Data Flows, Copy Activities, and Stored Procedures — and I’d love to get…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-07T17:31:45.4033333+00:00
sudadigitus az 20 Reputation points
accepted 2025-08-23T05:51:32.25+00:00
sudadigitus az 20 Reputation points
0 answers

Data Flow Sink with "Continue on Error" and Output to separate file not getting created

Hello, I have a Sink destination linked to Azure SQL DB in a data flow. Where I wanted to capture database rejected records into Azure BLOB file. I have tried choose to Continue on error with Transaction commit Single/Batch and selected Output to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-22T18:53:48.97+00:00
Karthik K Koneru 1 Reputation point
commented 2025-08-22T21:08:24.2+00:00
Karthik K Koneru 1 Reputation point
1 answer

Why are ADF Pipeline Parameters Not Appearing in Override Template Section?

I am trying to parameterize ADF Pipelines parameters in azure devops so that it can be overwritten before deployment to prod, I have added the below mentioned code in arm-template-parameters-definition.json file for the same but still I don't see the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-07-24T09:34:03.1066667+00:00
Deepa M 20 Reputation points
commented 2025-08-22T17:48:19.1566667+00:00
Braun, Bill 5 Reputation points
1 answer One of the answers was accepted by the question author.

How to set column name as the file name?

I've set the original file name as a column through the copy activity, but I can't get the data flow to set the column as the file name. Copy behavior is set to flatten hierarchy, then I use a data flow for some additional transformation. In the data…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-20T17:41:55.11+00:00
AE90 80 Reputation points
commented 2025-08-22T14:44:42.4833333+00:00
AE90 80 Reputation points
2 answers

Azure Data Factory: Issues with Oracle Linked Service v2 in Copy Activity

We are working on updating Oracle Linked Services in Azure Data Factory (ADF) to version 2.0 and we have found several differences in behaviour compare with Oracle Connector v1.0: In Lookup activities, version 1.0 allowed semicolons at the end of…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-07-29T07:37:00.6+00:00
Violeta Stangaciu 25 Reputation points
commented 2025-08-22T13:59:27.0866667+00:00
Justin Hyland 5 Reputation points
3 answers One of the answers was accepted by the question author.

Does Microsoft perform regular penetration testing on their own systems

As per the heading, does Microsoft perform regular penetration testing on their own systems? I can't seem to find an answer to what I would have thought is a simple question and something they would surely advertise if they did?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2024-09-11T13:17:01.8333333+00:00
EJR 26 Reputation points
answered 2025-08-22T10:08:15.2433333+00:00
Karishma Ray 0 Reputation points
1 answer

How to overcome ADF lookup activity limitation

Scenario I have a use case where data needs to be read from a source (SQL, Blob, S3, etc.), sent to a REST API for a custom transformation, and then the API response must be written back to a sink (SQL, S3, Blob, etc.). Current Approach Step 1: Use a…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-20T17:44:26.0833333+00:00
Saravanan R 0 Reputation points
edited a comment 2025-08-22T10:07:22.0233333+00:00
Saravanan R 0 Reputation points
2 answers

How to make the Chekpoint Key / SAP to stage subscriber process a static value?

Context: I am currently building a pipeline that extracts data from a SAP CDC source, where I want to utilize the delta mechanism this connector enables. My setup is: SAP CDC connector to source system (SAP CRM)- static subscriber name and ODP…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2023-12-09T11:16:12.3066667+00:00
Georgi Ivanov 0 Reputation points
edited an answer 2025-08-22T09:44:06.7+00:00
Bröthaler Thomas 0 Reputation points
1 answer

Error_Message

Hello Team, I have setup the email Notification via logic app in ADF Pipeline.Please refer the screenshot below : But whenever the Pipeline get failed.In the email notification will not give the exact error.It will display only the pipeline got…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-21T05:20:13.9166667+00:00
Rohit Kulkarni 731 Reputation points
commented 2025-08-22T04:06:14.7566667+00:00
VRISHABHANATH PATIL 0 Reputation points Microsoft External Staff
4 answers One of the answers was accepted by the question author.

Unable to download Purview exports

Hello I have tried to download the export packages from the ediscovery menu. I have tried on mac & linux Edge and firefox browsers. When I push the download button, nothing happens. The close button does work. Is there another way to access these…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-03-03T07:21:08.6833333+00:00
Christopher Lowden 170 Reputation points
edited an answer 2025-08-22T02:07:50.05+00:00
Andrew Karpinski 0 Reputation points
0 answers

: Error during execution: Azure Data Factory Oracle connector 2.0 → Parquet write failure (ArrayIndexOutOfBounds: Index 126 out of bounds for length 38)

Hello — I'm hitting a blocking error when running an ADF pipeline that reads from Oracle (using Oracle connector 2.0) and writes to Parquet. I suspect it’s related to decimal/float precision mapping, but I can't find a definitive fix. Any guidance…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-21T19:20:47.48+00:00
Maria Gabriele Alves Ramos 0 Reputation points
commented 2025-08-21T22:57:47.0866667+00:00
Smaran Thoomu 29,165 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

How to use system variables in Azure data factory ?

I am new to ADF and need some help or guidance to use system variables in Azure data factory ? Thank you Himanshu

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-18T21:56:11.8+00:00
HimanshuSinha-msft 19,516 Reputation points Microsoft Employee Moderator
accepted 2025-08-21T21:55:33.2266667+00:00
HimanshuSinha-msft 19,516 Reputation points Microsoft Employee Moderator
1 answer One of the answers was accepted by the question author.

The Data Factory Spark v2 linked service reports "The remote certificate is invalid" even though enableServerCertificateValidation is false.

Error message: System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-20T14:17:09.1533333+00:00
B Campbell 20 Reputation points
commented 2025-08-20T19:01:00.69+00:00
Smaran Thoomu 29,165 Reputation points Microsoft External Staff Moderator
1 answer

How to get the new data factory oracle 2.0 connector to connect?

We currently connect to oracle using the 1.0 connector. We must be moved to the new oracle 2.0 connector before end of support, 7/31/25. I was able to get this to connect to a non encrypted database, but unable to connect to an encrypted database. I had…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-05-22T14:01:06.2+00:00
Blasko, Dan W 55 Reputation points
commented 2025-08-20T08:22:12.9166667+00:00
Arek Sławek 0 Reputation points
2 answers One of the answers was accepted by the question author.

PostgreSQL V2 Connector runs into an error when trying to copy timestamp with time zone values

We have an on-premises PostgreSQL (9.0.5) database from which I am staging data to an Azure SQL Database. So far, we have been using the PostgreSQL V1 Connector. I am currently trying to switch to V2, as V1 will be deprecated at the end of next…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-19T10:11:45.9+00:00
Philipp Sous 20 Reputation points
accepted 2025-08-19T21:06:38.6866667+00:00
Philipp Sous 20 Reputation points
4 answers

Unable to create new linked service for salesforce. As it changed from username password to Client ID and Client secret

Unable to establish connection to the Salesforce environment using provided client id and client secret, please verify the details. Error message: {"error":"invalid_grant","error_description":"no client credentials user…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2024-01-23T17:50:05.8433333+00:00
Manikanta Naluvala 20 Reputation points
answered 2025-08-19T17:46:00.8866667+00:00
gayathri rokkam 0 Reputation points
1 answer One of the answers was accepted by the question author.

ADLS Archive Path (compared to Hyperscale)

In our project we have a set of archive tables from DB2 iSeries. These are static datasets (historical claims data from 2007–2019, frozen and never updated). They are not part of FlashCopy or CDC, but the customer wants them in a queryable form. One…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,780 questions
asked 2025-08-17T16:02:40.0266667+00:00
Janice Chi 620 Reputation points
accepted 2025-08-19T17:09:35.2333333+00:00
Janice Chi 620 Reputation points