Read WebApp connection string credentials from Keyvault
I have hosted an ASP.Net web form webapp in Azure. Right now the password of the backend DB is being exposed in the connection string. Is there a way to read it through Azure Keyvault ? Please share your recommendations. Thanks in advance.
Incomplete data output from GET API ADF copy transformation
I am trying to copy data using Bing GET API in ADF, during the preview data stage the data looks fine. However when pushing it to sink as JSON, the output data is incomplete. Is there any steps i missed/ misconfigured? Configurations: Data…
ADF - run an activity if any one of a set of activities fail
Is there a way in which I can run one generic activity (say log error), when any of the activities in the pipeline fail ? Thanks, Kothai
How to Resolve Missing or Null Columns When Copying Data from Dataverse to Azure SQL Using Dataverse Connector?
I am currently working on a project where I need to copy data from a Dataverse table to an Azure SQL database using the Dataverse connector in Azure Data Factory (ADF). However, I am encountering the following issues: Some columns from the Dataverse…
Intermittent Issue with Lookup Activity in ADF Fetching Data from Snowflake
I'm encountering an intermittent issue with the Lookup activity in Azure Data Factory (ADF) when working with a Snowflake dataset. My setup involves an admin table in Snowflake where I store details of multiple tables. When I try to fetch all rows using…
Dataflow not using the Base and Relative URLs in Dataset
I've set Base and Relative URLs in REST Linked Service and Dataset. These are read correctly by the dataset, which connects successfully. However, Dataflows do not read the Base and Relative URLs correctly from Dataset / Linked Service (only reads the…
Synchronizing Tables from Multiple Azure SQL Databases Using Azure Data Factory
Currently facing a challenge involving two different Azure SQL Servers, each containing 4-5 databases. The goal is to select specific tables from each database and synchronize them into a single Azure SQL Database to serve as a data warehouse for…
ADF data load
Hi friends, I have datalake by structure subject area ABC 2024 12 13 It is YYYY mm and dd partitioned created data lake, now I am trying to load the file in date 13 but I can only point it to the subject area,…
Is it possible to get User Assigned Managed Identity bearer token via ADF on MSIT tenant?
Can you please confirm if it is possible to do UAMI token generation via ADF on MSIT tenant? We do not have a PME subscription currently. Stackoverflow questions: https://stackoverflow.microsoft.com/questions/435481 …
Data factory permissions to storage account
I can copy data from on premise database into data lake but when I try to add a dataflow to the pipeline I get this error I think the IAM is set up correctly I only have one resource group so it's pointing to the correct one I've also made data…
How to write to two tables using pipelines
Hello I'm trying to achieve the following: Get data from one table using a stored procedure Compare this data to another table Write to 2 tables using a stored procedure It seems i can use dataflow to do 1&2 but not 3, and I can use pipeline to…
Analysis of Cost Saved
Hi, Current Solution: Delta table from HiveMetastore -> Staging Table with Business Logic in Databricks -> PowerBI Current Solution in Detailed: Main responsibilities for me is to consume the data the delta tables from hive_metastore in…
How to transfer azure domain name one account to another account
Hello Teams, I want to transfer domain names from one account to another. How can I process this? I have been trying to figure this out since last week but haven’t found any proper documentation on Google or Microsoft. I found a document suggesting…
ADF is unable to place logs inside Log analytics workspace.
Hello, I would like to bring your attention to an issue we are currently experiencing: We have an Azure Data Factory (ADF) setup and are trying to configure logging for it. A Log Analytics workspace has been created and attached to ADF. We set up…
Need to execute the java jar from ADF
I have jar file which perform certain operation. I need to call that from ADF , Is there any solution or way to execute the jar file from ADF?
Copy data from oracle to azure sql server
I have a use case where I need to copy the data from the Oracle database (on-prem) to the Azure SQL server. I need to upsert the data along with the timestamp column to the destination table which indicates the date a new record has been inserted or an…
The remote server returned an error: (400) Bad
Our PIPELINE file process was running perfectly, but couple of days ago it started failing below it is copying sql data to .parquet format on Hot Storage, please let me know what to check. Failure happened on 'Sink' side.…
Encountering 'Bad Request' Error Whenever Debugging Azure Simple Pipeline
I am trying to debug a simple Azure pipeline, but I consistently encounter the following error: "code": "BadRequest", "message": null, "target": "pipeline//runid/94168fdf-d49b-4ca8-9d01-dc17298b4b5d",…
Azure Data Factory / Synapse: paging method for copy activity and data flow source work diffrently
Hi everyone, me again fighting with the REST sources in synapse. I finally discovered how paging methods work in a copy activity. This is the projection of the payload coming from the graph api: If more than 100 records in the value…
Resultsets returned by script activity after switching to new SNF connector for ADF are not backward compatible - breaking pipelines
Hi MS support team, I am trying to upgrade Snowflake connector to V2 in my ADF pipelines. I have script activity which has multiple SQL commands (eg. set query_tag, call a stored procedure). Although these get executed successfully now, my subsequent IF…