Standard (Dataverse) Dataflow vs Power Automate

[KoenJanssensPD]

Dear all,

I have an external data source that is being updated regularly (every few minutes during a workday), from which I get its data through a REST API.

Currently I am fetching all the data of the current day with a Standard Dataflow, every 5 minutes. This means that I read the same (hundreds of) records each time, plus any additional records that are created during the day. My dataflows will CRUD these records into multiple (standard and custom) entities in my Dataverse.

It seems that this method is causing a lot of overhead on the system, and that these dataflow executions are getting beyond the available capacity and are being slown down after a while...

Should I try to convert these Dataflows into Power Automate flows that are only triggered when data is changed in the source data (if I somehow can detect that)?

I start wondering why one should choose either a Standard Dataflow or a Power Automate Scheduled Cloud Flow? In this case, they both achieve the same, namely reading an external source and copying the data into Dataverse...

Could anyone shine a light on the "best-practices" on choosing between Dataflows and Cloud Flows?

Thanks,

Koen

[cchannon]

Try using Incremental Refresh with Dataflows. It can let you selectively pull content based on when it was created or modified (assuming your source data tracks these)

Using incremental refresh with dataflows | Microsoft Docs

[KoenJanssensPD]

Hi @cchannon , that seems a legit solution; I will try that, but I will be struggling, since not all entities that we fetch from this source have audit tracks...
I am still wondering how can get more details on the usage of these flows. The flows I am talking about suddenly stopped working this night, while the flow itself and datasource have not been changed...

The error in the logs only says "Failed: There was a problem refreshing the dataflow. please try again later. (request id: e36b2f5e-703d-4877-8b89-5ef382c6bd1e)"... Is there any place where we can get more details on these (failing) requests?

This is driving us crazy and makes the system unusable for our use case...

[cchannon]

hmmm... That is frustrating.

Did you try executing them manually to see if you might get a clearer error message?

[KoenJanssensPD]

Hi @cchannon ,

manual refresh gave the same problem...
After quite some messaging with Microsoft support the last two weeks, they admitted there were some intermittent problems with Power Platform Dataflows.

It still strikes me that we, as a Microsoft Partner, do not have access to status reports on these kind of issues in the Power Platform. This means we have no clue when such problems arise and we have nothing to tell our customers when this happens... Also the lack of good documentation and analytics about the available and used capacity for dataflows is problematic...

[cchannon]

This is a good point, @KoenJanssensPD . This would have been helpful recently with Dataflows in particular; I have seen several posts from apps in EU regions complaining about DataFlows running crazy long, though this is the first I have seen of them failing unexpectedly. 

@chass - a "Known/Active Issues" section might prove helpful for things like this, since people often come here to the community first. Something to consider.

[dave-jorgensen · Accepted solution]

Did you try using Azure Data Factory instead of Dataflow / flow for syncing your data?

[KoenJanssensPD]

Hi @dave-jorgensen , never heard of it , but will take a look for sure, thanks!!!

[dave-jorgensen]

No problem. Azure data factory (ADF) is intended for these exact scenarios, migrating data. The only tricky bit is setting up alternate keys. Basically, the unique id field in your datasource, you add that field to your dataverse table (I'm sure you already have this). In dataverse table, you designate this as an alternate key. Then when you setup the data source / data sink, you map fields in source to destination, and there is a dropdown to set the alternate key. That's what allows ADF to determine if it's an insert or update - that alternate key.

[cchannon]

@KoenJanssensPD Did these posts resolve your issue? If so, please mark the solution so future visitors will see this issue as resolved.