Scale-up compute power for complex data-heavy dataflows

[CharlesS]

Is there any way to scale up the compute power used to process dataflows? Dataflows can get quite complex and can process pretty large amounts of data. As such, many of my flows can take two or three hours to load into Dataverse.

It would be great if we could pay to "scale-up" the compute power, a little bit like you would do with a Virtual Machine in Azure. Yes, yes, I know, Dataverse isn't infrastructure as a service (IaaS). Nevertheless, bottom line, we would be willing to pay to process our dataflows faster.

Is there any way to do this?

[ChrisPiasecki · Accepted solution]

Hi @CharlesS,

Dataflows are currently best suited for smaller data integration needs and currently have limitations/throttling limits on how much data it can handle.

You may want to look at Azure Data Factory for larger data volumes that you can scale up/down as needed. 

Have a review of the documentation below to see if it fits your need. 

https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/why-cds-import-export

---
Please click Accept as Solution if my post answered your question. This will help others find solutions to similar questions. If you like my post and/or find it helpful, please consider giving it a Thumbs Up.

[HSheild · Accepted solution]

Hi @CharlesS ,

To add to what @ChrisPiasecki mentioned, you also need to consider the Dataverse API limits.  Azure Data Factory may be more scalable in terms of processing data but at the end of the day you are still putting the data into Dataverse and it does not matter what method you use, you will be throttled by the Dataverse API limits.

There isn't an option to scale up the processing speed but if you have a genuine interest in this and would be willing to pay for the scale then it might be worth talking to Microsoft directly about this.

[CharlesS]

@HSheild @ChrisPiasecki 

Thanks for your replies! I had indeed considered using Azure DataFactory, but there are problems with this:

1. Many users have access to PowerPlatform, but not Azure.

2. Data Factory requires a different skill set.

My work often puts me on the blurry line between Azure and the PowerPlatform. Ultimately, my greatest wish would be for PowerPlatform to extend its functionalities further, to allow for the treatment of more data and more complex data (with premium associated fees to be paid to MSFT and greater API limits), and for Azure to "democratize" further to a "citizen developer" level. You know what? I think it will eventually happen because the business opportunity here for Microsoft is huge.

[nicknow]

That will inevitably happen and it has been happening for a number of years now. As the Azure platform expands its capabilities those capabilities eventually get democratized via the Power Platform. That said, there will always be a blurred line at the edge and we'll have to navigate it, by helping citizen developers understand where that limitation sits (at the current time) and the options for getting past it by utilizing more pro-dev based solutions.

And then certain things may not get democratized because the usage telemetry tells Microsoft there is not enough demand to make it a successful offering. This is why I've come to very much appreciate the forward guidance Microsoft gives us twice a year on Power Platform.