App retreiving and processing data from Dynamics 365 based software (complex tables & big amounts of data)

[asa9]

Hi community,

I'm researching the ways to retrieve and process large datasets from several tables and was thinking if it was possible with using Dataverse using Power Automate and Power Apps. The tables are quite large, containing millions of records with complex relationships and the data is retrieved from Dynamics 365 and based on Dynamics 365 official accountant software. The main purpose is retreiving data and conduct checking this data with dates of checking and PIC name and possible print or save as PDF when necessary. 

Requirements:

1. Efficient data retrieval and processing.
2. Ability to handle large volumes without significant performance issues.
3. Output the processed data into Excel or CSV format
4. Avoiding extra costs on licenses and premium connectors

I wonder if it is possible without usage of premium connectors and BI and if there is a need of special permissions and security roles for each user who is accessing the app. Will be grateful for all your kind ideas and suggestions but thank you for your consideration as well. Wish you a pleasant day!

[GuidoPreite]

which exact Dynamics 365 software you are talking about?

If you need to handle millions of records, Power Automate is not the answer for your problems

[asa9]

Hello! Thank you for your reply! The software is HI-KORT. As for the Power Automate capacities limitations, so even some advanced filtering and pagination won't work? 

[GuidoPreite]

I asked which Dynamics 365 software you need to work, your question is not clear, from your latest reply maybe you need to move data from HI-KORT (it's related to SAP?) to Dynamics 365?

Dynamics 365 can identify different products:

Dynamics 365 (CRM, CE/Customer Engagement) it's a software based on Dataverse

Dynamics 365 F&O (Finance & Operations, new version of Dynamics AX) is an ERP software

there is also Business Central (new version of Dynamics NAV) that is also an ERP as well.

Let's say you need to migrate 2 million records from HI-KORT to Dynamics 365 CE (based on Dataverse) doable with Dataverse API but with that amount of data Power Automate is not the answer if you need to migrate this data in a reasonable amount of time, you should look at other tools like Azure Data Factory or SSIS (SQL Server Integration Server) with a tool like KingswaySoft

[Jmanriquerios]

Hi @asa9 

To process large volumes of data, I would not recommend Power Automate because you have a call limit and you would always need a premium dataverse connector.

Now within dataverse maybe you can use elastic tables to do this scenario

https://learn.microsoft.com/es-es/power-apps/maker/data-platform/create-edit-elastic-tables

In that sense, the processing could use Azure functions combined with servi bus, everything will depend on the scenario you have.

Then maybe to create the PDF you can use Power Automate (but you return to the premium connector), but if you use dataverse you can use Power Automate.

[asa9]

My sincere apologies for the unclear response: Data is processed between Dynamics 365 F&O  and HI-KORT (specialized accounting template for Dynamics 365 Finance and Operations) and the issue is that there is no way to portably and simply retrieve data from it and was not possible to add check columns and such so here is a need to create a solution app that would let to access data sets for management purposes and download it in convenient form, not really migrate whole stuff into the app... 

[GuidoPreite]

I suggest to ask your question to this forum https://community.dynamics.com/forums/thread/?groupid=5b012a23-c24f-46fc-bfb6-45a1c401d526 as it is for F&O and probably you can find a better answer there than here.