Understanding CDS - data and connection limits

[PAM-thrive]

Hello,

I'm completely new to CDS, trying to test it out and I'm coming up against some issues that I find really strange in this day and age of high velocity, high volume data.

https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/api-limits

As described in this link above, I've been hitting a 6000 limit in 300 seconds when trying to transfer when trying to get an initial loading of data into a custom entity in CDS. (the data loading is 6150 rows!)

Between this, and sharepoint's 5000 limit, it just all seems unusable and this is why I'm questioning our understanding of these tools.

Going forwards I have lots of data I was planning to drop in CDS as our organsiation is considering increasing our reliance on Power Platform and eventually also look at Dynamics.

In many cases it is very difficult to understand when data has changed or new rows added to our SQL table, so a complete refresh on a schedule would be a common approach, but until we migrate to Dynamics I was hoping to keep the CDS data up to date each night, to use with powerApps and flows.

Are we thinking about CDS wrongly? as this sort of limitations listed in the above link just make me think it isn't a production ready environment. How are companies.

I should probably point out that our SQL server is in a VM so requires on-premise data gateway to shift the data across. We are a small non-profit org, and the idea of going down a full-on data warehouse sounds a bit OTT and as we move into dynamics, a DWH sounded unnecessary.

thanks for any pointers.

[SeanHenderson · Accepted solution]

Hi @PAM-thrive 

Are you looking at a one time data dump or a persistent connection between SQL and CDS?

If you need a connection, have you tried setting up a dataflow?

If it's a one time dump, I would recommend downloading your data as a csv file and then importing it using the Data Import Wizard, which can be found from make.powerapps.com by clicking the gear in the top right, then Admin Center > Select Your Environment > Settings > Data Management > Imports

or Gear > Advanced Settings > Click the arrow next to settings along the top > Data Management > Imports

Using the wizard will require you to set up the entities in CDS first in order to have fields you can map to.

---
Please hit the "Accept as Solution" button if my post answered your question! If my post was helpful please consider giving it a "Thumbs Up."

[PAM-thrive]

Thanks @SeanHenderson ,

Its a bit of both. Right now I'm trying to get an understanding of CDS. Because we haven't actually moved applications onto using CDS yet but I'd like to keep the data on there up to date (allows me to run example PowerApps side-by-side with existing systems).

I'll try the idea you've suggested for a one-time dump of data, then update my dataflows to see if I can "identify changes" and upload only those which should hopefully stop us hitting the limits.

Thanks