Upper record-limit when loading a data source into a collection: “Data row limit”-setting

[JakobAnkerHanse]

Is it a fixed unchangeable thing, that we are limited by the “Data row limit”-setting in Power Apps

Clearcollect (ColName, DataSource);

Can only up to 500 or 2000 records be loaded from a data source to a collection, based on the setting you have ?

What is the best practice to deal with this upper-limit-challenge ???

Of course, you can try to use filters to lower the number of records returned… But if this is not possible, and if the number of records still exceeds the “Data row limit”-setting…

Is the only (and best) way to deal with this challenge to, directly after data load, check the “record count” ("CountRows(ColName)") of the “data-loaded”-collection, and if the count is equal to the data row limit, create an alert informing the user that the limit has been reached and the results might not be trustworthy?

Because of some user-interface-defined constraints, I need to potentially load big data sources into collections to do different types of filtering. I will run into this limitation sooner than later. I know that there are techniques to use to double the upper level, but I will still reach this limitation at one point in time. I am considering moving from Dataverse to SQL server, and then do all my filtering server side through stored procedures... I hope I don't have to do it like this, and hope to find a scalable way of doing this using Dataverse.

Am I missing something here ? Is there a way to overcome the “Data row limit”-setting when loading a data source to a collection ?

[FLMike]

Hi @JakobAnkerHanse 

Let's start by clearing something up. Filtering and Collections are not simply tied to Dataverse.

You cannot build massive datasets client side. That goes against every single possible architectural platform design ever. 

Forget Dataverse, you wouldn't want to download 20,000 records to a phone or a tablet from SQL either.

You need to build Delegatable Filters (regardless of the back end), otherwise you will not do what you want.

No, you cannot do what you a are asking and again its not a dataverse thing at all. Non-delegatable things like 'Count' or badly designed tables etc will force you to do different things to avoid missing out on records due to delegation.

I would re-think the strategy of your data design and implementation if you are planning to use massive amounts of data on a client to then run client side filters (which won't work on most devices). Sure maybe on some laptops and desktops but...

If I have helped you, I would really appreciate if you please Mark my answer as Resolved/Answered, and give it a thumbs up, so it can help others

Cheers

Thank You
Michael Gernaey MCT | MCSE | MCP | Self-Contractor| Ex-Microsoft
https://gernaeysoftware.com
LinkedIn: https://www.linkedin.com/in/michaelgernaey

[JakobAnkerHanse]

Thank you very much, Mike 🙂 I hear you 🙂

So we should avoid working with large collections… I hear I need to build "Delegatable Filters" (regardless of the back end), and I believe the command to use here is the "Filter" command...?

Even though I might be able to accomplish this, here is the initial question: Is it a best practice to, directly after data load, check the “record count” ("CountRows(ColName)") of the “data-loaded”-collection, and if the count is equal to the "data row limit"-setting in Power Apps, create an alert informing the user that the limit has been reached and the results might not be trustworthy? See here, that I refer to using the "CountRows(ColName)" command on the collection - not on the data source, as I have learned this is not delegatable... If this command returns a count close to the "data row limit"-setting, you can conclude that you might only have loaded a sub set of the data source into the collection, and everything following hereafter might not be trust worthy... And similarly, if the count is lower than the "data row limit", we can conclude that all records have been loaded from the data source, and everything is fine...?

... Or will an error be raised, if the data row limit has been reached, and hereby something that will be caught by an error handler - if implemented ?