Collections maxes out at 2000 when populating with SharePoint list datasource

[CLS720]

I have an app where on the OnView property for a screen, a collection is created from a SharePoint list datasource.

ClearCollect(CompletedRequestsCollection, CompletedRequests)

I then try to work around the delegation issue  within a gallery by sorting/filtering the collection that is created from OnView. However, even though there is no filtering on this datasource and there is no indication of non-delegation, the collection that is created OnView always maxes out with the first 2000 items in the list. Is this expected? Are there any workarounds?

[Drrickryp · Accepted solution]

HI @CLS720,

Collect() and ClearCollect() are not delegatable so they max out at 2000k items.  However, once a collection is created, all powerapps functions can be delegated to the collection including Search() and all numeric functions.   @martinav and I (well mostly him) came up with a way to rapidly collect a maximum of 4000 items from a Sharepoint list.  As long as the list doesn't exceed 4000 items it will work and it works by collecting the first 2k items then sorting the list descending and collecting the bottom 2k items.  Then merging the lists and then removing duplicates.  It works this way.: In the OnVisible property of the screen:

Concurrent(
ClearCollect(List1,Filter(SortByColumns(List,"ID",Ascending),ID>0)),
ClearCollect(LIst2,Filter(SortByColumns(List,"ID",Descending),ID>0))
);
 

So two collections (List1 and List2) are created concurrently.  This greatly increases the speed of the process.

Then the two two collections are merged, elimininating duplicates, into ListAll and the two collections, List1 and List2 are emptied.  Just add this formula to follow the semicolon in the upper formula box. 

ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

 ListAll now contains up to 4000 unique rows and can be used with all Powerapps functions without regard to delegation,  and as an items property for galleries,  datatables, and Comboboxes (but not Dropdowns as these are hardcode limited to a maximum number of items, no one wants to scroll through thousands of items in a dropdown anyway.).    The execution speed of the formulas like Search() or Lookup() will astonish you. 

[martinav]

Hey  @Drrickryp,  THanks for the tag!  It was a combined effort, but I thought you came up with the first/last scenario... anyhwo, it is working very well.  I have a list with 2,400 items, and it does indeed work well.  Also, by using the ShowColumns() command, you can pull in only the columns you wish to have in the collection.  If you dont need it, no sense in wasting time and memory space with data you dont need.  

Here are my actual commands, which I put in the "OnStart" control for the app.  

Concurrent(
ClearCollect(MDL_List1,Filter(ShowColumns(SortByColumns(Master_Drawing_List,"ID",Ascending),"ID","MD_PartNumber","MD_Project","MD_LatestRevNumber","MD_DrawingRevDate","MD_EngrgRelDate","Title","MD_PDM_Link","MD_LinkToDrawing","MD_IsActive","MD_Data_Flagged"),ID>0)),

ClearCollect(MDL_List2,Filter(ShowColumns(SortByColumns(Master_Drawing_List,"ID",Descending),"ID","MD_PartNumber","MD_Project","MD_LatestRevNumber","MD_DrawingRevDate","MD_EngrgRelDate","Title","MD_PDM_Link","MD_LinkToDrawing","MD_IsActive","MD_Data_Flagged"),ID>0))
);

ClearCollect(MDL_All,MDL_List1,Filter(MDL_List2, Not(ID in MDL_List1.ID)));Clear(MDL_List1);Clear(MDL_List2)

So, everything is pulled one time.  I do have a data refresh button that executes the exact same commands if the user thinks his data is stale.  I also have a timer that runs, and reloads the data every 30min.  If you dont do that, since data in your collection does not update automatically, you might want to recollect the data from sharepoint regularly.  It will depend on your specific use.  Also, note that I cleared the two collections used to pull in the two separate lists.  To minimize memory usage.

This really did work well.  Much better than I could have hoped!  I did have another idea..  To perhaps get more items.  I was wondering if can collect 2,000 items at a time using the ID of the sharepoint list.  (if(ID>2000 AND ID<4000) etc etc. It might take using a Flow, though, because I think that using gt or lt functions are not delegable.  However, flow allows you to to get items, and use ID.  I think you can do get items using the get items filter.  Then sending back to powerapps using the Response action.  I have no idea how long it would take.   

[CLS720]

Thank you both! I applied this solution and it worked beautifully.

[martinav]

Sweet!  Another happy customer!

[Anonymous]

Hey Guys, can you help me please?

I have a excel spreadsheet wich cointains 37k rows and i want to bring it to powerapps.

the data does not change, so i uploaded the excel via import via excel, and when i collect it brings me 15k rows.

but how can i separate in 15k15k and another one with 8k to bring all the datas?

[martinav]

Hold that thought...  while you are waiting... what commands/objects are you using?

[Anonymous]

actually @timl figured out a way to do this with.

spread the excel table in 3 parts then collect all of them in powerapps.

worked 😉

thanks!  

[ubikwitous]

Seems to work @Drrickryp and @martinav , but more fun!

I have a SharePoint list with thousands of items, but I only want to pull items where the Start Date is greater than today. New items will come in over time. Is there a way to do this? I tried adding the Filter in, but with no change.

Concurrent(
ClearCollect(List1,Filter(SortByColumns('Events','Start Time'>Now(),"ID",Ascending),ID>0)),
ClearCollect(List2,Filter(SortByColumns('Events','Start Time'>Now(),"ID",Descending),ID>0))
);
ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

I have a feeling I'm going to have to somehow create a Flow to keep a 2nd copy of the SharePoint List that is cleaned up. Any advice would be appreciated - thanks!

[martinav]

@ubikwitous ,

Filter first, then sort.  The parenthesis work from the inside out.  

Concurrent(
ClearCollect(List1,SortByColumns(Filter('Events','Start Time'>Now()),"ID",Ascending),ID>0)),
ClearCollect(List2,SortByColumns(Filter('Events','Start Time'>Now()),"ID",Descending),ID>0))
);
ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

I didnt actually try this, I think the syntax is all ok.  But, this may not work because the ">" will be a delegation issue itself. This may not work.

Getting 2,000 rows first, and then filtering would look like this... (I think)

Concurrent(
ClearCollect(List1,Filter(SortByColumns('Events',"ID",Ascending),'Start Time'>Now()),ID>0)),
ClearCollect(List2,Filter(SortByColumns('Events',"ID",Descending),'Start Time'>Now()),ID>0))
);
ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

I'm not sure about your syntax for 'Start Time'.  You also will need to be sure you use time zone correction to get your time adjusted to UTC().