Delegation problems and wanting to filter more

[KimberlyM]

I am loading my gallery will ALL of the information in my Lead_Data SharePoint list dependent on the user. If  the user is listed in my Admin list then they get to see every single record other than that the user will only see what has been assigned to, or listed as lead member (1-4) in respect to their name. I have 5 tabs and the first tab is my default view. Tab one, default, pulls if my Status column equals "In Progress" or "Not Started". The other tabs pull each specific status (tab 2 = not started; tab 3 = in progress; tab 4 = dead; tab 5 = completed).

I have been successful to do this but not without delegation issues (and I can't even remember why I have these problems because I have been working on this issue for so long). Ugh. So if you have any suggestions for me, I would gladly receive them on this.

But here's the thing now I want to be able to filter further through text boxes: I would love to be able to filter by user (when logged in as an administrator); I would love to be able to filter by date; I would love to be able to filter by address; etc. I am so confused. I have watched various videos on how to do this but nothing gets as complex as I am now so I am not sure if I am doing things more complex than they need to be or if my desire is just this complex. Any suggestions on where I am now and where I want to go?

This is my code:

Sort(
     If(
          varAdmin,
          Filter(
               Lead_Data,
               If(
                    SelectedStatus = "All",Or(Status.Value = "In Progress", Status.Value = "Not Started"),Status.Value = SelectedStatus
               )
          ),
          Filter(
               Lead_Data,
               AssignedTo.Value = varUser || LeadMember1.Value = varUser || LeadMember2.Value = varUser || LeadMember3.Value = varUser || LeadMember4.Value = varUser,
               If(
                    SelectedStatus = "All",Or(Status.Value = "In Progress", Status.Value = "Not Started"),Status.Value = SelectedStatus
                    )
               )
     //, Filter(Gallery_Dashboard, dash_FilterBy.Text = Dash_AssignedTo.Text)
     ),
Created, Descending)

[RandyHayes]

@KimberlyM 

Can you clarify if you are dealing with delegation issues (i.e. warnings) or actually record limits (over 200 records)?  Is your data source over 2000 (or will be)?

In your formula you are filtering by a complex column type - this is not delegable and will produce delegation warnings and false returns on large data sources.

Let's start with the clarification and then we can move to some possible solutions.

[KimberlyM]

I am dealing with warnings for now but as of this month I will be dealing with data limited +2000 records (hence why I am circling back to this problem again). By the end of the year I predict I should have minimum +6000 records.

[RandyHayes]

@KimberlyM 

But I want to be clear on this...do you expect to have more than 2000 records in your app?  Or is it that you need to filter more than 2000 records?  Both are two different things and have at least two different methods to resolve.

[v-siky-msft]

Hi @KimberlyM 

Try to apply the if condition out of Filter function.

Sort(
     If(
          varAdmin,
          If(
                    SelectedStatus = "All",
Filter(Lead_Data, Or(Status.Value = "In Progress", Status.Value = "Not Started")),
Filter(Lead_Data, Status.Value = SelectedStatus)
           ),
        If(SelectedStatus = "All" ,
Filter(Lead_Data,AssignedTo.Value = varUser || LeadMember1.Value = varUser || LeadMember2.Value = varUser || LeadMember3.Value = varUser || LeadMember4.Value = varUser,Or(Status.Value = "In Progress", Status.Value = "Not Started")
),
Filter(Lead_Data,AssignedTo.Value = varUser || LeadMember1.Value = varUser || LeadMember2.Value = varUser || LeadMember3.Value = varUser || LeadMember4.Value = varUser, Status.Value = SelectedStatus)
         )
      ),
Created, Descending
)

Hope this helps.

Sik

[KimberlyM]

Yes, unfortunately there will be more than 2000 leads loaded into the app. 

[RandyHayes]

@KimberlyM 

Okay...some general tips from experience to start:

So, at first glance I would say - reconsider your design.  This goes from App through data.  If you are going to be dealing with that much information in an App, will it be effective from a user standpoint?  Will a user actually be looking at over 2000 items at a time?  If the purpose of so many is to allow a user to review items for "conditions" and then choose them to act on, then consider that simply displaying any items that need action based on conditions should only be shown/loaded.

Your data...is SharePoint the most effective place for your data based on the volume?  SharePoint has a lot of limitations when you start getting into large data stores.  They can be overcome, but there are sometimes a significant amount of workarounds that need to be put in place.  Some of them can be "fragile" and will need to be monitored at times.  SQL becomes a much better alternative for this as you have much more control over the shaping of data both in and out of the database. 

Now, that said - some specifics (and I will address generally how to get more than 2000 records)

Loading more than 2000 records into the app is very possible.  But, you need to do it in stages - collect 2000, then another 2000 and so on.  The problem is, you need to be able to specify a filter to get 2000 unique items at a time.  One would think that a filter of ID<2000 then (ID>2000 and ID<4000) then (ID>4000 and ID<6000), and so on, would be effective, but, the ID column is only delegable with the equal operator - not with comparison operators.  So...then the thought is - perhaps by date.  But that also is not delegable.

So what to do?  One workaround is to have a numeric column in your list and, with the use of Flow, have it contain a duplicate of the ID.  Then, numeric fields are delegable, so the above scenario would work for gathering the data (just substitute your new numeric column for ID above).  But wait...don't skip Flow!  You need flow to trigger on the creation of a new list item and for it to populate the new numeric column with the ID.  This only has to be done once on a new item.  Also, don't forget you will need to populate this new column for any existing items - so, you'll need a process for that...again, only once.  AND, don't get to thinking that perhaps you can short-circuit the process and just create a calculated column in SharePoint that is based on the ID - calculated columns are not delegable.

So - with all the above in place, you can, within your app, gather all the data.  You will have to guess, or constantly change your app, depending on the number of items in your list.  There is no function in PowerApps for SharePoint that will tell you how many items there are.

ForAll([0,1,2,3],
    Collect(myLead_Data, 
         Filter(Lead_Data, dupID >= (Value * 2000) && dupID < ((Value * 2000) +2000))
         )
)

If your list has less than 6000 items, then your new local collection (myLead_Data) will have that many items in it.

You can now change your formulas to work with the myLead_Data collection rather than your datasource directly.

SO, specifically, with your posted formula:

SortByColumns(
   Filter(myLead_Data,
        If(varAdmin, true,
           AssignedTo.Value = varUser ||
           LeadMember1.Value = varUser ||
           LeadMember2.Value = varUser ||
           LeadMember3.Value = varUser ||
           LeadMember4.Value = varUser
           ) &&
        Status.Value = SelectedStatus || If(SelectedStatus = "All", Status.Value in "In ProgessNotStarted")
    ),
"Created", Descending)

                      

This would give you all items for varAdmin and also filter the collection based on the SelectedStatus.

Okay...all that said.  We focused on a workaround for dealing with getting all the records in the list by applying a duplicate ID column and letting Flow fill in the value.  You can also use this type of logic to turn complex columns (such as your Status column) in to a duplicate text column - text columns are delegable.   BUT remember delegation has to do with filtering/searching/limiting on an entire list - you are still limited on how many records are returned from any function, regardless of delegation.

You just have to plan around the limits and consider the design.

I hope this is clear and helpful for you.