Nested Lookup Delegation Warnings (Dataverse)

[paulhaagsefa]

Can someone explain to me the Dataverse delegation behavior I'm seeing here?

I am preparing to do an update of multiple records without knowing the primary keys, so I need to perform a lookup for each record by the effective composite key (Plant, Sample Location, Sample Date, and Sample Hour) and I can't for the life of me figure out why I sometimes am and sometimes am not getting a delegation problem trying to use attributes of the update collection.

As an additional note: I get the exact same behavior in other looping mechanisms (e.g., AddColumns instead of ForAll).

Issue 1: Sample Location (Id)

When I tried to filter with:

Record.'Sample Location'.'Sample Location' = Update.SampleLocation.'Sample Location'

it failed to delegate. For some reason, using AddColumns to flatten the ID directly onto the Update object fixes this problem, despite the fact that the data type isn't changing and I can't really think of anything else that is materially different.

Issue 2: Sample Date

I can't seem to get around this problem. For some reason, it refuses to delegate a date filter for a property of the Update itself. You can see that if I take one of the values and put it in the context (as "test"), it will delegate that just fine, so it's not a data type issue. Furthermore, you can see that it will delegate other Update attributes, like TestHour (which is a whole number).

I can sort of get around this problem by pre-filtering the existing samples using everything but date. This is better, but is still pulling more data locally than should be necessary and will eventually stop working if the number of samples matching on everything but date ever exceeds the data row limit.

[poweractivate]

@paulhaagsefa 

You are not always given the yellow triangle warning. The issue might be continuing with or without a warning. Sometimes a warning is not given at all. For example, simply using ClearCollect by itself gives no warning. However, you should note that the function itself is subject to the data row limit even with no warning.

Try these steps

1. Click Settings

2. Scroll down a bit and check the Data row limit . Change it to 1 temporarily.

Now try your queries. If you get only one result and expect two or more, you know the data row limit is applying.

Note that there are also other kinds of limits as well depending on the kinds of controls or methods used. For example, note that a Collection populated by using a single call to ClearCollect is subject to the data row limit despite all inner statements being delegable, because ClearCollect by itself is subject to the data row limit. To test it more effectively and try to isolate the cause to delegation, use the Items property of a Gallery, since there may be other limits when using something else.

Try the above and see if you are getting intermittent results between one result and the expected number of results, or are always getting just one result.

[paulhaagsefa]

I think you're getting hung up on my workaround. The real question is: why isn't using the Update SampleLocation.'Sample Location' or SampleDate delegable? I can't think of  good reason why they shouldn't be, since they are fully materialized locally before entering the ForAll function.

[poweractivate]

@paulhaagsefa 

Does enabling this feature help? Settings -> Upcoming Features - > Record scope one-to-many and many-to-many relationships (if it does not help, turn it back off)

[poweractivate]

---

@paulhaagsefa wrote:

: why isn't using the Update SampleLocation.'Sample Location' or SampleDate delegable? 

---

Did you check with the steps I gave? Note that even if you remove all yellow triangle warnings from the editor, there may still be a data row limit issue or something is still non-delegable. Is it really intermittently working where it is always delegable when the warning is not shown. and not delegable when the warning isn't shown? Or is it the case that even when it doesn't show yellow triangle warning it's still also non-delegable?

Although I have not seen this happen, also check to see and be sure if the yellow triangle warning is given, but there is no observable issue in there being less returned results than expected.

Note that the absence of the yellow triangle does not guarantee that the data row limit is not applying in some way and the best way to be sure is to lower that data row limit to 1 and test from there. Note that the yellow triangle warning does not always show up and there are cases where it never shows up (intended behavior) yet the data row limit is still applying and/or there is still something non-delegable in there, even with no warning. It's up to you to lower that data row limit to 1 and test with sufficiently large enough samples to make sure some sort of limit is not being triggered in the way you wrote the formula or used the controls. If you want to really be sure, lower the data row limit to 1, and even test with samples of greater than 2000 records to be sure. 

[paulhaagsefa]

Thanks for that suggestion, but it doesn't seem to make a difference for either problem. Especially for the date thing, it should be irrelevant as that has no dependency on a relationship on either side of the equation.

[poweractivate]

@paulhaagsefa Do you observe a difference in the number of records returned when you try it when use the date and when you don't? I mean when you use the Date are the number of returned results equal to the data row limit when there should be more results, and when you don't use the date the number of returned results are greater than to the data row limit as expected? Is this always corresponding to the presence or absence of the yellow triangle warning i.e. are there always less results than expected when the yellow triangle is present, and are there always the expected number of results when it is absent? (the way to be sure is to lower that limit to 1. Then it returns only 1 result when there is a problem).

One issue could be that even when the yellow triangle has disappeared, it "hasn't" really.  If you are getting only one result returned after lowering the data row limit to 1 in all possible versions of your formula with or without the yellow triangle warning, then all versions of the formula you provided, even the workarounds, are affected by the data row limit and either contain one or more non-delegable components or are otherwise triggering the data row limit. I wanted to know this first before I could help you further. 

[paulhaagsefa]

Let's focus on the date one, since that is the one I don't seem to be able to get around and should be more cut-and-dry anyway. If I try to use the SampleDate property of the Update record, the editor always shows a delegation warning and when I test the app it is always not delegated (you can see it requesting all rows in the monitor). When I replace it with a date value literal or something saved in the context, then the delegation warning goes away and it does delegate / filter appropriately.

I guess the title of this question was a little misleading. I don't mean that I am getting different results for the same code, I mean I am getting different results for different code, when it feels like I should be getting the same result (e.g. what is the difference between using a singular date value from the context vs. what should be a singular value from the current record in an iterator?).

[poweractivate]

@paulhaagsefa It can get complicated because Coalesce, AddColumns, ForAll, possibly a number of other things, and for that matter the outer ClearCollect itself you have are all subject to the data row limit. In this scenario it may therefore make a difference if you are using a literal as opposed to a property because usage of the property may involve the additional traversal of a structure in a non-delegable way. Does another property than a Date work or does it still also produce the delegation issue?

[paulhaagsefa]

Yes, SampleHour delegates, as does the flattened SampleLocationId.