Set Variable in Apply To Each Concurrency Control Message

[hilary_stoupa]

When I set a Variable inside an Apply To Each loop, I get this message:

However, when I go to set the concurrency control, as per the message, I see that the loop will execute sequentially by default, with Concurrency Control turned off:

I am confused by this - I feel like it is simpler to just leave Concurrency Control off, but the message makes me think that is the wrong decision and there is some subtlety here I am missing. 

[ScottShearer]

@hilary_stoupa

I just run a bunch of tests where I retrieve 200 items from a SharePoint list and loop through them incrementing a variable inside of the loop.  I ran test with the concurrency control set to off and turned on and set to 1.  While no two tests produced identical results, I didn't notice any significant difference between the two.  Based on my tests, I agree with you that just leaving the setting at the default Off setting makes the most sense.

[Jcook]

Hello @hilary_stoupa

Leaving concurrency turned off has a default value of 20. You must turn on and set the concurrency to 1 if you use a Set variable inside a apply to each loop.

You can alternatively use a Compose Action shape. Compose is not affected by concurrency

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others

[Jcook]

Hello @ScottShearer

Unfortunately it’s only the Set variable that is affected by Apply to each loops.

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others

[hilary_stoupa]

@Jcook - can you direct me to the documentation that describes this limit of 20? I did a quick web search but am not easily finding anything. Thanks!

[Jcook]

@hilary_stoupa
https://flow.microsoft.com/en-us/blog/gateway-ha-increased-apply-to-each/
The article does not say 20 but does mention that by default flow will run the loop sequentially.

If you use a set variable inside the loop without putting concurrency to 1, will make the set variable strange.

The reason behind this is that the variable is being set concurrently meaning multiple values from the loop will be set at the same time.

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others

[hilary_stoupa]

@Jcook thanks for the additional info - but I still don't understand. If the Apply to each runs sequentially by default, isn't that the same thing as concurrent with parallelism set to one?

That is, let's say the loop will run 10 times. And inside the loop, I'm setting my variable. If the Apply to each runs sequentially, doesn't that mean that the behavior will be:

Loop runs first time

Variable is set to first value

First Loop completes

Loop runs second time

Variable is set to second value

Second Loop completes

Etc...

Or am I somehow misunderstanding the meaning of "sequential" here?

[Jcook]

@hilary_stoupa 

The best way I can explain is; Here is my example:

Here are the details..

Lets say at Acme warehouse

There is 1 Database that needs to be filled with data daily

The records that need to be entered are on pieces of paper in a jar

Some may be duplicates as there are thousands of records

Lets say there are 5 workers who are assigned to go and enter these records into the database

The flow will be like this, each worker(5) will do the following at the same time

1. Grab 1 piece of paper
2. Query the record to see if the record exists or not

If record exists - Do nothing

if record does not exist - Enter the data in

------------- ------------ ----------- ---------- ---------

Now lets see how this work flow will work if we hypothetically say Concurrency set to 5: each column will represent a worker grabbing and querying the record at the same time -- each cell will have the record that worker is querying

Bananas	Tooth Paste	Tape	Gum	Carrots
Apples	Raspberries	Fish	Fish	Fish
Gum	Gum	Milk	Milk	Juice

Green = data was not found in query > and written to database

Yellow = data was not found in query > and written to database

Red = data was found > not written to database

So as you can see from the table above, in the second row. If all 5 workers grab a piece of paper and query it all at the same time.

The 3 workers that grabbed Fish will all query the database at the same time. And since Fish does not exist already in the database they will all write it to the database at the same time.

And the reason why Gum in the third row did not get affected by this is because it already existed in the database.

I hope this makes a bit more sense now. 

Using a Set Variable will not always cause you issues. It all depends on what you are doing inside the loop. But as a best practice. Its always a good idea to set the Concurrency to 1 when using a set variable to avoid things like the example above.

And again the Compose action shape does not get affected by the loop. Only "Set Variable" does.

Thanks

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others

[DavidYack · Accepted solution]

If the default is truly sequential, not sure how that matches the warning message.  Sequential can't mean concurrent greater than 1 so something is wrong either the warning or the label of the default setting

[Anonymous]

@Jcook, I have seen a couple of times "Compose action shape does not get affected by the loop" with little explanation of what that means or example of how you use different than using Set Variable.  Can you explain?

In my application, I am gathering metadata on a filtered set of files from blob storage.  I am using "Append to array variable" for each file.  I "Compose" a json object in each iteration then do the "Append to array variable" to a variable outside the loop. 

Right now, I have Concurrency set to 1.  Is there a way to do the same where I can increase the Concurrency level?

Thanks,

David

[Anonymous]

I think this has a lot to do with what is happening within the Apply to Each and Microsoft just set it as a default popup whenever someone uses set_variable within an apply-to-each loop. 

Solved: Concurrency issues with set variable - Power Platform Community (microsoft.com) shows an issue where different branches are using the same variable to set values (for different items) but the value of the variable is being used across the branches.  

(or a much scarier reason, somehow set_variable isnt local to a worker? So if one concurrent worker is setting a variable, another worker can somehow retrieve the value.)

[testasdfasfsafa]

Is the Append Variable  action (string or array) affected as well by concurrency?