merging array safely within parallel loop

[bdomzalski]

Hi,

Within Apply for each action I have another one that retrieves an array of objects. I want the loop to have concurrency enabled and need to merge all the retrieved arrays into a single one.

My solution is a popular one - have an array variable for results, create compose action to merge current results with the actual loop item (the retrieved "partial" array), set results to the output of that compose.

Spoiler

concept of flattening array

It seems to work fine but I have doubts if that is concurrency/thread-safe solution and will it always work as expected. I can easily imagine that multiple loop executions are halted after the compose action. This could result in overwriting the results when setting the variable by "later" loop execution.

I know that it can be achieved by adding another loop with Append to array variable action but I don't like the idea of introducing that loop only for concatenating arrays.

Maybe you know another method to flatten array which is concurrency-safe? Or maybe just more readable (those additional actions like the compose or the loop I'm no a fan...)?

Kind regards,

Bartosz Domżalski

[v-litu-msft · Accepted solution]

Hi @bdomzalski,

This method I first met in Jonh Liu 's channel, it indeed could work as appending an array into Compose action within an Apply to each and get output of the Compose could get an array, and there is no problem with it.

There is no document to introduce this method, it is a trick.

Generally, when we combine the array, we always use the Append to array action, it is standard and not get an error.

Otherwise, there is no other better way could do it like compose as far as I know.

There are all functions in Flow:

https://docs.microsoft.com/en-us/azure/logic-apps/workflow-definition-language-functions-reference#c...

Best Regards,
Community Support Team _ Lin Tu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[bdomzalski · Accepted solution]

Hi @v-litu-msft,

I just learn in practice that this approach (merging arrays in two steps: compose a union of a variable gathering final result with a partial result and in the next step set that variable) does not work correctly within loops with concurrency enabled!

Results are unpredictable because of the parallelism and might not contain all the partial results.

Spoiler

For-each loop concurrency enabled

So I guess we are left "Append to array variable" within an additional loop (on the partial result collection).

Kind regards,

Bartosz Domżalski

[v-litu-msft]

Hi @bdomzalski,

If it is not a very complicated process, for example, there is only one compose, concurrency test failure as you said, I have not encountered it. If it is merging array, using variables is more stable.

Best Regards,
Community Support Team _ Lin Tu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.