Apply to each - Parallel Processing - 50 limit

[Marty3012]

Hi,

I posted this before but for some reason i cant find it within any of the forums. 

I have the following flow;

Description

1 - Connect to an excel sheet and get all rows

2 - Within the Apply to each, loop through each row. (Set to Parallelism)

3 - Set condition that checks if the 'Expiration Date' field for each row equals the current date

4 - If so, set off approvals

5 - If it is approved then the comments provided are sent back to the excel file into another field.

All is working well. However there is the limit of 50 parallel tasks. After the dates are compared I could have between 1-100 rows returned. One possible solution I thought of was to create 2 copies of the Apply to Each loop described above. I could then have a condition that checks if there is say 57 rows returned, one loop could process 50 and the other could process 7. Would this be possible?

I've tried testing using variables and count functions but I'm getting nowhere. Any help appreciated.

Thanks

[Pstork1]

Other than the performance implications.  Why do you need to process all the rows in Parallel?  If you have 57 rows and set Parallel to 50 it will still process all 57 rows.  It will start processing for 50 of the rows and then start the final 7 as soon as other rows finish. So all 57 will be processed without changing anything.  They will just be throttled so that no more than 50 are being processed at a time.

[Marty3012]

The reason for parallel is that for each excel row there is an 'email address' field which is used to send the approval. For example, if 57 rows are returned then 57 different approvals must be sent out in parallel to 57 different email addresses on that particular day (Expiration Date).

So for in the example I gave, 7 approvals would not go out on the current day (Expiration Date) as they would have to wait for the other 50 to finish.

thanks for your reply

[Pstork1 · Accepted solution]

Ok, got it.  In that case I think you'll have to put an intermediate Loop between your Excel List action and your main processing Loop.  Prior to that Loop create two array variables. Inside the loop take the values from the first 50 objects and append them as objects to the first array variable.  Once the count reaches 50 append the rest to the second array variable.  Then create two parallel Apply to Each Loops for your main processing.  Process the first array in one and the second array in the other.  Set both for maximum parallel of 50.  If you have more than 100 you'll have to add a third variable and parallel loop.

[Marty3012]

Due to holidays and trying to work this out, I have eventually created a work around.

First of all, the flow which i created initially was too slow as the condition within the loop was being evaluated on each Excel row. This was causing a problem with the limit of 256 lines when Flow reads Excel. Instead I created a series of filters so that I got all the rows I needed and then fed them to the loops (see below);

I created 2 Apply to each loops. The 1st one reads the filtered data but only the first 50 rows. This is done with the following Expression:  

This is also with the Apply to Each concurrency control set to 50.

The 2nd loop does the same but for rows only after the first 50:  

Again, concurrency control is set to 50.

I could then add a 3rd apply to each with the expression  and so on.

Downside of course is that you have to keep creating more Apply to Each loops if you are working with a lot of data.

I've tested this and it works.

[Marty3012]

After further testing I realised that my expressions were incorrect. To use multiple Apply to Each functions to get round the 50 parrallel limit, you would use the following expressions instead;