Chunking an array down to manageable groups

[Anonymous]

I have an array of email addresses like this:

[

"email@email.com",

"another@email.com",

"qwerty@abc.com"

]

The array can contain anywhere from 1 to thousands of emails. I want to change the array so that the emails are in groups of 300 e.g.

[

{

"email@email.com",

"another@email.com",

"qwerty@abc.com"

...to 300

},

{

"more@email.com"

...to another 300

}

]

What is the best way to achieve this?

[eliotcole · Accepted solution]

Quick Answer

You will need to use the take() and skip() functions in an expression.

It's best to do this with the length() of the original array in mind.

If you're already pretty good with the logic side, then that should be all you need.

Solution

Here's my flow to handle this, you will replace the "3" with "100".

Full Flow Logic

This could be minimised a bit by moving some of the actions into expressions, but this shows you the logic I was aiming for.

I will edit in explanation shortly.

You can read on if you like.

Logic

Essentially, this works out how many runs are needed to divide the array into chunks of 3 (100 for you), then, for the set amount of turns, it will set the arrayVAR from the arrayWorkerVAR, with increasingly less (by 3) items than before.

Each run, it will also place those three deleted items into an object that will be placed into the newArrayVAR array.

Expressions/Code Used

Initialize divvyVAR

This looks more complicated than it is (see explanation) but essentially a normal division doesn't see the "remainder" so I had to see if the number of loops needed one more.

if(contains(string(div(float(length(variables('arrayVAR'))), 3)), '.'), add(div(length(variables('arrayVAR')), 3), 1), div(length(variables('arrayVAR')), 3))

jsonObject

This uses the take() function to take the first 3 items from the arrayVAR variable and create a new array.

The code:

{
  "chunk": @{take(variables('arrayVAR'), 3)}
}

The expression within:

take(variables('arrayVAR'), 3)

Set arrayWorkerVAR

This uses the skip() function to skip over the first 3 items in the arrayVAR variable, to create a new array.

The reason the flow needs an extra array here, is because you cannot set a variable's value from itself.

skip(variables('arrayVAR'), 3)

Detailed Explanation for the Long IF Statement in divvyVAR

Explaining quickly the basic functions:

• length() - This counts the number of entries in an array or characters in a string.
• float() - This formats a number in to a floating point type, allowing decimal points in any further actions.
• div() - In two parts separated by a comma this will divide the second number from the first.
• string() - This converts any other data type into a text string.
• contains() - In two parts this checks the first part (a string or array) before the comma, for the second part after it.
• add() - Adds any numbers (split by commas) to each other.
• if() - In three parts separated by commas, if the first part is 'true' this will return the value of the second part, but if it is 'false' it will return the third part.

So ... if I get a floating point number from my division, then I will need an extra step to catch the remainder of the items in the array ... meaning:

IF - The result of dividing the number of entries in the original array produces a number with a decimal point, then the contains() function will return TRUE. 

THEN - Add 1 to the number of a *normal* (not floating point) division of the length.

ELSE - Just take the normal division of the number.

[Anonymous]

This is fantastic thank you, I will try this out today and let you know how I get on.

[Anonymous]

You, sir, are a fantastic human being. This does exactly what I wanted! Very comprehensive answer and well explained, thank you so much.

[darren-h-gill-m]

I know someone has already offered a solution, but there is another way too.

My solution uses a range(i,n) expression to generate a list of index offsets into your array.

This is used in a simple for-each loop along with the take() and skip() operatations that the other solution mentions. The benefit of this approach is that I can avoid setting variables and so allow my loop interations to run concurrently!

Overview:

Step 1 - "get-segment-items"

In the example I just did a compose on an array of Id numbers I got from a SharePoint list

I had something like 640 records and I want to spilt it into chunks of 30. 

Step 2 - work out how many chunks needed

I used a compose (how-many) with this expression:

length(outputs('get-segment-items')?['value'])

NOTE: Could easliy skip this and just use the expression later. 

Step 3 - Directly express the list of chunk indexes needed

range(0,
  add(
    if(
      greater(
        mod(outputs('how-many'),30),
        0
      ),
      1,
      0
    ),
    div(
      outputs('how-many'),
      30
    )
  )
)

The range() expression nees a start index and a length. The start is easy, zero.

The number of chunks looks overly complex (as in the other posed solution) but is essentially just the div() of the number of items by a hard coded size, in this case 30.

That would allow you to get all the items in full mutiples of 30, but we need to add one more if there is a remainder after doing the integer divide. The other solution converted the number to a string and looked for a decimal point, that works too but here I keep everything in numbers. The if(<test>,1,0) is used to determine if an extract litem is needed in the list. It would be expressed more more simply as:-

If the total items divided by 30 (integer division) has a remainder then  add 1

The result of this on my 640 items is 22 iterations (#0 to #21)

Step 4 - Pick the items with Select

Inside the loop you need to reference back to the source array using take and skip 

take(
  skip(
    outputs('get-segment-items')?['value'],
    mul(30, item())
  ),
  30
)

The item() value here will be the number 0 to 21 from the range.

So no items will be skipped in the first iteration but by the last (no 21) 630 items will be skipped and the take expression gets the next 30 (or part thereof) items from the source array.

The bonus is that you can set the for-each cocurrency level and get parallel processing of your loop. No variables outisde are needed.

[RameshMukka]

@darren-h-gill-m and @eliotcole Woww, I landed on the right post and helped me a lot. Greatly appreciate your solution.

Thanks,

Ramesh

[eliotcole]

The chunk() function might also work these days.

Would need to revisit. 🙂

[RameshMukka]

@eliotcole How did I miss that out. Thanks a lot.