Using Power Automate to create a list from json file, every list item is being entered 3 times

[Diviner13]

Hi All,

I am relatively new to Power Automate.  I am having an issue here with my Power Automate flow.

I created a flow, that reads a json file generated by another process.  Then the json file is dropped in the SharePoint document library, the flow kicks off, reads the json, reads items out of an item list in the json file, and creates a list items in a SharePoint list.

When my sample size was small, (i.e 5 items) this works great.  Now I have increased my sample size to 100 items, and the json list that is generated has 300 items.  I have 3 copies of each item in the list.

I tried updating my flow to check to see if current item already exists in the list before creating it.  That drops number of items created from 300 down to 188 items.  It almost seems like the script is engaging in parallel processing in the For each loop.  Is there a way I can prevent this from happening?  It's more important that the data set is accurate, than the processing to be quick.  I learned through this process that Power Automate is anything but quick.

Here are some snippets out of my flow:

Then basically we check to see if we are looking at a json file, then start processing it:

We check to make sure the item has not already been created....

We then do other processing to ensure things like creating and adding attachments, etc.

But this is the version of my flow that reduces the number of items from 300 down to 188... but that of course is not good enough.

Regards,

Steve

[Redders]

HI @Diviner13,

I don't know if I will be able to help however, I will try. I was encountering something similar when attempting to remove duplicate items from an array. I'm not sure if all of this is necessary now but it works. I don't know if this will help you but this is what I did. 

I was processing fuel transactions received via JSON, I was looking at a weeks worth of transactions but I only wanted to update my SP list if there was a new invoiced transaction (which meant that there was multiple transactions with the same invoice number). 

I created a separate list which just contained the invoice numbers, so that I didn't need to start finding unique numbers from my transaction list.

I used get items from this list and added them to a "SPInv" array

I added all of my JSON invoice numbers to an "JSON" array

I added a 2 separate composes, 1 for each array and used a union expression on each, here is the example for the SPInv 

union(variables('SPInvNo'),variables('SPInvNo'))

I then added a filter array

Outputs from JSONCompose

Outputs from SPCompose      "Does not contain"       Expression "item()"

This gave me all of the new invoice numbers that did not exist on the SP list

I then added the number/s to the SP list (so that they were there for next time) and added another filter array.

Body from original JSON

Body from Filter array 2    "Contains"    Invoice number (Dynamic content from my original JSON)

I then created an item for every transaction from the filter array as this then did not contain any records that I had previously added.

I think that this would only work IF you have some way of identifying unique records

I hope it helps anyway

━━━━━━━━━━━━━━━━━━━━━━━━━

If I have answered your question, please mark your post as Solved.

Solved questions helps users in the future who may have the same issue or question quickly find a resolution via search.

If you liked my response, please consider giving it a thumbs up. Thank you!

[Diviner13]

Thanks for your response.  As I under stand what you are describing here is that your data source, the json, had duplicate invoice numbers.

In my case, what I am doing to taking data from another system where each item has a unique identifier.  One of the first things I did was to verify that my json file that I am importing has 100 unique items each one with a unique ID.  This is how I discovered that, after my import was complete, I grouped my list of 300 by my imported unique identifier, and saw that I had 3 copies of the same item.  Each of the internal SharePoint IDs was unique as well, so it meant that each item was imported 3 times.

Thanks for your assistance, but I don't think this will work for me.

I'm wondering now if I should create a second flow, that I can trigger manually to go in and clean up all the duplicate items, but I would rather have it import correctly the first time, than have to rely on fixing data after the fact.

[Redders]

Hi @Diviner13,

Yes, that was the case however, when I was writing to the new list, I wanted to ensure that I wasn't copying the same transactions that existed in the list.

In either case, I think that you can use the union expression, I found it very useful, it certainly eliminated duplicate values from the same source

I hope that it helps you somehow

━━━━━━━━━━━━━━━━━━━━━━━━━

If I have answered your question, please mark your post as Solved.

Solved questions helps users in the future who may have the same issue or question quickly find a resolution via search.

If you liked my response, please consider giving it a thumbs up. Thank you!

[Diviner13 · Accepted solution]

Okay, you aren't going to believe this.  After implementing the union functionality, I was still running into the same issue.

Turns out I had previous versions of my flow still turned on.  So I had three flows running concurrently building my list.  Once I turned off the other two, everything worked as expected.  

Another question... is there such a thing as a booby prize on this forum?

[Redders]

Oh dear, we all live and learn

[Redders]

@Diviner13,

It might be worth adding my first comment to the solution. Whilst it didn't necessarily resolve your issue (with you obtaining the booby prize, haha), it is still good information for people struggling with similar issues