Mass Detection of Files, Files getting duplicated Analyzed

[HansenvomDeich]

Hey Folks,

in my Project i running an complex Flow, which detects new Files in an Folder (Sharepoint) an analyze them with AI Builder to store the Data in Dataverse.
Also the readout is scriptet in a Dataverse Helplist to get an Export of the whole Files which are getting Analyzed.

The Flow is not running in a big Apply to Each Loop, it gets analyzed File by File.
I Set the Degree of Parallelism in Sharepoint Trigger When a file is created or modified (properties only) at 20 to reduce the Concurency of Files.

But when i analyze 500 Files i get 37 Duplicates in it (and i run that several Times with that Amount of Files).
When i anayze 200 Files i get 16 Duplicates in it.
And on 30 Files i get 8 Duplicates in it.

So not only do I have a corrupt database from which I have to throw out the duplicate files, but I have (supposedly) paid for the analysis of 61 additional Files.

How do you deal with this Case, is there an official Document from Microsoft regarding the problem and how to work around it?

[Tjan]

@HansenvomDeich  It might help to share how your flow is set-up! It will help me check what's happening.

[HansenvomDeich]

Hi @Tjan , 

i try to show you

 

In fact i thinkthese are the Steps which are interested:

[Tjan]

@HansenvomDeich  Thank you so much for the screenshots. From what I noticed immediately is that you use the trigger 'when an item is created or modified'. This means that when a Sharepoint item changes (no matter which column), this flow will be triggered. This may cause your problem because an item might trigger this flow multiple times. The first time being the creation, the second time being a modification and the third time as well etc. 

My recommendation would be to use another trigger. That is, "When an item is created". With this action you will not have this problem. If you however want to still trigger this flow if an item is modified (but only certain conditions) then try to set up trigger conditions.

[HansenvomDeich]

Hi @Tjan,

thanks for your help at first.

I know exactly what you mean, the Folder ive created is a Manual Folder, so i drop in there 200 Files, and wait until they got analyzed (after that the file is copied by the flow in target folder and get deleted from that "Auto Trigger" Folder)

So i can say 100 % that there is no modified Action in that Folder. Only created and only one time per Dataset.

[Tjan]

@HansenvomDeich  Ah alright! I also saw that you turned on 'split-on' already. That is also something that would help in your case. Then, one last question: do you have any 'for eaches' in your flow?

[HansenvomDeich]

@Tjan 
I think about why i turned that on.
I thougt i try to test first, build all that things, in one big for each to get start und finish messages, but it doesnt work as i want, or better to say i didnt have enough skills to build that. so i decide to use it as single flow for every file.

After Teams Collapse because ive got to many messages (start finish msg), i build the help table in dataverse to get feedback about the status of the runnings.
(the flow was original planned for a usecase with 100 in month not in a minute and i dont got time to rebuild that, because unfortunately i am an employee not an automate,app builder in our company)

But i think i have tested it, without turned it on, because i read about the function of that nipple and it doesnt work, but i am not sure.
I try to switch it off and look what happened.

There are nor for each Loops in the Flow, only multiple conditions to check plausibility of some actions.

Thanks for your Hints.

[HansenvomDeich]

Now i know why i turned it on:
With turned that Option on i can use the identifier from file created without for each loop

With turned it off, there i can´t run the flow when i wanna now use the identifier.

i have to build for each with body/value in for each and identifier in Action

[HansenvomDeich]

Hi @Tjan ,

i think over, and rebuild the flow (as it should) with an for each loop.

Now i get the exact Outputs of the Elements which will be readout e.g. 20 Files > 20 Results.

But the Problem, before i get a readout of 200-500 Files in about 5 to 10 Minutes because of the parallel run from Flow Trigger by 50.

Now with the For Each, its getting 50 Files in the Trigger Action, but the for Each reduced the Speed of readout to 1 by time.(50 Files in FlowTrigger, reduced by one by one in the For Each Loop).
So that now i need 10 Minutes for 20 Files 😄

[Tjan]

@HansenvomDeich  Ah, glad you have a solution via this way! Yeah.. This performance problem will occur if parallelism is set to 1.. If the time doesn't bother too much, you should keep your solution like this! Always better than having double data entires AND double charged AI Builder credits