Find duplicate files in a SharePoint folder via substring

[jdlindsey]

Hello! I am new to Power Automate and naturally I'm jumping in with a complex task.

I'm pulling files from a SharePoint folder and need to find duplicates based on a substring in the file name.

I have file names with hashes that look like this:

ABCD__0921455336__C2__005056B278AB1EECA9935156220494E6.PDF

ABCD__0921455336__C2__005056B278AB1EECA8881156243604E6.PDF

I need to deduplicate files based on that 10 character string starting at 6. I am having trouble finding a way to do so - do I need to filter an array? Use a union? I'm not sure how to iterate through the names to find the duplicate substrings.

Thanks in advance for saving my sanity!

[Nived_Nambiar]

Hi @jdlindsey 

is the below characters are the one by which u need to find non-duplicates ?

Thanks & Regards,

Nived N 🚀

LinkedIn: Nived N's LinkedIn
YouTube: Nived N's YouTube Channel
Blog: Nived Nambiar's Blogs

🔍 Found my answer helpful? Please consider marking it as the solution!
Your appreciation keeps me motivated. Thank you! 🙌

[jdlindsey]

Those ten digits are the identifier - everything else is useless in this case. I'm selecting the substring like this:

substring(items('For_each')?['Name'],6,10)

So I need to iterate through each file in the folder and check each to see if that substring is part of another filename. The substring doesn't change position - it is always 6,10.

Thank you!

[Nived_Nambiar]

Hi @jdlindsey 

Ok based on that you need files of first occurance right ?

Thanks & Regards,

Nived N 🚀

LinkedIn: Nived N's LinkedIn
YouTube: Nived N's YouTube Channel
Blog: Nived Nambiar's Blogs

🔍 Found my answer helpful? Please consider marking it as the solution!
Your appreciation keeps me motivated. Thank you! 🙌

[jdlindsey]

It doesn't matter which one I keep, just that I keep one. The number of duplicates vary, and some are unique. Here's a sample:

[Nived_Nambiar · Accepted solution]

Hi @jdlindsey 

Assuming files are at this location

Flow design

use list folder to list all files present in it

use select action to get substring from all files in array

Initialize FinalArr to store files names in array

use apply to each loop to loop through distinct subtring values 

Use filter array to filter list folder output again with substring equal to current iterating substring

use append to array to store the file name (first occurance from filter array)

outside loop use compose action to see the output list of files 

Results

Hope this helps !

Thanks & Regards,

Nived N 🚀

LinkedIn: Nived N's LinkedIn
YouTube: Nived N's YouTube Channel
Blog: Nived Nambiar's Blogs

🔍 Found my answer helpful? Please consider marking it as the solution!
Your appreciation keeps me motivated. Thank you! 🙌

[jdlindsey]

I will give this a shot, thank you!

[jdlindsey]

One more question - do you have a screenshot for the select action? I'm having trouble with that one. Thanks again!

[Nived_Nambiar]

Hi @jdlindsey 

sorry , i have updated the answer with select action 🙂

Thanks & Regards,

Nived N 🚀

LinkedIn: Nived N's LinkedIn
YouTube: Nived N's YouTube Channel
Blog: Nived Nambiar's Blogs

🔍 Found my answer helpful? Please consider marking it as the solution!
Your appreciation keeps me motivated. Thank you! 🙌

[jdlindsey]

IT WORKS! Thank you so much!