Extracting data from FTP to SharePoint with hourly record limitation

[Walter1989]

The information is being retrieved from the "Get file content" control from FTP (Image 01), converted into an array (Image 02), and then each record obtained creates a new item in SharePoint (Image 03). Several parameters are recorded every hour within a 24-hour range (Image 04). I only want it to create the item in SharePoint from the latest hour in FTP to the latest hour.

Image 01: FTP Control

Image 02: Compose

Image 03: Create element in sharepoint

Image 04: Information details from FTP Control

[Koen5]

Dear Walter,

I am not sure if I follow the entire story, but it sounds to me that you need to include a 'Filter Array' action.

This will allow you to filter the array based on a particular array element, in this case a date field it seems.

View example below. To filter based on a certain date limit you can use something like: "addDays(utcNow(), -1)", this expression filters out everything in the last 24 hours (-1 day).

Hope it helps, 

Happy flowing,

Koen

[Walter1989]

Hi @Koen5 ,

Thank you for the information, but it's not exactly what I was looking for. Let me explain that the automation creates items in SharePoint every 30 minutes. And the repository in FTP stores information from the last 24 hours, which means:

• If it's 10:00 AM today, it has information from 10:00 AM yesterday.
• When it reaches 11:00 AM today, it has information from 11:00 AM yesterday.

And every 30 minutes, it creates items in SharePoint. What I need is to avoid repeating the information that is created in SharePoint and only create items from the last recorded hour in SharePoint.

Note: For now, I'm leaving that automation there because the important part is to display it on a Power BI dashboard. There, I can remove duplicates more easily.

[Koen5 · Accepted solution]

Dear Walter,

I am thinking of 2 options:

• Match the sharepoint updates with the FTP updates ... so rather than doing 30min creation of items (which can create duplicates), do it every hour .. matching the FTP updates.
• Leave everything as it is regarding the updates and frequencies (you may not be in a position to change this), but de-duplicate any created items in the sharepoint. You can do this via a union() expression, of which there are plenty of examples around in forums. Any duplicates you find in the sharepoint, just delete (or mark as duplicate), just make sure you keep at least 1 original.

https://www.flowjoe.io/2020/12/17/remove-duplicates-from-a-power-automate-array-with-one-action/

Hope the suggestions help,

Happy flowing,

Koen

[Walter1989]

Hi @Koen5 ,

Thank you for the information and the link, I was actually thinking of doing that. Creating an automation to validate and eliminate duplicates.

Thanks for the support and guidance.

Have a great day!!

Walter Mosqueira