Move folder (with large files and subfolders) from 1 site to another site (within the same tenant)

[Femke1]

Hello everyone,

Hope you are more experienced on this topic than me 😅

My situation:

I have the following situation: there is a specific folder in a sharepoint site (group) that can contain multiple folders and/or files. Those files can be large (2 GB). In the flow I can get the ID and ItemId of the folder (by using 'Get folder metadata using path').

What I want to do:

What I want is to move the folder with it's contents (small + large files + maybe folders with files in them) to another Sharepoint site.

Different methods to do that (as far as I know):

I can do that with different methods, I've already done some research:

1. Move folder

2. Copy folder (and delete old folder/files)

3. Create files (in new folder) + delete old folder/files - but then I miss creating the subfolders?

4. Copy files (in new folder) + delete old folder/files - but then I miss copying the subfolders?

5. Move folders/files with HTTP request

I got errors trying method 1 and 3:

I had different errors when testing with some of the above methods (1 and 3, I did not test 2, 4 and 5 yet). Most of them had to do with filesize. I stopped to look into other errors (such as: Missing or mismatched field definition on the destination list for source field of Failed to verify the existence of destination location) for now, because I first want to know which method is the most reliable one for copying/creating/moving large files (such as files that are 2 GB in size).

So I started to dig into that, but I couldn't find an overview of limitations using these methods. Spent quite some time trying to find the information in pieces, but I'm starting to drown 😵.

My question:

Can somebody tell me if it is possible moving folders (with large files of 2 GB + subfolders)? And which method can I use best?

Thanks a lot in advance 🙏

Femke

[fchopo]

Hi Femke,

Honestly, I would recommend using other tools different than Power Automate, as you will probably hit throttling limits (file size, number of requests).

Options:

1) Use 3rd party tools like ShareGate.

2) Use OneDrive for Business to sync docs with your laptop: Sync SharePoint and Teams files with your computer - Microsoft Support

Steps:

1) Sync source SharePoint doc library with your laptop (OneDrive for Business)

2) Sync new SharePoint doc library with your laptop (OneDrive for Business)

3) Move items from 1 to 2

Hope it helps!

Ferran

[Femke1]

Hi @fchopo ,

Thank you for your reply (and time and effort), but my wish to move a folder is not a 1 time thing. It is part of an existing flow (maybe I did not explain that well enough, I'm sorry 🙈). This flow runs several times a day.

OneDrive is not relevant I guess?

Extra info: I currently use the 'upload file' in Forms, but that has a limitation of max. 10 files of 1 GB each. We are now experiencing trouble with that, that is why I'm searching a different way of doing things.

The test flow does this now:

- One person submits a form (and asks for a personal folder)

- When the form is submitted, the flow will create a folder on a SP site that exists for this purpose only (not my OD)

- The folder gets a name with the employee-number of the responder of the form (+date & time) - so is unique

- The responder gets edit control over the folder

- The responder uploads folders and files in this personal folder

- After 1 hour, edit control is removed

- A group of people get view control over the folder

- An approval is send to the group of people (are they happy with the content?)

- After everyone of the group approved, the folder with the contents must be moved to a different site

- Nobody of that group has any access whatsoever to that different site (but other people have)

And here I get stuck, because I've reached some limits with the above methods as well. So, maybe I can get the information about limitations ahead, before I go and try all the different options (and resolve different errors etc.) 😊

Femke

[fchopo]

Hi @Femke1 

I would definitely try to move objects using "Send an HTTP Request to SharePoint". Here you can find an excellent article on how to do that: Cross-site collection file copy with SharePoint Online REST APIs in Microsoft Flow.md · GitHub

Honestly, I haven't tried copying an entire folder instead of a specific file (as it's shown in the sample).

Hope that helps!

Ferran

[Femke1]

 Hi @fchopo ,

Thanks again for your reply and help. This is one of the articles that I've come across during my research 🥳

Do you maybe have information about the limitations? What did you mean by 'you will probably hit throttling limits (file size, number of requests)'?

In the meantime, I think I'm going to go ahead and try that method (HTTP request). Thanks again 🙏

[fchopo]

Hi @Femke1 

When I talk about throttling limits I refer to those: Limits of automated, scheduled, and instant flows - Power Automate | Microsoft Learn

Regarding SharePoint, there are some limitations: SharePoint - Connectors | Microsoft Learn

Hope it helps!

Ferran

[Femke1]

Thank you for the links!

I'm trying to replicate what zplume did in his article (so I can accept your answer as solution), but I keep getting errors:

Do you have a clue? I'm not getting anywhere 🙈. I'm going to built my replica again I guess 🙂

[fchopo]

Could you share the error you get when running the workflow?

[Femke1]

It's the error in the screenshot. It doesn't give me more error details, cause there is an action in the beginning of the flow that gives an error (it errors if I'm not a certain person - which I'm not, so I gave the next action a 'configure run after' and ticked all the boxes) and those error details are shown...

So I can only tell you this:

- In the Apply to each loop (Wait for copy job to complete'): ActionFailed. An action failed. No dependent actions succeeded.

- In the HTTP request to SP 'GetCopyJobProgress': BadRequest.

Hope this helps? I don't know a way to get the error details of a particular action?

[Femke1]

Been doing other stuff, but I'm back now trying to resolve this. I did find more details 🙂

In the body of the inputs of GetCopyJobProgress I can see this:

{
"copyJobInfo": {
"EncryptionKey": "zZLYtnSxP8FPM4M1+SOQNKei2alHgg3PvSsUrdRPgzM=",
"JobId": "ae3132c0-ec9c-4f12-ae6f-289c20694ba7",
"JobQueueUri": "https://spodb1am1m006p3.queue.core.windows.net/3-2553-20231220-afa8aceae18341688f251809621caab4?sv=2...",
"SourceListItemUniqueIds": []
}
}

In the body of the outputs of GetCopyJobProgress I can see this:

{

  "status": 400,

  "message": "{\"odata.error\":{\"code\":\"-1, Microsoft.Data.OData.ODataException\",\"message\":{\"lang\":\"en-US\",\"value\":\"A node of type 'StartArray' was read from the JSON reader when trying to read a value of a property; however, a 'PrimitiveValue' or 'StartObject' node was expected.\"}}}\r\nclientRequestId: b27c9c6b-3b66-4287-9b8b-ddd0e06688ee\r\nserviceRequestId: b8c4f2a0-f00d-7000-a6ee-873009dda23a",

  "source": "https://xxxxx.sharepoint.com/sites/xxxxxx/_api/site/GetCopyJobProgress",

  "errors": []

}

I see that SourceListItemUniqueIds is empty, and in the comments I see Reza Dorrani saying something about the SourceListItemsUniqueIds.

I think I'm going to rebuilt a test flow from scratch and dive back in to this topic.

Any help is welcome 🙏 😉

[Femke1 · Accepted solution]

Update: I got some great help with the errors that I encountered earlier and now it works! We've tested it with files (20 GB, 1 GB, 2 GB, 5 GB, 500 MB and 12 GB in one folder with subfolders) and everything went smoothly (even with these numbers and large files) with just the action 'move folder'...

Sometimes you have to work very hard for a simple solution 🙈

[szaj]

Hi Femke1,

What was your solution? I am having similar troubles and would love some help.

[Femke1]

Hi @szaj ,

For me the Sharepoint action 'Move folder' worked just fine after solving some errors. I don't actually remember what I solved, but I remember it was quite easy after I took a good look at it.

First use 'Get folder metadata using path':

Good luck! 💪

Femke