Mass moving files, folders from to other folder on same remote

Help and Support
1

Hi, i want to move a lot of files, and folders to another folder on the same (S3) remote, but i don't know how.
-The official web gui of the remote provider is useless.

  • rclone mount remote:/ ~/cloud/ --buffer-size=256M --vfs-fast-fingerprint -v on linux machine, moved with the supplied filebrowser, but it freezed for minutes while Copied (server-side copy) to:+deleted pairs appeared, not actual move. needed to sigterm rclone after minutes to make something happen.
    -rclone move works only one by one
2

s3 does not support move.
https://rclone.org/overview/#optional-features

and a snippet from rclone backend features aws:

	"Name": "aws",
	"String": "S3 root",
	"Features": {
		"DirMove": false,
		"Move": false,
3

It is idrivee2, but S3 compatible.

What about this? amazon s3 - Moving many files in the same bucket - Stack Overflow

4

what about it, specifically?

5

the highest rated answer writes about "mv" command that is supported by the official cli

6

about that rated answer, not sure it is relevant to your situation, as that is using a filter.
i think i read something about aws does some server side trickery to reduce the total number of files that need to be iterated, copied and deleted.

as far as i know, s3 move is copy+delete, or try --magic :wink:

aws s3 mv s3://zork.movetest/source s3://zork.movetest/dest --recursive --exclude "*" --include "*.txt" 
move: s3://zork.movetest/source/01.txt to s3://zork.movetest/dest/01.txt
move: s3://zork.movetest/source/02.txt to s3://zork.movetest/dest/02.txt
move: s3://zork.movetest/source/03.txt to s3://zork.movetest/dest/03.txt

when i run that same command with --debug,
for each file, there is a put and delete. here is a log snippet

PUT /zork.movetest/dest/0.file
DELETE /zork.movetest/source/0.file

https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html
https://www.idrive.com/cloud/guides/operations
there is no api operation for move

7

It has to do each file individually Copy then Delete - that is the way the API works.

However rclone move can move a directory so you can just run it and wait for it to finish.

Note that server side copies are relatively expensive operations on S3 backends.

8

I see. 1 serverside copy delete in each 2-3 minutes. I should have download the files, and re upload them, it would be much faster :smiley: (really)

I don't want to move directory, i want to move files from multiple directories to another single directory.

1 Like
9

true, my original post did a move from dir to dir.

however, the OP made mention of a post from stackoverflow, that used the aws clii and included --include
so i tried to mimic that.

just to note, that wasabi does not charge for api calls

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.