Mass moving files, folders from to other folder on same remote

Hi, i want to move a lot of files, and folders to another folder on the same (S3) remote, but i don't know how.
-The official web gui of the remote provider is useless.

  • rclone mount remote:/ ~/cloud/ --buffer-size=256M --vfs-fast-fingerprint -v on linux machine, moved with the supplied filebrowser, but it freezed for minutes while Copied (server-side copy) to:+deleted pairs appeared, not actual move. needed to sigterm rclone after minutes to make something happen.
    -rclone move works only one by one

s3 does not support move.

and a snippet from rclone backend features aws:

	"Name": "aws",
	"String": "S3 root",
	"Features": {
		"DirMove": false,
		"Move": false,

It is idrivee2, but S3 compatible.

What about this? amazon s3 - Moving many files in the same bucket - Stack Overflow

what about it, specifically?

the highest rated answer writes about "mv" command that is supported by the official cli

about that rated answer, not sure it is relevant to your situation, as that is using a filter.
i think i read something about aws does some server side trickery to reduce the total number of files that need to be iterated, copied and deleted.

as far as i know, s3 move is copy+delete, or try --magic :wink:

aws s3 mv s3://zork.movetest/source s3://zork.movetest/dest --recursive --exclude "*" --include "*.txt" 
move: s3://zork.movetest/source/01.txt to s3://zork.movetest/dest/01.txt
move: s3://zork.movetest/source/02.txt to s3://zork.movetest/dest/02.txt
move: s3://zork.movetest/source/03.txt to s3://zork.movetest/dest/03.txt

when i run that same command with --debug,
for each file, there is a put and delete. here is a log snippet

PUT /zork.movetest/dest/0.file
DELETE /zork.movetest/source/0.file
there is no api operation for move

It has to do each file individually Copy then Delete - that is the way the API works.

However rclone move can move a directory so you can just run it and wait for it to finish.

Note that server side copies are relatively expensive operations on S3 backends.

I see. 1 serverside copy delete in each 2-3 minutes. I should have download the files, and re upload them, it would be much faster :smiley: (really)

I don't want to move directory, i want to move files from multiple directories to another single directory.

1 Like

true, my original post did a move from dir to dir.

however, the OP made mention of a post from stackoverflow, that used the aws clii and included --include
so i tried to mimic that.

just to note, that wasabi does not charge for api calls

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.