Server side move of a large number of subdirectories to a new directory?

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

Can't move directory using rc command/movefile?

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2
- os/version: debian 11.8 (64 bit)
- os/kernel: 5.10.0-26-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive (w/crypt)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone rc -vv log-file=rclone.out operations/movefile srcFs=gcrypt: 'srcRemote=downloads/mybooks/Bearing An Hourglass - Recorded Books' dstFs=gcrypt: 'dstRemote=home/haynes/data/books/Bearing An Hourglass - Recorded Books'

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gcrypt]
type = crypt
remote = gdrive:crypt
filename_encryption = standard
password = XXX
password2 = XXX
directory_name_encryption = true

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
token = XXX
root_folder_id = XXX
team_drive =

A log from the command that you were trying to run with the -vv flag

2024/02/10 05:26:48 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "rc" "-vv" "--log-file=rclone.out" "operations/movefile" "srcFs=gcrypt:" "srcRemote=downloads/mybooks/Bearing An Hourglass - Recorded Books" "dstFs=gcrypt:" "dstRemote=home/haynes/data/books/Bearing An Hourglass - Recorded Books"]
2024/02/10 05:26:49 DEBUG : 6 go routines active
2024/02/10 05:26:49 Failed to rc: operation "operations/movefile" failed: is a directory not a file

Output of remote command

{
        "error": "is a directory not a file",
        "input": {
                "dstFs": "gcrypt:",
                "dstRemote": "home/haynes/data/books/Bearing An Hourglass - Recorded Books",
                "srcFs": "gcrypt:",
                "srcRemote": "downloads/mybooks/Bearing An Hourglass - Recorded Books"
        },
        "path": "operations/movefile",
        "status": 500
}

What I'm trying to do is reorganize my directory structure, and clean up. I'm trying to move the files and directories I want to keep from downloads/mybooks to home/haynes/data/books. There's tens of thousands of top level entries in downloads/mybooks, a mixture of files and directories. I want to move a few hundrets of them to home/haynes/data/books (as a bonus I'm trying to leave behind a shortcut to the new location but that's out of scope for this question.)

Now obviously from the name, "movefile" seems like it wouldn't move directories, but I haven't found a rc equivalent to the "moveto" command. I could use rclone moveto with a file list, but rclone moveto —files-from-raw=filelist seems to ignore directories and doing it in a loop seems inefficient and slow.

It seems like I must be missing something basic and obvious, but I can't figure out how to efficiently do a server side move of a bunch of subdirectories from one directory to another.

2

What about sync/move?

3

Thanks for the suggestion, I'll check it out.

1 Like
4

I've done a little more digging, and I remembered I was logging rcd's output and could examine why commands were failing (d'oh) at this point it looks to me like I have to use rclone rc operations/movefile to move files and rclone rc sync/move to move directories, but that between the two of them I can manage to do the server side moves I need. It still looks like I'll have to move them in a loop? Because I can't use file-list to specify the individual entries to move, but I'll see if I can't include them in the json parameters.

It was a little confusing because the syntax of rc sync/move requires srcFs=gcrypt:path dstFs=gcrypt:path and rc operations/move requires srcFs=gcrypt: srcRemote=path dstFs=gcrypt: dstRemote=path? but that's not hard to code around.

Still welcome any suggestions on how to simplify things, but I think I can do what I need to do.

5

operations/movefile is limited to a file, that's correct. sync/move, on the other hand, is the direct equivalent of rclone move, so, by default it will move everything in the srcFs, but there's no reason you couldn't filter this down to individual files if you wanted to.

You could use filter rules to include/exclude specific subdirectories (think --filter-from, not --files-from) (and shameless plug for my own genfilters tool to help generate them)

However, in your specific case, I think you'd be better off keeping your loop. The reason is because sync/move behaves differently when filters are in use. Without filters, you can take advantage of server-side moving at the directory level, if supported by your remote. Whereas if any filters are in use, it will server-side move each individual file in the directory. The end result should be the same, but it's not as efficient.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.