How to move multiple files to a remote?

I'm currently migrating a bunch of files to GSuite using rclone, and for the ones already in a directory, this is easy. But how can I move multiple files if these files aren't in a specific parent directory, without having to do them all individually?

You can use a --files-from flag if you have the paths of the files you want to transfer in a file. You then run the transfer from a parent directory and rclone will transfer them into the same directory structure on the other end.

I didn't even know that was a thing. That sounds super useful. Thanks! I'll give that a try!

1 Like

Allow me to piggy back on this as I have an issue relating to this.

I have an account with files and folders in the root (100's).

/test.jpg
/dir/randome.txt
...

I try to run the command:
rclone move remote: remote:testmove

As I want everything under the users root directory to be moved to the folder called testmove (which doesn't exist so rclone will create it)

Here's the error I get:

C:\rclone>rclone move remote: remote:testmove
2020/01/30 09:31:01 ERROR : Google drive root 'testmove': Server side directory move failed: can't move root directory
2020/01/30 09:31:01 ERROR : Attempt 1/3 failed with 2 errors and: can't move root directory
2020/01/30 09:31:01 ERROR : Google drive root 'testmove': Server side directory move failed: can't move root directory
2020/01/30 09:31:01 ERROR : Attempt 2/3 failed with 2 errors and: can't move root directory
2020/01/30 09:31:01 ERROR : Google drive root 'testmove': Server side directory move failed: can't move root directory
2020/01/30 09:31:01 ERROR : Attempt 3/3 failed with 2 errors and: can't move root directory
2020/01/30 09:31:01 Failed to move with 2 errors: last error was: can't move root directory

What i'm understanding from this is the following:

it created the folder testmove and it's trying to move it into it's own, which is giving an error (understandibly)

I looked into the --exclude command but i couldn't find howto exclude just a directory, it seems to only exclude a dir if there's a file inside (i.e.: /testmove/file.jpg).

What am I missing to make this work?

Do I have to run an lsf command outputting to a txt file and then use fromfile (I would like to avoid this path as the folders will be updated and I need to perform a couple of deltas and it would mean that every time I have recreate the fromfile ?
Is there a way to exclude just a directory from the move command (if so, how)?

Any help would be appreciated.

Thank you,

use the --exclude flag
https://rclone.org/filtering/#directories
" If you put any rules which end in / then it will only match directories."

you can use --dry-run to test the filter.

I tried this command:

rclone move --exclude testmove/ remote: remote:testmove/

But i get this error.

C:\rclone>rclone move --exclude testmove/ remote: remote:testmove/
2020/01/30 12:20:28 ERROR : Fatal error received - not attempting retries
2020/01/30 12:20:28 Failed to move: can't sync or move files on overlapping remotes

that error would apply to any copy or backup program, not specific to rclone.

did you try --files-from?

I haven't tried --files-from as that would imply creating a list of all the files and since there's multiple users and constantly updated files, it's a lot less convenient to do it this way,

If there's no other way, it would be a last resort option.

you cannot store files in the root of a remote and expect to be able to easily move those files to a sub-folder of the same remote. that is not the fault of rclone.

if you store files in different sub folders, then you would be able to move them as needed using rclone
using
rclone move remote:\subfolder1\ remote:\subfolder2\

You can work around this by disabling directory moves.

--disable DirMove

Rclone should maybe do that itself rather than returning that error.

I was expecting it to work, I don't see why it wouldn't honestly, could be a limitation of the API's since GDRIVE doesn't really understand directory structures.

Tried:

C:\rclone>rclone sync remote: remote:testmove -v --disable DirMove --dry-run
2020/01/30 14:02:31 ERROR : Fatal error received - not attempting retries
2020/01/30 14:02:31 Failed to sync: can't sync or move files on overlapping remotes

I'm thinking the underlying issue is the folder testmove that is in the same remote and when using the copy or move command it's including testmove which is why it's giving that error. I want it to move everything else and exclude that folder and it's contents.

Since I'm trying to do the transfers server side.

I tested the following and it worked:
I shared a folder with user1
using the GD Web GUI I added the folder to "My Drive" and I was able to sync folders server side, that worked great.

Now I'm trying to do the same thing with user2, since I don't have access to that users GD Drive (WEB), I don't want to have to change the credentials.

I thought what if I shared a folder with that user and used --drive-shared-with-me to mount that shared folder since I don't have a way to "Add to My Drive", but since it's on the same remote I can't define for only dst path (since i'ts the same remote).

C:\rclone>set RCLONE_DRIVE_SHARED_WITH_ME=TRUE

C:\rclone>rclone lsf remote:
My Docs/

C:\rclone>rclone sync --drive-shared-with-me remote: remote:"My Docs" -v --dry-run
2020/01/30 13:43:39 ERROR : Fatal error received - not attempting retries
2020/01/30 13:43:39 Failed to sync: can't sync or move files on overlapping remotes

C:\rclone>set RCLONE_DRIVE_SHARED_WITH_ME=FALSE

C:\rclone>rclone sync remote: --drive-shared-with-me="My Docs" -v --dry-run
Error: invalid argument "My Docs" for "--drive-shared-with-me" flag: parsing "My Docs" as bool failed: expected newline

So i'm open to any ideas you might have.

you changed your command?

rclone move remote: remote:testmove
to
rclone sync remote: remote:testmove -v --disable DirMove --dry-run

Same error on both, either one will work for my needs, so whichever one I get working, is fine.

While on this topic, I had a question

I created two configs for the same remote,

One configured as drive
and the second as teamdrive

Both impersonating the same user (admin) with the same clientID and JSON

how would i go about syncing between the two location using the regular drive on the remote1 and teamdrive on remote?

I ask because i thought by simply configuring remote2 as team drive then it would only show theam drive folders but when i do an lsf on remote1 i get what's on my drive but when i do lsf on remote2 i get also what's on my drive.

[remote1]
type = drive
client_id = vvv
scope = drive
service_account_file = C:\rclone\json\sss.json
root_folder_id = xxx

[remote2]
type = drive
client_id = vvv
scope = drive
service_account_file = C:\rclone\json\sss.json
root_folder_id = xxx
team_drive = yyy

the config lists the teamdrive ID but when i do an LSF it lists the same files/folders as remote1, so i don't know how to copy to teamdrive

You might need to delete this from the remote config.

That worked, it only left the teamdrive line and I was able to lsf and write to it.

Is there a way to force a sync/move/copy to create the empty folders from source on the destination path?

Is there a way to force a sync/move/copy on files that are named the same (if there's multiple files named the same in a folder it will only copy one even if their contents are different)?

Thank you,

For rclone move you want --create-empty-src-dirs

It should work already for copy/move

You really need to run rclone dedupe on the original files if you can - rclone doesn't deal well with things which have the same name in the same directory.

Thank you very much for your help

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.