[Gdrive] Copy with filter list all files in directory

Help and Support
1

What is the problem you are having with rclone?

Rclone seems to list all filtered file before starting a copy. Trying to copy 100k files from crypt drive to TD.

What is your rclone version (output from rclone version)

rclone v1.52.2
- os/arch: linux/amd64
- go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux instance-3 5.4.0-1021-gcp #21-Ubuntu SMP Fri Jul 10 06:53:47 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

rclone --config rclone.conf copy --rc --rc-addr="localhost:5572" -vv --size-only --include "S*.mkv" --include "T*.mkv" --include "U*.mkv" --include "V*.mkv" --include "W*.mkv" --tpslimit 10 --transfers 8 --drive-chunk-size 64M --buffer-size 128M --checkers 16 --user-agent Mozilla/5.0 --fast-list --drive-acknowledge-abuse --log-file=log_rclone.txt "gd-crypt3:" "dst072:/T" &

A log from the command with the -vv flag

2020/08/02 14:28:48 DEBUG : Bxxxxxxxx/Season 3/7xxxxxxxxxxxxx: Excluded from sync
(and deletion)
2020/08/02 14:28:48 DEBUG : Bxxxxxxxx/Season 3/7xxxxxxxxxxxxx: Excluded from sync
(and deletion)
2020/08/02 14:28:48 DEBUG : Bxxxxxxxx/Season 3/77xxxxxxxxxxxxx: Excluded from sync
(and deletion)
2020/08/02 14:28:48 DEBUG : Bxxxxxxxx/Season 3/77xxxxxxxxxxxxx: Excluded from sync
(and deletion)

Tried to add -no-traverse flag, but still goes thru all the files.
Maybe there is a more efficient way of doing it ?

My guess is that filter removes --fast-list speed boost, but then how do I split the transfert so I can have severals instance doing the copy (on several server) ?

I appreciate any tips. :slight_smile:
Regards

2

hello,

so the problem is too much information in the log file?

have you tried -v instead of -vv

3

No, the problem is that it takes several minutes before coping the first file.
If it's skipped in the log that's means Rclone consider it for transfert.

4

even with fast-list, rclone still needs to compare each file againt the multiple filters.

5

Sure, but here it should not goes thru all the files before doing something...

Same with file-from, should only consider those files instead of listing all the file that are excluded first.

6

Rclone is clever enough to skip directories it doesn't need to read, but it looks like you are asking for all files starting with certain letters so it will have to read all the directories to find them.

That will be --fast-list travsersing the file system first before doing any copies. If you remove --fast-list rclone will start transferring as soon as it lists a file that it should transfer.

Note you can write this as

--include "[STUVW]*.mkv"

or

--include "[S-W]*.mkv"
7

Hi @ncw and thank you for your reply,

My file structure is a folder, with several subfolders containing severalsfiles :

Following your response, I guess --include "[STUVW]*/**.mkv" should skip folders and files inside path like Axxxx/xxxx/xxxx.xxx then ?

I'll try that and let you know. Splitting the input should improve drastically the startup time then.

Also, will a mount with VFS help on this ? Or will it be not significant enough to bother with ? (Several run with the same parameters are needed)

8

That is correct, yes

Having a mount saves a few seconds of startup time only. Using the rclone commands direct on google drive is more efficient and I think in this case that is what you want to do.

9

Thank you @asdffdsa and @ncw. Filtering on the top level folder and removing the --fast-list did the trick.
Without parsing the 100k files each time, transfert starts faster and seems to have less api error.

1 Like
10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.