Complete checking of destination before copy starts?

Suspected Bug
1

Hello,

I have a question / bug / change request for Rclone.

I use the copy command on Google Drive like this for example:
rclone copy “C:\Upload” “GoogleDrive:Folder/Subfolder” --stats 10s

“C:\Upload” only contains a few folders which I want to upload to “GoogleDrive:Folder/Subfolder”

“GoogleDrive:Folder/Subfolder” already contains up to a few thousand folders from earlier uploads

My expected behaviour of rclone was, that it only checks if the folders/files from the source exist at the destination and then start transfering them.

The actual behaviour is, that “GoogleDrive:Folder/Subfolder” and all subfolders are scanned before the transfer starts, even if they dont exist in “C:\Upload”. This causes a delay of over 10 minutes before the first transfer starts.

I am wondering about this delay for a longer time now, but recently I noticed, that the unnecessary destination-checks are the source, because Rclone informed me about a duplicate file in a sub-sub-subfolder which only exists in the destination and should never have been scanned for this task.

I hope, you understand what Im trying to tell you. Here is a screenshot of my problem:

Regards,
GulliGirl

2

Use this flag when only coping a small number of files to a system with lots of files…

  --no-traverse                       Don't traverse destination file system on copy.
1 Like
3

Thanks! Now it runs just beautiful. Seems like my own stupidity was the problem.

I got one more question. Is there any reason, why this is not the default behaviour?

I can’t think of any usecase, when it would be useful to scan the complete destination.

Regards,
GulliGirl

4

I was wondering the same here. What are disadvantages of using --no-traverse with large nr of files

5

rclone can generally list a whole remote using a small number of API calls.

When you are using --no-traverse rclone will do a list API call for each file you upload.

When increasing the number of files you are copying, at some point --no-traverse will become less efficient.

rclone doesn’t do this by default because I’ve been trying to think of a heuristic to as to when to use one or the other. At the moment rclone will use --no-traverse if you copy a single file or use --files-from - otherwise it won’t unless you explicitly supply the flag. This makes it work exactly like sync

1 Like