Rclone attempts to read files in the destination directory when the source doesn't exist

[dandv (Dan Dascalescu)]

What is the problem you are having with rclone?

When passing a mistaken path to a source, rclone seems to attempt to read files at the destination, and also outputs an error message about the source being not found that does not mention the exact path of the source. This makes debugging harder.

What is your rclone version (output from rclone version)

1.55.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20, 64-bit

Which cloud storage system are you using? (eg Google Drive)

Local

The command you were trying to run

rclone copy ~/nonexistent.file /tmp/ --progress --verbose

A log from the command with the -vv flag

$ rclone copy ~/nonexistent.file /tmp/ --progress -vv
Enter configuration password:
password:
2021/04/09 20:02:00 DEBUG : Using config file from "/home/dandv/.config/rclone/rclone.conf"
2021/04/09 20:02:00 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "copy" "/home/dandv/nonexistent.file" "/tmp/" "--progress" "-vv"]
2021/04/09 20:02:00 DEBUG : Creating backend with remote "/home/dandv/nonexistent.file"
2021/04/09 20:02:00 DEBUG : Creating backend with remote "/tmp/"
2021-04-09 20:02:00 NOTICE: OSL_PIPE_1001_SingleOfficeIPC_7b794f157196c61e807114f15779e7: Can't transfer non file/directory
2021-04-09 20:02:00 NOTICE: sddm-auth2b773263-4e30-4d8a-8b8a-6ea2a7acad86: Can't transfer non file/directory
2021-04-09 20:02:00 NOTICE: OSL_PIPE_1000_SingleOfficeIPC_41cbe25fb892f49ac6d770f2d4e7da1: Can't transfer non file/directory
2021-04-09 20:02:00 NOTICE: Ipne7fMKBNNvaJqT0SbGZ8OJiqHnXYkkLYvz2Mlx4Xw=: Can't transfer non file/directory
2021-04-09 20:02:00 NOTICE: qtsingleapp-DaVinc-7fe3-3e8: Can't transfer non file/directory
2021-04-09 20:02:00 NOTICE: sddm-:0-vBKXsN: Can't transfer non file/directory
2021-04-09 20:02:00 ERROR : : error reading source directory: directory not found
2021-04-09 20:02:00 DEBUG : Local file system at /tmp/: Waiting for checks to finish
2021-04-09 20:02:00 DEBUG : Local file system at /tmp/: Waiting for transfers to finish
2021-04-09 20:02:00 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
[...]
2021-04-09 20:02:00 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.9s
2021/04/09 20:02:00 INFO  : 
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.9s

2021/04/09 20:02:00 DEBUG : 2 go routines active
2021/04/09 20:02:00 Failed to copy: directory not found

Suggestions

1. Would it be possible to fail faster in this case, and not attempt to read (and transfer?) the files at the destination?
2. Would it be possible to output a more specific error message for "Failed to copy: directory not found", including the name of the offending directory?

[ncw (Nick Craig-Wood)]

@dandv

Would it be possible to fail faster in this case, and not attempt to read (and transfer?) the files at the destination?

When rclone make a remote for a path, it needs to be prepared for the path not to exist. So it isn't actually an error at the point of creation for the directory not to exist.

The only way to find out whether the directory exists is to list it, which rclone leaves to the sync routine to save API transactions.

However the sync routine starts listing both the source and the destination at the same time, which is what you see in your log.

Rclone will only read the top directory of the destination in this case.

So it would be possible to change this and check the source directory exists first, but it would cost another API transaction.

@dandv

Would it be possible to output a more specific error message for "Failed to copy: directory not found", including the name of the offending directory?

What rclone logged was this

2021-04-09 20:02:00 ERROR : : error reading source directory: directory not found

Which actually means that the source directory "" wasn't found which is the root directory of the transfer.

However that error is easy to improve.

How about

2021/04/10 09:31:04 ERROR : Local file system at /nonexistent: error reading source root directory: directory not found

Give this a try

v1.56.0-beta.5384.63849bd2f.fix-march-warning on branch fix-march-warning (uploaded in 15-30 mins)