Check command not observing 'compare-dest' flag to add additional destination directories for comparison

rclone v1.62.2
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 4.15.0-211-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Using src: dropbox, dst: local

rclone --min-age 30d --compare-dest "/storage/media/pictures/Simon's Pictures/cloud_sync/2023/2023-05/" check "dropbox:/Camera Uploads" "/storage/media/pictures/Simon's Pictures/cloud_sync/Undated/" --one-way --max-depth 1 --match "/var/log/rclone/matches.txt"

Expected output: /var/log/rclone/matches.txt contains names of files that were present EITHER in the local storage "Undated" directory OR "2023/2023-05/" directory.

Actual output: /var/log/rclone/matches.txt only contains names of file that were present in the local storage "Undated" directory and gives log errors like:
2023/06/01 10:13:38 ERROR : 2023-05-01 15.51.07.mp4: file not in Local file system at /storage/media/pictures/Simon's Pictures/cloud_sync/Undated

I have files on a remote (dropbox) that I want to be the source of a check, and my local backup server that is the destination. However, while dropbox organizes my photos into YYYY/YYYY-MM/ directories, it does not do the same for videos. So I have a cron job that downloads all "top-level" files in the camera uploads directory, compares the filenames and then ships them off to the right backup folder. What I want to do is come along say 4-5 weeks later after I am confident all downloads and cron jobs have had multiple chances to run, and delete any files in dropbox that have already been downloaded. However, the issue is the top-level files in the camera uploads folder now don't match the folder structure of my local storage, so the "check" command cannot compare like-for-like. No problem I thought, i'll just use few "$(date +%Y-%m)" and "$(date -d "-1 month" +%Y-%m) entries in my cron bash script to search the last few months folders to see if they have already been downloaded. However while this works for "copy" it doesn't work for "check". I still get errors like

So for comparison, this command runs successfully and only downloads files from dropbox that DO NOT already exist in the temporary staging folder: /storage/downloads/photo_sync, "Undated". "2023/2023-05". "2023/2023-04" ... etc.

/usr/bin/rclone copy dropbox:/Camera Uploads /storage/downloads/photo_sync --log-level INFO --log-file /var/log/rclone/rclone-dropbox-top-level-files-dl.log --max-duration 4h --max-depth 1 --min-age 8d --immutable --compare-dest "/storage/media/pictures/Simon's Pictures/cloud_sync/2023/2023-05/" --compare-dest "/storage/media/pictures/Simon's Pictures/cloud_sync/2023/2023-04/" --compare-dest "/storage/media/pictures/Simon's Pictures/cloud_sync/2023/2023-03/" --compare-dest "/storage/media/pictures/Simon's Pictures/cloud_sync/2023/2023-02/" --compare-dest "/storage/media/pictures/Simon's Pictures/cloud_sync/Undated/"

config:

[dropbox]
type = dropbox
app_key =
app_secret =
token = {"access_token":"XXXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}

Log output:

simon@sserver:~/Downloads/photo_sync$ rclone --min-age 30d --compare-dest "/storage/media/pictures/Simon's Pictures/cloud_sync/2023/2023-05/" check "dropbox:/Camera Uploads" "/storage/media/pictures/Simon's Pictures/cloud_sync/Undated/" --one-way --max-depth 1 --match "/var/log/rclone/match.txt" -vv
2023/06/01 10:24:53 DEBUG : --min-age 1M to 2023-05-02 10:24:53.040560103 +0100 BST m=-2591999.933909847
2023/06/01 10:24:53 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "--min-age" "30d" "--compare-dest" "/storage/media/pictures/Simon's Pictures/cloud_sync/2023/2023-05/" "check" "dropbox:/Camera Uploads" "/storage/media/pictures/Simon's Pictures/cloud_sync/Undated/" "--one-way" "--max-depth" "1" "--match" "/var/log/rclone/match.txt" "-vv"]
2023/06/01 10:24:53 DEBUG : Creating backend with remote "dropbox:/Camera Uploads"
2023/06/01 10:24:53 DEBUG : Using config file from "/storage/homes/simon/.config/rclone/rclone.conf"
2023/06/01 10:24:53 DEBUG : Dropbox root '': Using root namespace "157717896"
2023/06/01 10:24:53 DEBUG : fs cache: renaming cache item "dropbox:/Camera Uploads" to be canonical "dropbox:Camera Uploads"
2023/06/01 10:24:53 DEBUG : Creating backend with remote "/storage/media/pictures/Simon's Pictures/cloud_sync/Undated/"
2023/06/01 10:24:53 DEBUG : fs cache: renaming cache item "/storage/media/pictures/Simon's Pictures/cloud_sync/Undated/" to be canonical "/storage/media/pictures/Simon's Pictures/cloud_sync/Undated"
2023/06/01 10:24:53 INFO  : Using dropbox for hash comparisons
2023/06/01 10:24:53 DEBUG : Local file system at /storage/media/pictures/Simon's Pictures/cloud_sync/Undated: Waiting for checks to finish
2023/06/01 10:24:54 DEBUG : 2023-05-15 16.35.35.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-15 16.30.07.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-15 16.33.39.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-06 16.57.30.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-14 13.00.05.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-24 10.17.45.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-24 10.16.45.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-24 10.11.42.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-24 10.09.02.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-24 10.05.45.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-24 10.04.02.mp4: Excluded
2023/06/01 10:24:54 DEBUG : 2023-05-26 19.42.40.mp4: Excluded
2023/06/01 10:24:54 ERROR : 2023-05-01 15.51.07.mp4: file not in Local file system at /storage/media/pictures/Simon's Pictures/cloud_sync/Undated
2023/06/01 10:24:54 ERROR : 2023-05-01 16.46.23.mp4: file not in Local file system at /storage/media/pictures/Simon's Pictures/cloud_sync/Undated
2023/06/01 10:24:54 ERROR : 2023-05-01 19.59.59.mp4: file not in Local file system at /storage/media/pictures/Simon's Pictures/cloud_sync/Undated
2023/06/01 10:24:54 NOTICE: Local file system at /storage/media/pictures/Simon's Pictures/cloud_sync/Undated: 3 files missing
2023/06/01 10:24:54 NOTICE: Local file system at /storage/media/pictures/Simon's Pictures/cloud_sync/Undated: 3 differences found
2023/06/01 10:24:54 NOTICE: Local file system at /storage/media/pictures/Simon's Pictures/cloud_sync/Undated: 3 errors while checking
2023/06/01 10:24:54 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 3 (retrying may help)
Elapsed time:         1.1s

2023/06/01 10:24:54 DEBUG : 7 go routines active
2023/06/01 10:24:54 INFO  : Dropbox root 'Camera Uploads': Committing uploads - please wait...
2023/06/01 10:24:54 Failed to check with 3 errors: last error was: 3 differences found

You are right, rclone check takes no notice of --compare-dest at the moment.

I guess it could though!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.