I had a search around with both the forum search and Google but couldn’t find any similar topics.
I’m running rclone 1.45-DEV on Linux uploading from several local directories to a crypted Google Drive remote. I’m using “rclone sync” - pretty much all my data has been uploaded to Google Drive before now and so the sync operations seem to consist mostly of a few small uploads/changes and a lot of checks. I’ve noticed that the number of checks is much, much higher than the number of files in the local directories.
Here’s the full command I’m running:
rclone sync --transfers 50 --stats-log-level NOTICE --log-file=/home/gus/rclone-cron.log --delete-during -L /home/gus/ gdrive-crypt:gus
Here’s the latest stats output from rclone:
2019/01/16 10:32:21 NOTICE:
Transferred: 5.546G / 5.546 GBytes, 100%, 92.925 kBytes/s, ETA 0s
Errors: 71 (retrying may help)
Checks: 878402 / 878402, 100%
Transferred: 622 / 622, 100%
Elapsed time: 17h23m2.9s
Just for reference, this is a local rsync’s interpretation of the file count for this directory:
Number of files: 156,721 (reg: 149,060, dir: 7,513, link: 144, special: 4)
Total file size: 13,247,976,672 bytes
Based on previous experience, I think that for this particular directory the sync does eventually finish, but I have several other directories to sync with higher file counts than this and the number of checks is similarly much, much higher than the number of files. On one directory (with around 500,000 files) I left rclone running for over a week and it just never finished. When I finally killed the rclone process, the number of checks was up past 1.5 million and hadn’t changed in many hours.
I started adding --delete-during to my commands because I noticed that with the standard behaviour of “rclone sync”, files weren’t deleted from the remote unless the transfer was completely free of errors. This is something that seems very difficult to achieve with large file counts as invariably at least one file will change during the time rclone runs and so the run always finishes with errors. I presumed originally that the high check count was because there were many old files present on the remote that were being checked. After adding --delete-during, a huge number of old files were deleted during sync operations and I thought this would bring the count of both remote files and checks down, but this is seemingly not the case.
Any idea why the number of checks is so high and what I could do to reduce it? Similarly, are there any recommended settings for the number of checkers and transfers I should use for relatively high file counts with Google Drive?
I’d ideally like to run one full sync with rclone every week but at the moment, I can’t have even one full sync finish within that timeframe due to the massive check counts.