#### What is the problem you are having with rclone?
When validating checksums for files that have been copied across to a remote (Microsoft OneDrive), the command is running well over 3 hours for a small number of files and directories.
The file sizes range between 200MB and 400MB. There are approximately 2,566 Files and 96 directories. The process also appears to result in increased CPU & memory utilization.
As it renders the device unusable, the process is stopped prior to completing the validation.
#### What is your rclone version (output from rclone version)
#### Which OS you are using and how many bits (eg Windows 7, 64 bit)
Microsoft Windows 10 Professional version 1909 Build 18636.1110
#### Which cloud storage system are you using? (eg Google Drive)
#### The command you were trying to run (eg rclone copy /tmp remote:tmp) rclone.exe check --stats 10s --progress --log-file=rclone_log.txt --log-level DEBUG /path/to/files/on/local/hard/drive remote:/path
#### The rclone config contents with secrets removed.
The configuration is standard i.e. it follows the Microsoft OneDrive rclone guide - https://rclone.org/onedrive/. There is no encryption configured.
#### A log from the command with the -vv flag
There is no additional information in the log files apart from a successful matching hash e.g.
2020/10/13 14:22:20 DEBUG : [REDACTED]: SHA-1 = 251939df1fb2ef1ae0372abbe480e8be51308780 OK
or a missing file such as;
2020/10/13 17:10:29 ERROR : [REDACTED]: File not in One drive root '[REDACTED]
@Animosity022, the file size ranges between 200MB - 400MB and there are approximately 2,000 files which would equate to approximately 35 minutes. In this instance, after 3 hours, the validation hadn't been completed and since the device was unusable, the process had to be stopped.
Checksums are primarily CPU as it is calculating a value for the file.
use-mmapis how rclone handles cleaning up memory and does well on low memory systems.
buffer-size is what is kept in memory and read ahead when a file is request sequentially before it's closed.
@ncw, the suggestion to reduce the checkers to 1 helped reduce the CPU and memory utilization. It still took a considerable time to validate the checksum though. It took approximately 3 hours to validate 2500 files with 54 errors (which I am unsure how best to address. Does running the command again only limit it to the errors or does it attempt to process all of them)
I'll continue tweaking the option to find a suitable value. It
Thanks @ncw. I'll give increasing --checkers a go.
Having had the look at the logs, it seems that if a file is missing it is considered an ERROR e.g. 2020/10/14 17:09:33 ERROR : Overview.mp4: File not in One drive root 'Media'
The files are media e.g. MP4.
2020/10/14 17:22:56 NOTICE: One drive root 'Media: 53 files missing
2020/10/14 17:22:56 NOTICE: One drive root 'Media': 53 differences found
2020/10/14 17:22:56 NOTICE: One drive root 'Media': 53 errors while checking
Although it isn't clear why it's reporting 54 errors i.e. 2020/10/14 17:22:56 Failed to check with 54 errors: last error was: 53 differences found and Errors: 54 (retrying may help). How do I find out the 54th error as filtering on the log only returns 53?
Is it possible to to generate a list of files so that if commands such as copy, sync, checksum, etc are run, it skips files that have been successfully processed?