Lots of disk IO when using sync operation, no uploading


I am using the below command line to perform an rclone sync from local disk to Backblaze B2

C:\apps\rclone\rclone.exe --config=C:\apps\rclone\rclone.conf --exclude-from=C:\apps\rclone\exclude.txt --b2-chunk-size=150M --transfers=1 --ignore-checksum --stats=30s -vv sync X:\Backups b2:mybucket

When this is running, I see a ton of disk IO but zero traffic uploading to B2. From what I understand, that command should not be doing any checksum-based operations and therefore should read metadata to compare size and time and then upload the file. However, I can let the operation run for an hour on a 200GB file and it will still have a ton of disk IO with zero uploading. I have several 200GB files and a few files > 1TB that I need to transfer but rclone does not seem to be making any progress on these files.

Have I missed something?

Thank you.

Note that this ignores the checksum after the transfer which isn’t a good idea in general.

In order to upload a file to B2 rclone has to generate the checksum of the file first - hence the very long pause before anything happens.

It is possible to upload large files to B2 without a checksum - in the s3 backend (which has a similar problem) we implemented the

  --s3-disable-checksum                      Don't store MD5 checksum with object metadata

In general it is a good idea to keep the checksums - rclone will be able to check them when you download the file again.

If you think rclone should have a --b2-disable-checksum flag which would disable that checksum calculation for large files (files which have to be chunked - b2 absolutely requires the checksum for small files) then please make a new issue on github and we can discuss further.

Note also you can run with -vv and rclone will show you the parts being transferred.