Bisync causes high CPU if `--ignore-checksum` is not passed, even with no diff

Help and Support
1

What is the problem you are having with rclone?

  • Problem: rclone bisync without --ignore-checksum takes a long time and high CPU.
  • Question 1: Does RClone calculate file checksum in order to find differences between local and remote even though --checksum was not passed? (Because I didn't pass the flag but CPU is high)
    • Question 1.1: If the answer is no - what's causing the high CPU on my machine?
    • Question 1.2: If the answer is yes - Can I make RClone look for differences by timestamps only (not by file sizes as I'm using OneDrive, and not by checksums as it's too much CPU), but STILL calculate checksum when downloading / uploading so I can assure the files are not corrupted?

Details about the Problem:

rclone bisync (using --ignore-size, because I'm syncing OneDrive), when there are no changes (no difference between local and remote, meaning no nothing was uploaded / downloaded, and operation is finished successfully):

  1. If runs with --ignore-checksum takes less then 1m and doesn't take much CPU:

2022-10-21_21-55
2022-10-21_21-553456×260 61.4 KB

  1. If runs without --ignore-checksum takes around 2m and runs with high CPU (and temperature):

2022-10-21_21-50
2022-10-21_21-503328×495 145 KB

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2
- os/version: arch (64 bit)
- os/kernel: 6.0.2-arch1-1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.1
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive.

I'm using --ignore-sizes since it has problem with sizes:

  1. OneDrive Size/Hash differs
  2. How to solve this error: "corrupted on transfer: sizes differ 2871808 vs 2883072" · Issue #4890 · rclone/rclone · GitHub

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone bisync /home/itay/Personal onedrive: --ignore-checksum --ignore-size --check-access --filters-file=/home/itay/.config/rclone/directories --log-file=/home/itay/.config/rclone/rclone.log -vv --log-format=date,time,microseconds,pid --transfers=10 --onedrive-chunk-size=40M --buffer-size=32M

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
client_id = <MY OWN ONE>
client_secret = <XXX>
token = <XXX>
drive_id = <XXX>
drive_type = personal

A log from the command with the -vv flag

Log when flag is used (low CPU, less then 1m):

2022/10/21 22:25:01.627816 [152534] DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "bisync" "/home/itay/Personal" "onedrive:" "--ignore-checksum" "--ignore-size" "--check-access" "--filters-file=/home/itay/.config/rclone/directories" "--log-file=/home/itay/.config/rclone/rclone.log" "-vv" "--log-format=date,time,microseconds,pid" "--transfers=10" "--onedrive-chunk-size=40M" "--buffer-size=32M"]
2022/10/21 22:25:01.627993 [152534] DEBUG : Creating backend with remote "/home/itay/Personal"
2022/10/21 22:25:01.628048 [152534] DEBUG : Using config file from "/home/itay/.config/rclone/rclone.conf"
2022/10/21 22:25:01.628107 [152534] DEBUG : Creating backend with remote "onedrive:"
2022/10/21 22:25:01.628141 [152534] DEBUG : onedrive: detected overridden config - adding "{7tn_C}" suffix to name
2022/10/21 22:25:03.123437 [152534] DEBUG : fs cache: renaming cache item "onedrive:" to be canonical "onedrive{7tn_C}:"
2022/10/21 22:25:03.123557 [152534] NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2022/10/21 22:25:03.124620 [152534] DEBUG : Lock file created: /home/itay/.cache/rclone/bisync/home_itay_Personal..onedrive{7tn_C}_.lck
2022/10/21 22:25:03.125006 [152534] INFO  : Synching Path1 "/home/itay/Personal/" with Path2 "onedrive{7tn_C}:/"
2022/10/21 22:25:03.125092 [152534] INFO  : Using filters file /home/itay/.config/rclone/directories
2022/10/21 22:25:03.127437 [152534] INFO  : Path1 checking for diffs
2022/10/21 22:25:03.137361 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:03.137369 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:03.137395 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:03.151590 [152534] INFO  : Path2 checking for diffs
2022/10/21 22:25:04.509587 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509637 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509647 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509653 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509660 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509667 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509674 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509680 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509692 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509700 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509708 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:04.509716 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879701 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879736 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879744 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879751 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879757 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879764 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879770 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879777 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879784 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879790 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879796 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879803 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879810 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879820 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:05.879847 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384208 [152534] DEBUG : <ONENOTE_OF_MINE>: OneNote file not shown in directory listing
2022/10/21 22:25:07.384259 [152534] DEBUG : <ONENOTE_OF_MINE>: OneNote file not shown in directory listing
2022/10/21 22:25:07.384314 [152534] DEBUG : <ONENOTE_OF_MINE>: OneNote file not shown in directory listing
2022/10/21 22:25:07.384344 [152534] DEBUG : <ONENOTE_OF_MINE>: OneNote file not shown in directory listing
2022/10/21 22:25:07.384458 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384472 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384485 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384495 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384504 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384521 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384532 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384542 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384553 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384563 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384574 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384584 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384595 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384604 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384613 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384622 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384632 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384641 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384651 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384660 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384668 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384678 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384687 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384696 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384705 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384714 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384722 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384733 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384742 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384752 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384761 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384771 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384780 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384789 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384798 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384808 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384817 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384826 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384835 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384847 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384856 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384865 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384874 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384884 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384893 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384929 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384942 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384952 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384961 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384973 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384982 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.384993 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385002 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385012 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385021 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385032 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385041 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385052 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385062 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385073 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385084 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385094 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385105 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385115 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385126 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385136 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385146 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385158 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385169 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385179 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385189 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385198 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385207 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385216 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385227 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385241 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385251 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385260 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385268 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385277 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385287 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385296 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385305 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385314 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385322 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385331 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385339 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385348 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385356 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385374 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385384 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385392 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385401 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385410 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385418 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385427 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385435 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385445 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385454 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385462 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385472 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385481 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385490 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385499 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385508 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385517 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385525 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385531 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385535 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385539 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385542 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385546 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385551 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385555 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385559 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385563 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385567 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385571 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385575 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385579 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385583 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385587 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385592 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385596 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385600 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385604 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385608 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385612 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385616 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385620 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385624 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385632 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385636 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385640 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385644 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385649 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385653 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385657 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385661 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385665 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385669 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385673 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385677 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385681 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385685 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385689 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:07.385692 [152534] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 22:25:52.930899 [152534] INFO  : Checking access health
2022/10/21 22:25:52.930909 [152534] INFO  : Found 2 matching "RCLONE_TEST" files on both paths
2022/10/21 22:25:52.930912 [152534] INFO  : No changes found
2022/10/21 22:25:52.930914 [152534] INFO  : Updating listings
2022/10/21 22:25:52.938055 [152534] INFO  : Validating listings for Path1 "/home/itay/Personal/" vs Path2 "onedrive{7tn_C}:/"
2022/10/21 22:25:52.947774 [152534] DEBUG : Lock file removed: /home/itay/.cache/rclone/bisync/home_itay_Personal..onedrive{7tn_C}_.lck
2022/10/21 22:25:52.947779 [152534] INFO  : Bisync successful
2022/10/21 22:25:52.947829 [152534] INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        51.3s

2022/10/21 22:25:52.947831 [152534] DEBUG : 5 go routines active

Log when flag is not used (high CPU, around 2m):

2022/10/21 21:50:07.963058 [136493] DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "bisync" "/home/itay/Personal" "onedrive:" "--ignore-size" "--check-access" "--filters-file=/home/itay/.config/rclone/directories" "--log-file=/home/itay/.config/rclone/rclone.log" "-vv" "--log-format=date,time,microseconds,pid" "-P" "--transfers=10" "--onedrive-chunk-size=40M" "--buffer-size=32M"]
2022/10/21 21:50:07.963273 [136493] DEBUG : Creating backend with remote "/home/itay/Personal"
2022/10/21 21:50:07.963363 [136493] DEBUG : Using config file from "/home/itay/.config/rclone/rclone.conf"
2022/10/21 21:50:07.963450 [136493] DEBUG : Creating backend with remote "onedrive:"
2022/10/21 21:50:07.963488 [136493] DEBUG : onedrive: detected overridden config - adding "{7tn_C}" suffix to name
2022/10/21 21:50:07.963746 [136493] DEBUG : onedrive{7tn_C}: Loaded invalid token from config file - ignoring
2022/10/21 21:50:08.496655 [136493] DEBUG : Saving config "token" in section "onedrive" of the config file
2022/10/21 21:50:08.506519 [136493] DEBUG : onedrive{7tn_C}: Saved new token in config file
2022/10/21 21:50:10.458924 [136493] DEBUG : fs cache: renaming cache item "onedrive:" to be canonical "onedrive{7tn_C}:"
2022/10/21 21:50:10.459136 [136493] NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2022/10/21 21:50:10.459514 [136493] DEBUG : Lock file created: /home/itay/.cache/rclone/bisync/home_itay_Personal..onedrive{7tn_C}_.lck
2022/10/21 21:50:10.459809 [136493] INFO  : Synching Path1 "/home/itay/Personal/" with Path2 "onedrive{7tn_C}:/"
2022/10/21 21:50:10.459933 [136493] INFO  : Using filters file /home/itay/.config/rclone/directories
2022/10/21 21:50:10.460723 [136493] INFO  : Path1 checking for diffs
2022/10/21 21:50:10.469380 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:50:10.469394 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:50:10.469479 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:13.864474 [136493] INFO  : Path2 checking for diffs
2022/10/21 21:51:15.004113 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004165 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004177 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004185 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004194 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004203 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004211 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004220 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004235 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004245 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004254 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:15.004263 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385846 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385897 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385909 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385918 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385926 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385936 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385945 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385955 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385964 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385973 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385982 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.385992 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.386002 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.386015 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:16.386024 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900457 [136493] DEBUG : <ONENOTE_OF_MINE>: OneNote file not shown in directory listing
2022/10/21 21:51:18.900572 [136493] DEBUG : <ONENOTE_OF_MINE>: OneNote file not shown in directory listing
2022/10/21 21:51:18.900743 [136493] DEBUG : <ONENOTE_OF_MINE>: OneNote file not shown in directory listing
2022/10/21 21:51:18.900785 [136493] DEBUG : <ONENOTE_OF_MINE>: OneNote file not shown in directory listing
2022/10/21 21:51:18.900890 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900898 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900906 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900911 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900918 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900930 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900938 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900961 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900970 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.900975 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901132 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901154 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901162 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901167 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901172 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901176 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901181 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901186 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901190 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901194 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901198 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901203 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901209 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901214 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901218 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901222 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901226 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901231 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901235 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901240 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901244 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901249 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901253 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901257 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901261 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901265 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901270 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901274 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901278 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901284 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901288 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901293 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901297 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901301 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901306 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901310 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901315 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901319 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901323 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901328 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901333 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901337 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901341 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901346 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901350 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901355 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901359 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901364 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901368 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901373 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901378 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901390 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901396 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901401 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901407 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901411 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901416 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901423 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901429 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901434 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901439 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901443 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901447 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901451 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901455 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901462 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901467 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901471 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901475 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901479 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901484 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901488 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901492 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901496 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901500 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901504 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901508 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901512 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901516 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901520 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901524 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901528 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901532 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901536 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901540 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901544 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901548 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901552 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901556 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901560 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901564 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901568 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901573 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901581 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901585 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901590 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901594 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901598 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901602 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901607 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901611 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901615 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901619 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901623 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901627 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901631 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901635 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901640 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901644 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901648 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901652 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901656 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901661 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901665 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901669 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901673 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901677 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901681 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901685 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901690 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901694 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901698 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901702 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901706 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901710 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901714 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901719 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901723 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901727 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901731 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901735 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901739 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901743 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901751 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901756 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901760 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:51:18.901764 [136493] DEBUG : <FILE_OF_MINE> Excluded
2022/10/21 21:52:05.997924 [136493] INFO  : Checking access health
2022/10/21 21:52:05.997934 [136493] INFO  : Found 2 matching "RCLONE_TEST" files on both paths
2022/10/21 21:52:05.997936 [136493] INFO  : No changes found
2022/10/21 21:52:05.997937 [136493] INFO  : Updating listings
2022/10/21 21:52:06.005356 [136493] INFO  : Validating listings for Path1 "/home/itay/Personal/" vs Path2 "onedrive{7tn_C}:/"
2022/10/21 21:52:06.023326 [136493] DEBUG : Lock file removed: /home/itay/.cache/rclone/bisync/home_itay_Personal..onedrive{7tn_C}_.lck
2022/10/21 21:52:06.023331 [136493] INFO  : Bisync successful
2022/10/21 21:52:06.023357 [136493] INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      1m58.0s

2022/10/21 21:52:06.023359 [136493] DEBUG : 5 go routines active
2

Apparently it does calculate hashes when doing the listings unless you pass --ignore-checksum.

I'm not sure why this is though - @ivandeex ?

--ignore-checksum is the way to avoid checksums in the listing according to the code, but that will then ignore hashes as you said.

You could try working around this by wrapping the local backend in a hasher backend

3

Thanks @ncw for your quick response!

Too bad that --ignore-checksum combines ignoring checksum both for (1) finding changes and (2) verifying that copied files were copied successfully...

I'll try to use the hasher backend and see its performance but I also came up with another idea:

  • Keep running (as a cronjob) rclone bisync every 15 minutes with --ignore-checksum.
  • Running (as an extra crobjob) rclone check once a day to verify all checksums.

What do you think? Does it sound like a reasonable plan?

Thanks again!!

4

Sounds like a good idea!

1 Like
5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.