What is the problem you are having with rclone?
I recently started my journey with rclone and I'm still looking for an resource-saving but still stable configuration for the following scenario:
- rclone is running on a Synology DiskStation (the model is some years old and has only 1GB RAM)
- I want to backup data using Hyper Backup
- the backup destination is OneDrive, which is not directly supported by Hyper Backup
- so I'm using the webdav capabilities of rclone to make it work
Initially I started with VFS File Caching (--vfs-cache-mode writes). But Rclone consumed a lot of memory which lead to heavy swapping on the DiskStation and thus an I/O load that slowed down the NAS significantly. I could not find any set of options with enabled VFS File Caching that limited the memory consumption or number of rclone threads to an acceptable level.
I tried:
- reducing --transfers
- reducing --checkers
- reducing --vfs-cache-max-size
- reducing --onedrive-chunk-size
- reducing --tpslimit
- setting --buffer-size 0
- --use-mmap
In the end I found that rclone's resource consumtion was very acceptable with VFS File Caching disabled. So currently I'm running rclone with the options shown in the corresponding section below and it works fine most of the time.
System load is now absolutely fine but it is not yet as stable as required.
These are the stability issues I face:
SHA-1 hash differs
2021/05/03 11:54:43 ERROR : Backup/Data.hbk/Pool/0/0/63.bucket.2: corrupted on transfer: SHA-1 hash differ "8adcfa29ec45e30a04cf6ce818cde821a39acf86" vs "79d8ead431eb92aed2b40bd6d8bea9800de4808b"
2021/05/03 11:54:44 ERROR : Backup/Data.hbk/Pool/0/0/63.bucket.2: WriteFileHandle.New Rcat failed: corrupted on transfer: SHA-1 hash differ "8adcfa29ec45e30a04cf6ce818cde821a39acf86" vs "79d8ead431eb92aed2b40bd6d8bea9800de4808b"
Here I'm aware of this information:
Unexpected file size/hash differences on Sharepoint
But Hyper Backup chunks and compressed the data anyway. So there are no "office files" transferred to OneDrive that could be automatically opened and altered by OneDrive. (See the affected file from the log: 1521.index.2)
So I'm hesitating to use the options --ignore-checksum and --ignore-size as I believe these checks protect me from corrupt data. If not, meaning disabling the checks would not affect the data integrity in a bad way, please let me know.
InvalidAuthenticationToken: CompactToken validation failed
2021/05/04 02:20:31 ERROR : Backup/Data.hbk/Pool/0/0/1521.index.2: Failed to copy: InvalidAuthenticationToken: CompactToken validation failed with reason code: 80049228.
2021/05/04 02:20:31 ERROR : Backup/Data.hbk/Pool/0/0/1521.index.2: WriteFileHandle.New Rcat failed: InvalidAuthenticationToken: CompactToken validation failed with reason code: 80049228.
Looks like the token is refreshed every hour. So why does it not bother the running backup most of the time while sporadically come up with InvalidAuthenticationToken: CompactToken validation failed with reason code: 80049228?
Both types of errors make the backups fail. Especially for the initial backups which take several days this is very disturbing.
So I would appreciate any help to either
- run rclone with VFS File Caching enabled but limiting the memory consumption
- keep running rclone with VFS File Caching disabled but get the "stability issues" fixed.
What is your rclone version (output from rclone version
)
rclone v1.56.0-beta.5475.2833941da
- os/version: unknown
- os/kernel: 3.2.40 (i686)
- os/type: linux
- os/arch: 386
- go/version: go1.16.3
- go/linking: static
- go/tags: none
(I did not pay attention to the errors described above with rclone v1.55.0 (the first version I used) since "Unable to initialize RPS" was the main issue at this time.)
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Synology DSM 6.2 (linux), 32 bit
Which cloud storage system are you using? (eg Google Drive)
OneDrive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone serve webdav onedrive_data: --addr 127.0.0.1:12345 --use-mmap
The rclone config contents with secrets removed.
[onedrive_data]
type = onedrive
region = global
token = {"access_token":"[secret]","expiry":"2021-05-04T08:03:16.190094121+02:00"}
drive_id = [secret]
drive_type = personal
client_id = [secret]
client_secret = [secret]
A log from the command with the -vv
flag
Currently I don't have detailed logs available, since the errors mostly happen after many hours and --vv option is not used to keep system load low. I will enable --vv after the current backup jobs have been finished or interrupted.
Paste log here