Optimize rclone on Raspberry pi 4 8GB

gwildor · July 6, 2022, 1:32pm

rclone v1.58.1

os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.4.0-1065-raspi (aarch64)
os/type: linux
os/arch: arm64
go/version: go1.17.9
go/linking: static
go/tags: none

[annica_encrypt]
type = crypt
remote = google_annica:
info_age = 72h
filename_encryption = standard
directory_name_encryption = true
password = xxxx
password2 = xxx

[google_annica]
type = drive
client_id = xxx.apps.googleusercontent.com
client_secret = xxx
scope = drive
token = {"access_token":"ya.....
root_folder_id = ....
team_drive =

On my server this settings has been optimal to use, but now i run raspbery pi 4 8GB with sata to ssd and ssd disk (Samsung evo 840).
How can i optmize the settings so i got maximum upload to google drive?
I use this code now as mount:
/usr/bin/rclone --config "/root/.config/rclone/rclone.conf" mount --allow-other --umask=000 --vfs-cache-mode writes --vfs-cache-max-size 90G --drive-chunk-size=64M --cache-dir=/root/cache/rclone --dir-cache-time=1h --tpslimit 7 --allow-non-empty annica_encrypt:backup /mnt/backup &

My connection is 500/500 and the backup of whole Raspberry is going to be run one a week at night when i have no visitiors on my webpage. So rclone can use up to 90 cache and 6GB ram without any problem.

The code that runs:
sudo dd if=/dev/sda conv=sync,noerror bs=4M of=/mnt/backup/raspberryn/backup_date -I-KL$(date +%H).img status=progress (progress is only for testing now, that removes when upload works perfect)

Any idea?
Thanks

asdffdsa · July 6, 2022, 1:42pm

hello and welcome to the forum,

i have a similar setup, but use ubuntu server and 1Gbps/1Gbps

the settings look ok. might take optimized settings from this well tested command.
https://github.com/animosity22/homescripts/blob/master/systemd/rclone-drive.service
note: for the fastest file transfer, use rclone copy/sync

that flag is for the deprecated cache remote

did you mean usb to ssd or what?

gwildor · July 6, 2022, 2:42pm

Yes, sata to usb

Can i use copy or sync when the file is 250GB and i only have 90GB storage left.
This code sudo dd if=/dev/sda conv=sync,noerror bs=4M of=/mnt/backup/raspberryn/backup_date -I-KL$(date +%H).img status=progress thats backup of the whole harddrive. I have no sdcard installed.

asdffdsa · July 6, 2022, 2:47pm

rclone sync/copy/move does not write to local storage, copies direct to cloud.

gwildor · July 6, 2022, 2:51pm

Perfect! Thanks!
Do u know how i can do to not use the mount options in this case?
/mnt/backup is mounted with rclone.

asdffdsa · July 6, 2022, 2:56pm

can try rclone_rcat
but the odds are rclone cannot verify the file transfer using hash checksums.
so a no go for backups.

or run dd to save to a local file and than rclone copy that file to cloud.
tho you might need more local storage.

and there might be other ways, as i am not a linux expert.

ncw · July 6, 2022, 4:08pm

Rclone will do its best to check checksums while using rclone rcat - it calculates the checksum of the data as it runs through rclone then checks the checksum on the remote when it is uploaded. You can't do a post copy rclone check but it's a pretty good check.

asdffdsa · July 6, 2022, 4:40pm

i think it is a bit more convoluted.

these two commands are the same, except for --checksum
--- in both cases, rclone check works.
--- only when using --checksum does rclone compare checksums after transfer

type d:\files\file.ext   | C:\data\rclone\rclone.exe rcat gdrive:zork/file.ext -vv 
DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\data\\rclone\\rclone.exe" "rcat" "gdrive:zork/file.ext" "-vv"]
DEBUG : Creating backend with remote "gdrive:zork/"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : Google drive root 'zork': 'root_folder_id = 0AIYnsu88uXytUk9PVA' - save this in the config to speed up startup
DEBUG : fs cache: renaming cache item "gdrive:zork/" to be canonical "gdrive:zork"
DEBUG : file.ext: Sending chunk 0 length 1448900
DEBUG : file.ext: Size and modification time the same (differ by -532.5Âµs, within tolerance 1ms)
DEBUG : 6 go routines active

rclone.exe lsf gdrive:zork --format="ph" 
file.ext;c7f5af9b93f5aa17934c84ad53fd2cea

rclone.exe check d:\files\file.ext gdrive:zork -vv 
DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\data\\rclone\\rclone.exe" "check" "d:\\files\\file.ext" "gdrive:zork" "-vv"]
DEBUG : Creating backend with remote "d:\\files\\file.ext"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : fs cache: adding new entry for parent of "d:\\files\\file.ext", "//?/d:/files"
DEBUG : Creating backend with remote "gdrive:zork"
DEBUG : Google drive root 'zork': 'root_folder_id = 0AIYnsu88uXytUk9PVA' - save this in the config to speed up startup
INFO  : Using md5 for hash comparisons
DEBUG : Google drive root 'zork': Waiting for checks to finish
DEBUG : file.ext: md5 = c7f5af9b93f5aa17934c84ad53fd2cea OK
DEBUG : file.ext: OK
NOTICE: Google drive root 'zork': 0 differences found
NOTICE: Google drive root 'zork': 1 matching files
INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.9s

rclone.exe delete gdrive:zork -v 
INFO  : file.ext: Deleted

type d:\files\file.ext   | C:\data\rclone\rclone.exe rcat gdrive:zork/file.ext -vv --checksum 
DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\data\\rclone\\rclone.exe" "rcat" "gdrive:zork/file.ext" "-vv" "--checksum"]
DEBUG : Creating backend with remote "gdrive:zork/"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : Google drive root 'zork': 'root_folder_id = 0AIYnsu88uXytUk9PVA' - save this in the config to speed up startup
DEBUG : fs cache: renaming cache item "gdrive:zork/" to be canonical "gdrive:zork"
DEBUG : file.ext: Sending chunk 0 length 1448900
DEBUG : file.ext: md5 = c7f5af9b93f5aa17934c84ad53fd2cea OK
DEBUG : file.ext: Size and md5 of src and dst objects identical
DEBUG : 6 go routines active

rclone.exe lsf gdrive:zork --format="ph" 
file.ext;c7f5af9b93f5aa17934c84ad53fd2cea

rclone.exe check d:\files\file.ext gdrive:zork -vv 
DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\data\\rclone\\rclone.exe" "check" "d:\\files\\file.ext" "gdrive:zork" "-vv"]
DEBUG : Creating backend with remote "d:\\files\\file.ext"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : fs cache: adding new entry for parent of "d:\\files\\file.ext", "//?/d:/files"
DEBUG : Creating backend with remote "gdrive:zork"
DEBUG : Google drive root 'zork': 'root_folder_id = 0AIYnsu88uXytUk9PVA' - save this in the config to speed up startup
INFO  : Using md5 for hash comparisons
DEBUG : Google drive root 'zork': Waiting for checks to finish
DEBUG : file.ext: md5 = c7f5af9b93f5aa17934c84ad53fd2cea OK
DEBUG : file.ext: OK
NOTICE: Google drive root 'zork': 0 differences found
NOTICE: Google drive root 'zork': 1 matching files
INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.9s

ncw · July 7, 2022, 10:40am

I love these replies where you are keeping me honest @asdffdsa - don't ever stop!

It looks like you found a bug - the checksum was calculated, but it was never checked! I made an issue here: rcat isn't checking checksums like it should · Issue #6305 · rclone/rclone · GitHub

The fix to rcat is very easy and I've had a go at that here

v1.59.0-beta.6314.4fc174705.fix-rcat-checksum on branch fix-rcat-checksum (uploaded in 15-30 mins)

The fix to rclone rcat --size is harder though and I'll probably delay that until after the v1.59 release as it is a much less used function - its used here and in rclone restic serve and restic has its own corruption detection.

asdffdsa · July 7, 2022, 12:32pm

will do.

looks fixed.

type d:\files\file.ext | C:\data\rclone\rclone-v1.59.0-beta.6314.4fc174705.fix-rcat-checksum-windows-amd64.exe rcat gdrive:zork/file.ext -vv 
DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
DEBUG : rclone: Version "v1.59.0-beta.6314.4fc174705.fix-rcat-checksum" starting with parameters ["C:\\data\\rclone\\rclone-v1.59.0-beta.6314.4fc174705.fix-rcat-checksum-windows-amd64.exe" "rcat" "gdrive:zork/file.ext" "-vv"]
DEBUG : Creating backend with remote "gdrive:zork/"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : Google drive root 'zork': 'root_folder_id = 0AIYnsu88uXytUk9PVA' - save this in the config to speed up startup
DEBUG : fs cache: renaming cache item "gdrive:zork/" to be canonical "gdrive:zork"
DEBUG : file.ext: Sending chunk 0 length 1448900
DEBUG : file.ext: md5 = c7f5af9b93f5aa17934c84ad53fd2cea OK
DEBUG : file.ext: Size and md5 of src and dst objects identical

ncw · July 7, 2022, 3:10pm

Thanks for testing. I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.59!

gwildor · July 14, 2022, 9:09pm

Is this something that is included in the original release now?

I have try alot of settings and got 25MB/s only, on my windows i got full speed.
Raspberrys image (ubuntu 20.04) has the 5.4.x kernel thats why im asking <3

system · August 13, 2022, 9:09pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.