What is the problem you are having with rclone?
My situation: mainly multiple 2GB files, to be copied to B2 via crypt.
Multi-core x64 machine, 2GHz clock.
2 things:
- I see that crypt is single threaded: on a multi-core machine, it peaks out 1 core, and does not use other cores. And it seems to require to have encrypted the whole file before starting to transfer data. Result: long delays (up to 10 minutes) before start of transfer, and regular pauses in transfers when it picks up new files to transfer.
- I have the impression that with the newer B2 backend these delays have gotten worse. (since I moved from 6.43.x to 6.44.2 with --b2-upload-concurrency 4)
Run the command 'rclone version' and share the full output of the command.
rclone v1.64.2
- os/version: debian 11.8 (64 bit)
- os/kernel: 6.1.0-0.deb11.11-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
B2
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone --password-command="pass B2/backup-bc" --fast-list --transfers 10 --b2-hard-delete --b2-upload-concurrency 4 --log-level=INFO --stats-file-name-length 0 -x sync /srv/dev-disk-by-label-fs1/backup/ crypt:
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[b2]
type = b2
account = XXX
key = XXX
hard_delete = true
[crypt]
type = crypt
remote = b2:XXX
password = XXX
password2 = XXX
A log from the command that you were trying to run with the -vv
flag
Sorry, have no log with -vv. Can provide one when needed, but I hope my issue is clear from a software architectural standpoint.
My question
I have the impression all of crypt is in 1 thread, even when encrypting multiple files. Would it be possible to let crypt use multiple threads? 1 thread per file to be encrypted?
Will this behaviour be impacted when the memory optimised version for B2 (with 6.45?) will arrive?