Crypt and multithreading (crypt+B2 got slower?)

What is the problem you are having with rclone?

My situation: mainly multiple 2GB files, to be copied to B2 via crypt.
Multi-core x64 machine, 2GHz clock.

2 things:

  1. I see that crypt is single threaded: on a multi-core machine, it peaks out 1 core, and does not use other cores. And it seems to require to have encrypted the whole file before starting to transfer data. Result: long delays (up to 10 minutes) before start of transfer, and regular pauses in transfers when it picks up new files to transfer.
  2. I have the impression that with the newer B2 backend these delays have gotten worse. (since I moved from 6.43.x to 6.44.2 with --b2-upload-concurrency 4)

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

  • os/version: debian 11.8 (64 bit)
  • os/kernel: 6.1.0-0.deb11.11-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --password-command="pass B2/backup-bc" --fast-list --transfers 10 --b2-hard-delete --b2-upload-concurrency 4 --log-level=INFO --stats-file-name-length 0 -x sync /srv/dev-disk-by-label-fs1/backup/ crypt:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[b2]
type = b2
account = XXX
key = XXX
hard_delete = true

[crypt]
type = crypt
remote = b2:XXX
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

Sorry, have no log with -vv. Can provide one when needed, but I hope my issue is clear from a software architectural standpoint.

My question

I have the impression all of crypt is in 1 thread, even when encrypting multiple files. Would it be possible to let crypt use multiple threads? 1 thread per file to be encrypted?

Will this behaviour be impacted when the memory optimised version for B2 (with 6.45?) will arrive?

If you use B2 through crypt you won't get multi threaded uploads.

It's technically possible to implement but it is quite tricky so I haven't done it yet.

Crypt should use one thread per file transfer. The encryption is very fast though so it is rarely a bottleneck. Do you see the crypt using lots of CPU?

I see a large wait period (10 minutes+) before the upload really starts, during which the CPU is at about 115%, and a batch is completed (--transfers 10, batches of 10 complete at about the same time), the upload stalls for 1 or 2 minutes while CPU again peaks at about 115%.

Before 6.44, the initial wait was about 5 minutes, and there were no large stalls.

I can provoke a log with -vv if you want, but I have the impression you know where this is coming from.

This is probably hashes being generated - try disabling it with

  --b2-disable-checksum   Disable checksums for large (> upload cutoff) files

Which version do you mean? 1.64?

If you could do a before with 1.63.x and an after with 1.64.x with the same transfer that would be useful.

(of course, 1.64)
OK, will set that up.

1 Like

Sorry, life happened, priorities moved. I am no longer able to provide more info here. It was a low priority subject anyway. You can close this topic.

1 Like