Crypt and multithreading (crypt+B2 got slower?)

[hb020]

What is the problem you are having with rclone?

My situation: mainly multiple 2GB files, to be copied to B2 via crypt.
Multi-core x64 machine, 2GHz clock.

2 things:

1. I see that crypt is single threaded: on a multi-core machine, it peaks out 1 core, and does not use other cores. And it seems to require to have encrypted the whole file before starting to transfer data. Result: long delays (up to 10 minutes) before start of transfer, and regular pauses in transfers when it picks up new files to transfer.
2. I have the impression that with the newer B2 backend these delays have gotten worse. (since I moved from 6.43.x to 6.44.2 with --b2-upload-concurrency 4)

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

• os/version: debian 11.8 (64 bit)
• os/kernel: 6.1.0-0.deb11.11-amd64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.21.3
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --password-command="pass B2/backup-bc" --fast-list --transfers 10 --b2-hard-delete --b2-upload-concurrency 4 --log-level=INFO --stats-file-name-length 0 -x sync /srv/dev-disk-by-label-fs1/backup/ crypt:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[b2]
type = b2
account = XXX
key = XXX
hard_delete = true

[crypt]
type = crypt
remote = b2:XXX
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

Sorry, have no log with -vv. Can provide one when needed, but I hope my issue is clear from a software architectural standpoint.

My question

I have the impression all of crypt is in 1 thread, even when encrypting multiple files. Would it be possible to let crypt use multiple threads? 1 thread per file to be encrypted?

Will this behaviour be impacted when the memory optimised version for B2 (with 6.45?) will arrive?

[ncw (Nick Craig-Wood)]

If you use B2 through crypt you won't get multi threaded uploads.

It's technically possible to implement but it is quite tricky so I haven't done it yet.

Crypt should use one thread per file transfer. The encryption is very fast though so it is rarely a bottleneck. Do you see the crypt using lots of CPU?

[hb020]

I see a large wait period (10 minutes+) before the upload really starts, during which the CPU is at about 115%, and a batch is completed (--transfers 10, batches of 10 complete at about the same time), the upload stalls for 1 or 2 minutes while CPU again peaks at about 115%.

Before 6.44, the initial wait was about 5 minutes, and there were no large stalls.

I can provoke a log with -vv if you want, but I have the impression you know where this is coming from.

[ncw (Nick Craig-Wood)]

@hb020

I see a large wait period (10 minutes+) before the upload really starts, during which the CPU is at about 115%, and a batch is completed (--transfers 10, batches of 10 complete at about the same time), the upload stalls for 1 or 2 minutes while CPU again peaks at about 115%.

This is probably hashes being generated - try disabling it with

  --b2-disable-checksum   Disable checksums for large (> upload cutoff) files

@hb020

Before 6.44, the initial wait was about 5 minutes, and there were no large stalls.

Which version do you mean? 1.64?

@hb020

I can provoke a log with -vv if you want, but I have the impression you know where this is coming from.

If you could do a before with 1.63.x and an after with 1.64.x with the same transfer that would be useful.

[hb020]

(of course, 1.64)
OK, will set that up.

[hb020]

Sorry, life happened, priorities moved. I am no longer able to provide more info here. It was a low priority subject anyway. You can close this topic.