Slowdown with chunker vs with no chunker

Help and Support
1

What is the problem you are having with rclone?

I have noticed that operations have a noticeable slowdown when operating through the chunker vs when not, even if the files in question being operated on are small enough to not apply to the chunker's rules. Is this normal?

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
  • os/kernel: 10.0.22621.2070 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Box

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move "Z:\rclone\vfs\boxenc" "boxencnochunk:" -vvvvvv -P --fast-list --order-by 'size,ascending' --transfers 20

The rclone config contents with secrets removed.

[box]
type = box
token = 

[boxenc]
type = crypt
remote = boxchunk:Encrypted
filename_encoding = base32768
directory_name_encryption = true
password = 
password2 = 

[boxencnochunk]
type = crypt
remote = box:Encrypted
filename_encoding = base32768
directory_name_encryption = true
password = 
password2 = 

[boxchunk]
type = chunker
remote = box:
chunk_size = 4Gi
hash_type = sha1quick
fail_hard = true

A log from the command with the -vv flag

an excerpt from the log as I unfortunately can't share the full thing.

2023-08-08 21:30:18 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000045157: Computing sha1 hash of encrypted source
2023-08-08 21:30:18 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000045956: Computing sha1 hash of encrypted source
2023-08-08 21:30:18 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000045015: Computing sha1 hash of encrypted source
2023-08-08 21:30:19 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000046075: Computing sha1 hash of encrypted source
2023-08-08 21:30:22 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000046595: Computing sha1 hash of encrypted source
2023-08-08 21:30:23 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000046155: Computing sha1 hash of encrypted source
2023-08-08 21:30:24 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000046795: Computing sha1 hash of encrypted source
2023-08-08 21:30:25 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000046995: Computing sha1 hash of encrypted source
2023-08-08 21:30:26 DEBUG : Synced/Roms Cloned/Nintendo - Nintendo 3DS/3DS English Eshop Archive/ninja/US/en/titles/online_prices%3Ftitle%5B%5D%3D50010000046796: Computing sha1 hash of encrypted source
Transferred:      208.211 MiB / 1.442 TiB, 0%, 2.763 MiB/s, ETA 6d7h58m
Checks:                92 / 112, 82%
Deleted:               32 (files), 0 (dirs)
Renamed:               56
Transferred:           28 / 10049, 0%
Elapsed time:      1m26.9s
Checking:

Transferring:
 * Synced/Roms Cloned/Nin…Aftermarket) (Unl).gbc:100% /64Ki, 823/s, -
 * Synced/Roms Cloned/Nin…ridge (Japan) (En).gbc:100% /64Ki, 938/s, -
 * Synced/Roms Cloned/Nin…Unknown) (Proto 2).gbc:100% /64Ki, 897/s, -
 * Synced/Roms Cloned/Nin…market) (Homebrew).GBC:100% /64Ki, 772/s, -
 * Synced/Roms Cloned/Nin…Aftermarket) (Unl).GBC:100% /64Ki, 981/s, -
 * Synced/Roms Cloned/Nin…ennis (USA, Europe).gb:100% /64Ki, 1.178Ki/s, -
 * Synced/Roms Cloned/Nin…5B%5D%3D50010000043855:113% /343, 8/s, -
 * Synced/Roms Cloned/Nin…5B%5D%3D50010000044015:113% /343, 8/s, -
 * Synced/Roms Cloned/Nin…5B%5D%3D50010000044635:113% /343, 11/s, -
 * Synced/Roms Cloned/Nin…5B%5D%3D50010000044636:113% /343, 15/s, -
 * Synced/Roms Cloned/Nin…5B%5D%3D50010000044735:113% /343, 21/s, -
 * Synced/Roms Cloned/Nin…5B%5D%3D50010000045015:113% /343, 22/s, -
 * Synced/Roms Cloned/Nin…5B%5D%3D50010000045157:113% /343, 24/s, -
2

The log is from running through the chunker. The issue does not show up without the chunker, so I didn't include an excerpt from that log.

3

Yes it is normal. Chunker requires at least some optimizations and ideally remaining issues fixing (and some are not trivial):

If you are willing to help you are welcomed.

4

Ok, thanks, I wanted to make sure I wasn't just being a bonehead somehow and flubbing a setting up or something.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.