Sorry for the delay. These tests took time to run but I made some progress:
rclone check --download --log-file=check_download.log --log-level=DEBUG GCSTakeout:bucket GCSTakeout:bucket —progress
I checked the log file for pacer and found nothing.
Also, I did the same command with GDrive and still found nothing.
Here are the bandwidth numbers in MiB/s:
| rclone check --download --checkers=4 GCSTakeout:bucket GCSTakeout:bucket ---progress | 303 |
| rclone check --download --checkers=8 GCSTakeout:bucket GCSTakeout:bucket ---progress | 275 |
| rclone check --download --checkers=16 GCSTakeout:bucket GCSTakeout:bucket ---progress | 280 |
| rclone check --download --checkers=32 GCSTakeout:bucket GCSTakeout:bucket ---progress | 280 |
| rclone check --download --buffer-size=256M GCSTakeout:bucket GCSTakeout:bucket ---progress | 360 |
| rclone check --download --use-mmap GCSTakeout:bucket GCSTakeout:bucket ---progress | 355 |
| rclone check --download --multi-thread-streams=8 GCSTakeout:bucket GCSTakeout:bucket ---progress | 353 |
| rclone check --download --disable-http2 GCSTakeout:bucket GCSTakeout:bucket ---progress | 2158 |
| ^ scaled to 16 via tmux | 6461 |
| rclone check --download --disable-http2 --checkers=32 --buffer-size=8G GCSTakeout:bucket GCSTakeout:bucket ---progress | 6400 |
| rclone check --download --disable-http2 ---checkers=32 ---buffer-size=128M GCSTakeout:bucket GCSTakeout:bucket ---progress | 6553 |
| rclone check --download --disable-http2 --checkers=128 --buffer-size=1G GCSTakeout:bucket GCSTakeout:bucket ---progress | 5632 |
| rclone check --download --disable-http2 --checkers=64 --buffer-size=1G GCSTakeout:bucket GCSTakeout:bucket ---progress | 5836 |
| ^ scaled to 16 via tmux | 6144 |
As you can see --disable-http2 seemed to really have unlocked some bottleneck with increasing checkers really helping.
I also tried the same commands with GDrive instead of GCSTakeout. Here's the table, with CPU utilization and speed in MiB/s
|rclone check --download --checkers=4 GDrive:folder GDrive:folder —progress|1%|175|
|rclone check --download --checkers=8 GDrive:folder GDrive:folder —progress|1.8%|370|
|rclone check --download --checkers=16 GDrive:folder GDrive:folder —progress|2.5%|710|
|rclone check --download --checkers=32 GDrive:folder GDrive:folder —progress|2.8%|1400|
|rclone check --download --checkers=128 GDrive:folder GDrive:folder —progress|26%|5632|
|rclone check —download —checkers=512 GDrive:folder GDrive:folder —progress|26%|5632|
Seems like here, checkers were the only thing that unlocked the bottleneck.
Again, can't get above ~6GiB/s
But we've made a lot of progress!
I tried a new copy command:
rclone -v copy --disable-http2 --checkers=32 GCSTakeout:bucket rGDrive:folder --drive-chunk-size=1G --transfers=32 --progress and was able to get 2GiB/s at ~17% CPU Utilization.