I am trying to move large files (several hundred megabyte to several gigabyte) from a remote server to my local machine. The local machine is on a gigabit network, the remote server is on a 100gigabit network. I would like to use rclone to do this I need assistance with the right method/configuration. I am able to use rsync to saturate my network connection if I run 4-5 separate instances of rsync.. looking for something easier to use.
$ rclone version rclone v1.62.2 - os/version: ubuntu 22.04 (64 bit) - os/kernel: 6.2.11-2-pve (x86_64) - os/type: linux - os/arch: amd64 - go/version: go1.20.2 - go/linking: static - go/tags: none
I have tried a few different connections these are the results. I should be able to hit >100MB/sec reliably but cant get the configuration quite right:
$ rclone move -Pv --transfers 8 --sftp-chunk-size 252K --sftp-concurrency=64 xx:/home/xx/files/ ~/test/ Transferred: 505.723 MiB / 9.779 GiB, 5%, 35.421 MiB/s, ETA 4m28s Checks: 8 / 16, 50% Transferred: 0 / 17, 0% Elapsed time: 16.2s Checking: Transferring: * xx: 6% /667.289Mi, 2.918Mi/s, 3m34s * xx: 7% /715.390Mi, 3.692Mi/s, 2m59s * xx: 9% /544.178Mi, 3.568Mi/s, 2m18s * xx: 5% /481.266Mi, 1.934Mi/s, 3m54s * xx:12% /543.059Mi, 4.623Mi/s, 1m43s * xx: 15% /510.189Mi, 5.590Mi/s, 1m16s * xx: 18% /500.551Mi, 6.539Mi/s, 1m2s * xx: 14% /656.244Mi, 6.557Mi/s, 1m25s
$ rclone copy -Pv --transfers 8 xx-new:/files ~/test/ Transferred: 54.870 MiB / 9.779 GiB, 1%, 12.602 MiB/s, ETA 13m10s Transferred: 0 / 17, 0% Elapsed time: 5.4s Transferring: * xx: 8% /644.473Mi, 12.602Mi/s, 46s * xx: 0% /484.078Mi, 0/s, - * xx: 0% /498.227Mi, 0/s, - * xx: 0% /737.565Mi, 0/s, - * xx: 0% /822.344Mi, 0/s, - * xx: 0% /512.079Mi, 0/s, - * xx: 0% /621.024Mi, 0/s, - * xx: 0% /505.861Mi, 0/s, -
$ rclone config Current remotes: Name Type ==== ==== xx sftp xx-new ftp
The log has a lot of PII to strip out, let me know if you need this.