Rclone transfer speed is very slow with respect to normal copy paste from local drive to external drive

Help and Support
1

What is the problem you are having with rclone?

I'm facing slow transfer speeds when moving files from a local drive to an external APFS drive on a Mac M3 (8-core, 16GB RAM). The transfer is even slower with exFAT. We have 500,000 files, each 512 bytes in size. Using rclone, the transfer takes 1 hour, 21 minutes, and 16.4 seconds, whereas a normal copy-paste operation takes only 4-5 minutes.

On a Mac M1 (8-core, 8GB RAM) with the same data and external drive, rclone takes over 7 hours, while normal copy-paste takes just 8-10 minutes.

Is there any way to make rclone's copy speed comparable to that of a normal copy-paste?

Run the command 'rclone version' and share the full output of the command.

sh-3.2# ./rclone version
rclone v1.68.0

  • os/version: darwin 14.6 (64 bit)
  • os/kernel: 23.6.0 (arm64)
  • os/type: darwin
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.23.0
  • go/linking: dynamic
  • go/tags: none
    sh-3.2#

Which cloud storage system are you using? (eg Google Drive)

not using cloud storage. source is local drive and destination is external drive.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

sh-3.2# ./rclone copy /Users/e2test/Downloads/5lac /Volumes/Idrive/5lac_threads_4 --progress
Transferred:   	  244.141 MiB / 244.141 MiB, 100%, 128.271 KiB/s, ETA 0s
Transferred:       500000 / 500000, 100%
Elapsed time:   1h21m16.4s

sh-3.2# ./rclone copy /Users/e2test/Downloads/5lac /Volumes/Express\ Drive/5lac/ --transfers=16 --fast-list --progress
2024/10/25 23:00:40 NOTICE: Config file "/var/root/.config/rclone/rclone.conf" not found - using defaults
Transferred:   	  244.141 MiB / 244.141 MiB, 100%, 76.269 KiB/s, ETA 0s
Transferred:       500000 / 500000, 100%
Elapsed time:   1h22m27.9s

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

not required as we are not adding remote

A log from the command that you were trying to run with the -vv flag

Paste  log here
2

And curiousity question. What real life problem you want to solve? If cp is much faster than use cp....

rclone can be much slower as it validates all transfers with checksums (so everything is read again from destination). You can see what rclone is doing by using DEBUG mode:

./rclone copy /Users/e2test/Downloads/5lac /Volumes/Idrive/5lac_threads_4 --progress -vv

Search forum how to disable it - then for sure it will be faster.

3

I have need to run copy command asynchronously and get progress stats on regular interval using api call. rclone rc is fulfilling my requirement but it is extremely slow w.r.t cp command.

yes I went through the rclone docs and tried here is the result.
sh-3.2# ./rclone copy /Users/e2test/Downloads/5lac/ /Volumes/test/5lac_test1 --ignore-checksum --local-no-check-updated --progress --inplace --no-check-dest -vv
Transferred: 244.141 MiB / 244.141 MiB, 100%, 281.409 KiB/s, ETA 0s
Transferred: 500000 / 500000, 100%
Elapsed time: 58m33.8s

I see the improvement of ~25 mins, without disabling checks It was taking 1hr22min.
But still it is very far from normal copy paste(5mins). Do you see here I can improve here by any means.

I tried with --transfers=16 --buffer-size=64Mi but it didn't helped
./rclone copy /Users/e2test/Downloads/5lac/ /Volumes/test/5lac_test1 --transfers=16 --buffer-size=64Mi --ignore-checksum --local-no-check-updated --inplace --no-check-dest --progress -vv --log-file=rclone.log

Transferred: 244.141 MiB / 244.141 MiB, 100%, 272.551 KiB/s, ETA 0s
Transferred: 500000 / 500000, 100%
Elapsed time: 59m3.7s

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.