Abnormal transfer speed during rclone sync

Help and Support
1

What is the problem you are having with rclone?

rclone sync --metadata --checkers=50 --transfers=50 --buffer-size=64M --cache-chunk-size=64M --create-empty-src-dirs mnt/testhr3/.snap/snap1 mnt/testhr4 --progress
The transfer progress is stuck.

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.0-DEV

  • os/version: newstart 6.06 (64 bit)
  • os/kernel: 5.10.134-14.zncgsl6.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.1
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

local

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --metadata --checkers=50 --transfers=50 --buffer-size=64M --cache-chunk-size=64M --create-empty-src-dirs mnt/testhr3/.snap/snap1 mnt/testhr4  --progress -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

Paste config here

A log from the command that you were trying to run with the -vv flag

2025/02/12 18:27:02 INFO  : vdb.1_14.dir/vdb_f0240.file.ac74ca9c.partial: Removing failed copy
Transferred:       13.623 GiB / 23.476 GiB, 58%, 3.609 MiB/s, ETA 46m35s
Transferred:        13940 / 24040, 58%
Elapsed time:      3m21.4s
Transferring:
 *                   vdb.1_44.dir/vdb_f0010.file:  0% /1Mi, 0/s, -
 *                    vdb.1_1.dir/vdb_f0016.file:  0% /1Mi, 0/s, -
 *                   vdb.1_32.dir/vdb_f0046.file:  0% /1Mi, 0/s, -
 *                   vdb.1_52.dir/vdb_f0188.file:  0% /1Mi, 0/s, -
 *                   vdb.1_25.dir/vdb_f0311.file:  0% /1Mi, 0/s, -
 *                   vdb.1_22.dir/vdb_f0296.file:  0% /1Mi, 0/s, -
 *                   vdb.1_19.dir/vdb_f0278.file:  0% /1Mi, 0/s, -
 *                   vdb.1_36.dir/vdb_f0278.file:  0% /1Mi, 0/s, -
2025/02/12 18:27:02 INFO  : Exiting...
Transferred:       13.623 GiB / 23.476 GiB, 58%, 3.609 MiB/s, ETA 46m35s
Transferred:        13940 / 24040, 58%
Elapsed time:      3m21.4s
Transferring:
 *                   vdb.1_44.dir/vdb_f0010.file:  0% /1Mi, 0/s, -
 *                    vdb.1_1.dir/vdb_f0016.file:  0% /1Mi, 0/s, -
 *                   vdb.1_32.dir/vdb_f0046.file:  0% /1Mi, 0/s, -
 *                   vdb.1_52.dir/vdb_f0188.file:  0% /1Mi, 0/s, -
 *                   vdb.1_25.dir/vdb_f0311.file:  0% /1Mi, 0/s, -
 *                   vdb.1_22.dir/vdb_f0296.file:  0% /1Mi, 0/s, -
 *                   vdb.1_19.dir/vdb_f0278.file:  0% /1Mi, 0/s, -
 *                   vdb.1_36.dir/vdb_f0278.file:  0% /1Mi, 0/s, -
 *                   vdb.1_41.dir/vdb_f0278.file:  0% /1Mi, 0/s, -
 *                   vdb.1_43.dir/vdb_f0296.file:  0% /1Mi, 0/s, -
 *                   vdb.1_16.dir/vdb_f0278.file:  0% /1Mi, 0/s, -
 *                   vdb.1_16.dir/vdb_f0283.file: 49% /1Mi, 15.610Ki/s, 33s
 *                   vdb.1_22.dir/vdb_f0327.file: 49% /1Mi, 15.609Ki/s, 33s
 *                    vdb.1_1.dir/vdb_f0224.file: 49% /1Mi, 15.609Ki/s, 33s
 *                    vdb.1_2.dir/vdb_f0299.file: 49% /1Mi, 15.610Ki/s, 33s
 *                   vdb.1_48.dir/vdb_f0291.file: 49% /1Mi, 15.610Ki/s, 33s
 *                   vdb.1_46.dir/vdb_f0283.file: 49% /1Mi, 15.608Ki/s, 33s
 *                   vdb.1_52.dir/vdb_f0283.file: 49% /1Mi, 15.609Ki/s, 33s
 *                   vdb.1_39.dir/vdb_f0291.file: 49% /1Mi, 16.650Ki/s, 30s
 *                    vdb.1_4.dir/vdb_f0284.file: 49% /1Mi, 16.650Ki/s, 30s
 *                   vdb.1_31.dir/vdb_f0301.file: 49% /1Mi, 16.650Ki/s, 30s
 *                   vdb.1_24.dir/vdb_f0299.file: 49% /1Mi, 16.650Ki/s, 30s
 *                   vdb.1_30.dir/vdb_f0290.file: 49% /1Mi, 16.650Ki/s, 30s
 *                   vdb.1_25.dir/vdb_f0352.file: 49% /1Mi, 16.649Ki/s, 30s
 *                   vdb.1_41.dir/vdb_f0299.file: 49% /1Mi, 16.650Ki/s, 30s
 *                   vdb.1_27.dir/vdb_f0241.file: 49% /1Mi, 16.649Ki/s, 30s
 *                   vdb.1_35.dir/vdb_f0284.file: 49% /1Mi, 16.650Ki/s, 30s
 *                   vdb.1_51.dir/vdb_f0284.file: 49% /1Mi, 16.649Ki/s, 30s
 *                   vdb.1_20.dir/vdb_f0283.file: 24% /1Mi, 8.259Ki/s, 1m33s
 *                   vdb.1_33.dir/vdb_f0284.file: 24% /1Mi, 8.260Ki/s, 1m33s
 *                   vdb.1_10.dir/vdb_f0224.file: 24% /1Mi, 8.260Ki/s, 1m33s
 *                   vdb.1_23.dir/vdb_f0296.file: 24% /1Mi, 8.260Ki/s, 1m33s
 *                   vdb.1_37.dir/vdb_f0294.file: 24% /1Mi, 8.259Ki/s, 1m33s
 *                   vdb.1_13.dir/vdb_f0241.file: 24% /1Mi, 8.260Ki/s, 1m33s
 *                   vdb.1_49.dir/vdb_f0292.file:  5% /1Mi, 1.966Ki/s, 8m10s
 *                   vdb.1_43.dir/vdb_f0302.file:  5% /1Mi, 1.966Ki/s, 8m10s
 *                   vdb.1_38.dir/vdb_f0291.file:  5% /1Mi, 1.966Ki/s, 8m10s
 *                   vdb.1_26.dir/vdb_f0242.file: 24% /1Mi, 8.260Ki/s, 1m33s
 *                   vdb.1_28.dir/vdb_f0292.file:  5% /1Mi, 1.966Ki/s, 8m10s
 *                  vdb.1_100.dir/vdb_f0296.file:  5% /1Mi, 1.966Ki/s, 8m10s
2

And what is abnormal here? It looks like you are trying to bite off more than you can chew:)

I would get rid of these flags unless you have NVME SSD over some very capable link like TB4. And even then you might have to lower it. Otherwise you are asking your drive to do too much.

3

But the problem still exists when not set these flags.

rclone sync --metadata --create-empty-src-dirs mnt/testhr3/.snap/snap1 mnt/testhr4 --progress
2025/02/12 18:57:01 ERROR : Local file system at /home/vtu/lhrtest/mnt/testhr3/.snap/snap1: xattrs not supported - disabling: xattr.get /home/vtu/lhrtest/mnt/testhr3/.snap/snap1/vdb.1_1.dir ceph.dir.quota_users: no data available
Transferred: 8.989 GiB / 18.767 GiB, 48%, 42.993 MiB/s, ETA 3m52s
Transferred: 9206 / 19218, 48%
Elapsed time: 6m1.5s
Transferring:

  •               vdb.1_26.dir/vdb_f0311.file:  0% /1Mi, 0/s, -

  •               vdb.1_29.dir/vdb_f0009.file:  0% /1Mi, 0/s, -

  •                vdb.1_3.dir/vdb_f0296.file:  0% /1Mi, 0/s, -

  •               vdb.1_27.dir/vdb_f0009.file:  0% /1Mi, 0/s, -^C[
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.