--no-traverse too slow with lot of files

Help and Support
1

What is the problem you are having with rclone?

--no-traverse is too slow with lot of files.
example test: We have millions of files in the remote backend and hundreds localy to send it.

with --no-traverse we get 167.521 KiB/s

without --no-traverse we get 31.446 MiB/s

like we talk here: Rclone copy local to remote downloading data from remote

if we dont't use --no-traverse a lot of api cals with older files in remote backend are done and and it incurs a lot of cost.

we don't need the full 31.446 MiB/s but if it could be a litte fast would be a great improvement. Any ideia?

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.13.0-1019-gcp (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.8
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

google cloud storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy 18 remote:teste-bucket-sp/18 --no-traverse -vv --config /root/rclone.conf
rclone copy 18 remote:teste-bucket-sp/19 -vv --config /root/rclone.conf

The rclone config contents with secrets removed.

[remote]
type = google cloud storage
service_account_file =

A log from the command with the -vv flag

the two logs bellow the remote backend are clean with no files. trying to send hundreds of files.

2022/03/24 16:32:03 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "18" "remote:teste-bucket-sp/18" "--no-traverse" "-vv" "--config" "/root/rclone.conf"]
2022/03/24 16:32:03 DEBUG : Creating backend with remote "18"
2022/03/24 16:32:03 DEBUG : Using config file from "/root/rclone.conf"
2022/03/24 16:32:03 DEBUG : fs cache: renaming cache item "18" to be canonical "/local/18"
2022/03/24 16:32:03 DEBUG : Creating backend with remote "remote:teste-bucket-sp/18"
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 8.812356ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 17.986841ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 27.954072ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 37.535528ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 47.397513ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 57.118006ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 66.492779ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 76.454167ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 75.789037ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 85.635785ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 95.587729ms
2022/03/24 16:32:03 DEBUG : pacer: Reducing sleep to 99.397171ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 9.165719ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 17.514978ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 9.763753ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 18.58938ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 28.467771ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 37.490386ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 47.184043ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 57.062211ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 58.731724ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 63.112734ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 11.96018ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 9.720835ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 19.206224ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 20.96118ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 28.8606ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 36.885306ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 44.473724ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 54.076377ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 63.395375ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 72.310109ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 80.914237ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 67.708502ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 3.555997ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 7.340804ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 9.083025ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 18.994705ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 27.379992ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 37.169534ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 43.623681ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 52.910564ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 60.952829ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 70.281792ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 75.049709ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 80.329992ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 9.727351ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 19.161479ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 28.812367ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 9.108832ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 18.165682ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 28.016599ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 37.643842ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 47.123062ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 48.604088ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 50.893486ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 55.535087ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 64.729488ms
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 54.341616ms
2022/03/24 16:32:04 DEBUG : 1/18ED105A/8427D9E6/8427DCAE: md5 = 002f990a88b5146ecf14aac43da862be OK
2022/03/24 16:32:04 INFO  : 1/18ED105A/8427D9E6/8427DCAE: Copied (new)
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 60.655218ms
2022/03/24 16:32:04 DEBUG : 1/2C69F9E4/2C69FA64/2C69FA65: md5 = b5fc3fee1ae03e0a17d6b37c738fe8f6 OK
2022/03/24 16:32:04 INFO  : 1/2C69F9E4/2C69FA64/2C69FA65: Copied (new)
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 69.021477ms
2022/03/24 16:32:04 DEBUG : 1/2A731949/EDEE2086/58522E3A: md5 = c2f47284c80d525236f50b5c8d8fe20f OK
2022/03/24 16:32:04 INFO  : 1/2A731949/EDEE2086/58522E3A: Copied (new)
2022/03/24 16:32:04 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:04 DEBUG : 1/00BBD4E0/097580C2/097580C1: md5 = bc6bfc80d6288ee3db5f73fb8aa6a667 OK
2022/03/24 16:32:04 INFO  : 1/00BBD4E0/097580C2/097580C1: Copied (new)
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 8.72787ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 16.424543ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 26.210325ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 36.091226ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 46.019478ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 55.762037ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 61.052987ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 67.87264ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 73.481326ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 82.988443ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 89.723127ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 38.171559ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 9.017984ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 18.757363ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 27.990585ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 37.794677ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 47.700368ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 57.488797ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 66.466527ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 76.239594ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 84.305194ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 92.446813ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 101.96623ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 110.747672ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 118.813931ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 119.862315ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 18.076697ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 9.181573ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 18.35942ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 28.28402ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 38.076219ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 47.778888ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 57.569782ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 66.927683ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 76.69799ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 85.982171ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 95.865457ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 101.796646ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 109.981627ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 109.605425ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 6.812633ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 7.989857ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 17.057684ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 26.749938ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 36.67687ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 46.480012ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 48.481127ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 49.429723ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 53.738706ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 50.889774ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 54.687893ms
2022/03/24 16:32:05 DEBUG : 1/262DA13B/26CA6CE1/11837991: md5 = 3763e67916d0c4110934b9ee3302e7cb OK
2022/03/24 16:32:05 INFO  : 1/262DA13B/26CA6CE1/11837991: Copied (new)
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 50.843618ms
2022/03/24 16:32:05 DEBUG : 1/46A60529/68ABE51A/68ABE51B: md5 = 2f2a875c79fc178d57c67c50dae0e5ce OK
2022/03/24 16:32:05 INFO  : 1/46A60529/68ABE51A/68ABE51B: Copied (new)
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 60.520195ms
2022/03/24 16:32:05 DEBUG : 1/00BBD4E0/25572813/2A4B210A: md5 = 9b05b00c6c6efc76a5a1049d103610e9 OK
2022/03/24 16:32:05 INFO  : 1/00BBD4E0/25572813/2A4B210A: Copied (new)
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 68.437744ms
2022/03/24 16:32:05 DEBUG : 1/46A60529/46A608AB/46A608AC: md5 = 9a43fe9f06222c04ff9902b80ce37bd2 OK
2022/03/24 16:32:05 INFO  : 1/46A60529/46A608AB/46A608AC: Copied (new)
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 73.472024ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 27.968463ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 9.016787ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 18.924779ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 28.151229ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 38.026178ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 47.715475ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 57.364999ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 67.037386ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 76.324567ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 84.967871ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 88.90329ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 89.579655ms
2022/03/24 16:32:05 DEBUG : pacer: Reducing sleep to 90.718493ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 10.454657ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 8.018494ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 1.298034ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 4.148721ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 13.083623ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 22.700772ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 32.219915ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 40.959033ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 50.831445ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 60.732432ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 70.355073ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 79.92081ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 89.08458ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 98.86092ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 108.663848ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 116.00073ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 40.984869ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 9.070396ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 18.870902ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 28.3967ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 38.274361ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 47.576525ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 57.443461ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 67.253866ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 77.026472ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 86.724818ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 96.176496ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 105.618268ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 111.884896ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 120.268547ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 120.007746ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 7.4788ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 9.146169ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 18.226246ms
2022/03/24 16:32:06 DEBUG : pacer: Reducing sleep to 27.633088ms

2022/03/24 16:35:56 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "18" "remote:teste-bucket-sp/19" "-vv" "--config" "/root/rclone.conf"]
2022/03/24 16:35:56 DEBUG : Creating backend with remote "18"
2022/03/24 16:35:56 DEBUG : Using config file from "/root/rclone.conf"
2022/03/24 16:35:56 DEBUG : fs cache: renaming cache item "18" to be canonical "/local/18"
2022/03/24 16:35:56 DEBUG : Creating backend with remote "remote:teste-bucket-sp/19"
2022/03/24 16:35:56 DEBUG : 1/2A731949/EDEE2086/58522E3A: md5 = c2f47284c80d525236f50b5c8d8fe20f OK
2022/03/24 16:35:56 INFO  : 1/2A731949/EDEE2086/58522E3A: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 8.774937ms
2022/03/24 16:35:56 DEBUG : 1/2C69F9E4/2C69FA64/2C69FA65: md5 = b5fc3fee1ae03e0a17d6b37c738fe8f6 OK
2022/03/24 16:35:56 INFO  : 1/2C69F9E4/2C69FA64/2C69FA65: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 14.977883ms
2022/03/24 16:35:56 DEBUG : 1/18ED105A/8427D9E6/8427DCAE: md5 = 002f990a88b5146ecf14aac43da862be OK
2022/03/24 16:35:56 INFO  : 1/18ED105A/8427D9E6/8427DCAE: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:56 DEBUG : 1/2C69FD0A/2C69FD0D/B9DF0295: md5 = 81a14287c61e951a57a198f1b240ede6 OK
2022/03/24 16:35:56 INFO  : 1/2C69FD0A/2C69FD0D/B9DF0295: Copied (new)
2022/03/24 16:35:56 DEBUG : 1/46A60529/46A608AB/46A608AC: md5 = 9a43fe9f06222c04ff9902b80ce37bd2 OK
2022/03/24 16:35:56 INFO  : 1/46A60529/46A608AB/46A608AC: Copied (new)
2022/03/24 16:35:56 DEBUG : 1/2C69FD0A/2C69FD0D/B9DEFB12: md5 = 51ccad6b416f7ece0c8c1bdfe46279b2 OK
2022/03/24 16:35:56 INFO  : 1/2C69FD0A/2C69FD0D/B9DEFB12: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 7.253141ms
2022/03/24 16:35:56 DEBUG : 1/00BBD4E0/25572813/2A4B210A: md5 = 9b05b00c6c6efc76a5a1049d103610e9 OK
2022/03/24 16:35:56 INFO  : 1/00BBD4E0/25572813/2A4B210A: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 11.631882ms
2022/03/24 16:35:56 DEBUG : 1/00BBD4E0/097580C2/097580C1: md5 = bc6bfc80d6288ee3db5f73fb8aa6a667 OK
2022/03/24 16:35:56 INFO  : 1/00BBD4E0/097580C2/097580C1: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:56 DEBUG : 1/00BBD4E0/25572813/AF36D239: md5 = e675866093d58d17ced167fa97ad42ce OK
2022/03/24 16:35:56 INFO  : 1/00BBD4E0/25572813/AF36D239: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 447.324µs
2022/03/24 16:35:56 DEBUG : 1/00BBD4E0/25572813/62DB7585: md5 = e108ed6f26ef031c4dc85e8c245584cc OK
2022/03/24 16:35:56 INFO  : 1/00BBD4E0/25572813/62DB7585: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:56 DEBUG : 1/00BBD4E0/25572813/80248969: md5 = fe048e3fa743960d24667c25ed188762 OK
2022/03/24 16:35:56 INFO  : 1/00BBD4E0/25572813/80248969: Copied (new)
2022/03/24 16:35:56 DEBUG : 1/4B1B14AC/8FD0996C/92FE1A86: md5 = ef5c81250ffae53f825e48088746de21 OK
2022/03/24 16:35:56 INFO  : 1/4B1B14AC/8FD0996C/92FE1A86: Copied (new)
2022/03/24 16:35:56 DEBUG : 1/46A60529/68ABE51A/68ABE51B: md5 = 2f2a875c79fc178d57c67c50dae0e5ce OK
2022/03/24 16:35:56 INFO  : 1/46A60529/68ABE51A/68ABE51B: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 5.318892ms
2022/03/24 16:35:56 DEBUG : 1/67507933/16775FFF/CB67C568: md5 = a8739f1ebfb34cfa8b22854b7dbd7c69 OK
2022/03/24 16:35:56 INFO  : 1/67507933/16775FFF/CB67C568: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 12.819196ms
2022/03/24 16:35:56 DEBUG : 1/67507933/16775FFF/746CC9FB: md5 = c379bf89cbb0b32f4df88cf677ecf151 OK
2022/03/24 16:35:56 INFO  : 1/67507933/16775FFF/746CC9FB: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:56 DEBUG : 1/49F533DF/2586FE80/69C826F6: md5 = 01e4e447dd8b33f4551c35acacd8f607 OK
2022/03/24 16:35:56 INFO  : 1/49F533DF/2586FE80/69C826F6: Copied (new)
2022/03/24 16:35:56 DEBUG : pacer: Reducing sleep to 7.49162ms
2022/03/24 16:35:56 DEBUG : 1/49F533DF/2586FE80/3BC4B226: md5 = adaf19ed25a5832da40198530ab5c34b OK
2022/03/24 16:35:56 INFO  : 1/49F533DF/2586FE80/3BC4B226: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:57 DEBUG : 1/262DA13B/26CA6CE1/11837991: md5 = 3763e67916d0c4110934b9ee3302e7cb OK
2022/03/24 16:35:57 INFO  : 1/262DA13B/26CA6CE1/11837991: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 9.90834ms
2022/03/24 16:35:57 DEBUG : 1/49F533DF/2586FE80/B85B021F: md5 = a63dfc7ee39876feb8761d43575b32f3 OK
2022/03/24 16:35:57 INFO  : 1/49F533DF/2586FE80/B85B021F: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD86: md5 = c1b78118b1d43a9eadffd3c8848bb92b OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD86: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 3.525761ms
2022/03/24 16:35:57 DEBUG : 1/67507934/DBA4147E/CA271B00: md5 = 097095818b0668ccc28577694ea4fd21 OK
2022/03/24 16:35:57 INFO  : 1/67507934/DBA4147E/CA271B00: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 8.106101ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD87: md5 = 63386f3b9826caee7bf80e4ed92c46ef OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD87: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 9.825295ms
2022/03/24 16:35:57 DEBUG : 1/67507934/DBA4147E/F94006CB: md5 = 9395a405987d935020c9753efb879530 OK
2022/03/24 16:35:57 INFO  : 1/67507934/DBA4147E/F94006CB: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 2.459486ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD88: md5 = 8d06b36f380ae534a84b5aac960d8e90 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD88: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD89: md5 = d0877e7dfcc09f21c833aa988cca6a8a OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD89: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 7.339393ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD8A: md5 = 8148e787c511c28ca19e9b06a26b4e86 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD8A: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 5.192189ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD8C: md5 = 6b0badd123742413aac8f58a0f149c06 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD8C: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 11.234228ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD8B: md5 = 3b7e782f20176d783f648d4fd706f953 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD8B: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 589.482µs
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD8D: md5 = 2e46553087cacc11351680afa24f3950 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD8D: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDA5: md5 = dac0f43af1119329ff18e8c2698c8aba OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDA5: Copied (new)
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD8E: md5 = 3e45d93cc03b651e5cf6572d68db8dd9 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD8E: Copied (new)
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDA6: md5 = 0b0cf6e638c7f22b57b7747cd48878ed OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDA6: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 9.646675ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DD8F: md5 = 2aba71d8a5aabee60ffe990b703b93bf OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DD8F: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDA8: md5 = ef5ed01d9c6a910379e5e1d09c2df1ba OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDA8: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 5.701622ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDA7: md5 = 1225d6a831f9286f90170ebb92e0b40b OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDA7: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 4.621649ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDA9: md5 = 8643acffdea651d900147c9836f527ad OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDA9: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 7.061849ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDAA: md5 = 4ba80cf1d8510146d64ac6311d5024e0 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDAA: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 11.316956ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDAB: md5 = c6a313d16d21e6d3c0b2772381e61762 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDAB: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 9.855594ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDAD: md5 = 53e94f8459c2f06f8ff3b9bb8f634607 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDAD: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 6.422275ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDAE: md5 = 91849cfa83fd651e32b29f08c997d7eb OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDAE: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 1.145831ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E050: md5 = fa05350f3f0505c4bf353d122c0c31ac OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E050: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E04F: md5 = 40fba9b8132d121ba63db211ee903627 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E04F: Copied (new)
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E051: md5 = e7a295a813b7630e46b3bc7b324fa0f1 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E051: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 3.502249ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427DDAC: md5 = e1c9ae6000c6a7c6821634d26d36794b OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427DDAC: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 3.378386ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E053: md5 = 3a379c07ff8f4b5ced20ab9d80a46824 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E053: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 9.333668ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E052: md5 = e7c856c24df47725f456270e428b9fc7 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E052: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 14.319731ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E054: md5 = 6c18d462ae4b41dc3b56011ad6b5b8ca OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E054: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 15.346453ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E055: md5 = a0aa6fe4d68386c1829f7f6976d86eac OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E055: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 10.553259ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E056: md5 = 4e4e3b4ce8a1d7a25fe0aea0bebf87ce OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E056: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 17.353891ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E057: md5 = f23a032bd4920ac65e9808ec623e9831 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E057: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 21.439237ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E058: md5 = bdfcb4ebb41c79eb506c3bf31fd4030e OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E058: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 18.362423ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E06E: md5 = 793f5ee63a6a23fbad6b33d0809fa7da OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E06E: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E070: md5 = 8db8949ec767146f1f4040c50c35bcfa OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E070: Copied (new)
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E071: md5 = 618b1f0316f3260f21fa502f581ab44f OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E071: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 9.722806ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E072: md5 = 642e1d2b565d79ff3e24c3edfbf7bfe7 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E072: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 18.084244ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E073: md5 = e13e034339cfa10843446b32d47f7cde OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E073: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 23.440503ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E06F: md5 = 5ce18984a1b2a9aeb2e9cb3b814e712c OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E06F: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 13.400484ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E074: md5 = 5689bbb4556e6d9dc589cd1f9b6243fc OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E074: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 22.680938ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E075: md5 = fed23aec6eafe8cffe09819c116b7f36 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E075: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 24.32233ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E076: md5 = af35bc52b6f171ae962ec3683acf38ad OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E076: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 4.811521ms
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E077: md5 = 496bdabe6e20b6849e603ca80b28e354 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E077: Copied (new)
2022/03/24 16:35:57 DEBUG : pacer: Reducing sleep to 0s
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E08D: md5 = 6fa63a5a5a72e79731368b945bb05c96 OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E08D: Copied (new)
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E08E: md5 = 274411ad8776fa42b863f5a8f4104b1a OK
2022/03/24 16:35:57 INFO  : 1/18ED105A/8427DD69/8427E08E: Copied (new)
2022/03/24 16:35:57 DEBUG : 1/18ED105A/8427DD69/8427E090: md5 = 8a3f5d6a1772ce81710c1e6926b05a71 OK
2

You can use a top-up sync strategy to speed up syncs enormously

Let's say you run a top-up sync every 1 hour, then you might do

rclone copy --max-age 1h --no-traverse /path remote:

To only consider files which has changed within the last hour.

Then once a day (say) you run a full rclone sync which will sync deletions and anything missed.

You probably would want to use --no-traverse for this but it might be faster without if all the updates to files tend to be in one directory.

1 Like
3

we already do that. the problem is hundreds of files in just one hour... :sweat_smile:

this can't be done because we are talking about files in the folder that are only on remote backend.

4

Are you sure that the files don't exist remotely and do exist locally?

If so you could try

  --no-check-dest          Don't check the destination, copy regardless

This doesn't scan the remote at all so will upload things multiple times if you let it but might be what you need.

5

the commands above are no files on remote backend.
but in production there will be many of the same files already on the remote backend. So this flag cannot be used.

6

The worst --no-check-dest will do is upload files that are already there. Is that likely to be a problem?

7

maybe not. but as it can be many gigabytes it may take too long.

8

I guess the question to ask is why is --no-traverse taking so long? I don't think it should take so long - on google cloud storage it should be quick.

Can you send me a log with -vv of the transfer? You can email it to me nick@craig-wood.com or share it with that email address

9

exactly

attached here 2 minutes log... if you need more let me know
rclone.log (507.7 KB)

10

Isn't the obvious answer in the logs the same as in your original post at the top?

You are getting rate limited by Google as that's what is making it slow.

11

what king of rate limited? so I could look on google platform...
why without --no-traverse don't get it?

12

I could be misreading the log, but those are what is slowing you down I'd imagine. The pacer is slowing you down, but I'm not familiar with GCS so not sure if that's a normal or non normal thing.

13

I don’t know GCS either, but I do have some knowledge of the pacer and I find it difficult to understand the program flow behind the pacer entries in your debug log.

I would therefore like to see the first 10 seconds of a log where you have added --dump headers - if that isn’t too much trouble or contradicted/obsoleted by @ncw

14

with --dump headers
rclone.log (3.6 MB)

15

Thanks, my first impression in this somewhat unknown territory (GCS):

Rclone requests the (metadata) of each of your source objects from GCS and receives a HTTP404/NotFound response as expected in your scenario, but it seems like the HTTP404 is somehow interpreted as an (retriable) error that triggers the pacer to do a slowdown.

This is either a bug in rclone or in my current understanding of rclone/GCS.

I will dig deeper, it will probably take some time.

16

here may have the google cloud rates...

we have also AWS s3 and we have also similar questions. rclone sync is mutch faster (big diference) than rclone copy with --no-traverse.

17

Are you able to reproduce on a (free) OneDrive Personal account? Then I will probably be able to reproduce and debug in known territory and code :slight_smile:

18

done with OneDrive!

also has big difference (but big) with and without --no-traverse.

rclone.log (1.3 MB)

19

OK, I think I found my misunderstanding and learned something new :slightly_smiling_face:

Short conclusion: rclone pacing is fine and you are not being slowed by GCS.

Still in somewhat unknown territory, but it seems like the rclone default pacing parameters for GCS are too conservative. I therefore suggest you try adding --drive-pacer-min-sleep=0ms. If this causes throttling, then try 2ms or 10ms.

I will take a look at the OneDrive log tomorrow, time to sleep here.

20

that is for gdrive, not google cloud storage.
perhaps GCS should have an equivalent flag.