Optimize speed on sync

Help and Support
1

What is the problem you are having with rclone?

Hello, I would like to ask you if you still see optimization potential for the speed. I think there should be more speed possible. the souce has a 10gbit connection.

or do you think the speed is ok? (show upload.log)

is it ok or should it be increased? -> --tpslimit 3 --transfers 3

What is your rclone version (output from rclone version)

rclone v1.52.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

debian10

Which cloud storage system are you using? (eg Google Drive)

Google dive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

rclone sync gcrypt_upload:/archive gcrypt_upload3:/archive --checkers 3 --log-file /root/upload.log -v --tpslimit 3 --transfers 3 --drive-chunk-size 32M --create-empty-src-dirs --ignore-existing


Paste config here

[gdrive]
type = drive
client_id = cxxx.apps.googleusercontent.com
client_secret = cxx
scope = drive
root_folder_id = 0APoPI86apLCmUk9PVA
token = cxx

[gcache]
type = cache
remote = gdrive:/gdrive
plex_password = cxx
chunk_size = 5M
info_age = 1d
chunk_total_size = 10G

[gcrypt]
type = crypt
remote = gcache:/crypt
filename_encryption = standard
directory_name_encryption = true
password = cxxx
password2 = cxxx

[gcrypt_upload]
type = crypt
remote = gdrive:/gdrive/crypt
filename_encryption = standard
directory_name_encryption = true
password = cxxx
password2 = cxxx

[gdrive2]
type = drive
client_id = cxxx.apps.googleusercontent.com
client_secret = cxx
scope = drive
token = cxx
root_folder_id = 0ANvaj_UBU2XNUk9PVA

[gcache2]
type = cache
remote = gdrive2:/gdrive2
plex_password = cxx
chunk_size = 5M
info_age = 1d
chunk_total_size = 10G

[gcrypt2]
type = crypt
remote = gcache2:/crypt
filename_encryption = standard
directory_name_encryption = true
password = cxxx
password2 = cxxx

[gcrypt_upload2]
type = crypt
remote = gdrive2:/gdrive2/crypt
filename_encryption = standard
directory_name_encryption = true
password = cxxx
password2 = cxxx

[gdrive3]
type = drive
client_id = cxxx.apps.googleusercontent.com
client_secret = cxx
scope = drive
token = cxx
root_folder_id = 0ANf_5Ya7rv0uUk9PVA

[gcache3]
type = cache
remote = gdrive3:/gdrive3
plex_password = cxx
chunk_size = 5M
info_age = 1d
chunk_total_size = 10G

[gcrypt3]
type = crypt
remote = gcache3:/crypt
filename_encryption = standard
directory_name_encryption = true
password = cxxx
password2 = cxxx

[gcrypt_upload3]
type = crypt
remote = gdrive3:/gdrive3/crypt
filename_encryption = standard
directory_name_encryption = true
password = cxxx
password2 = cxxx

smaller excerpt from upload.log:

Transferred: 533.214G / 585.116 GBytes, 91%, 3.711 MBytes/s, ETA 3h58m42s
Checks: 453 / 453, 100%
Transferred: 100940 / 110946, 91%
Elapsed time: 40h52m22.8s
Transferring:

  • xxx:100% /15.891M, 0/s, -

Transferred: 533.513G / 585.271 GBytes, 91%, 3.711 MBytes/s, ETA 3h58m
Checks: 453 / 453, 100%
Transferred: 100983 / 110989, 91%
Elapsed time: 40h53m22.8s
Transferring:

  • xx: transferring
  • xx: transferring
  • xxx: transferring

Transferred: 533.833G / 585.578 GBytes, 91%, 3.712 MBytes/s, ETA 3h57m54s
Checks: 453 / 453, 100%
Transferred: 101026 / 111031, 91%
Elapsed time: 40h54m22.8s
Transferring:

  • xxx: 0% /10.230M, 0/s, -

Transferred: 534.276G / 585.815 GBytes, 91%, 3.714 MBytes/s, ETA 3h56m51s
Checks: 453 / 453, 100%
Transferred: 101068 / 111073, 91%
Elapsed time: 40h55m22.8s
Transferring:

  • xxx: 51% /103.551M, 15.998M/s, 3s
  • xxx: 69% /20.740M, 2.014M/s, 3s
2

hello,

rclone has to process each and every file.
for each file
rclone has to download the file from gcrypt_upload
decrypt the file
encrypt the file using the passwords from gcrypt_upload3
upload the file to gcrypt_upload3
to be clear, rclone does not download the entire file, but download and upload chunks at a time.

if the same encryption settings including passwords are the same, then you can sync encrypted files, without have to use the above steps and use

rclone sync gdrive:/gdrive/crypt gdrive3:/gdrive3/crypt --drive-server-side-across-configs

3

very good tip, I use the exact same encryption. i will test .)

4

I have over 1000 lines in 5 minutes in the log file
2020/05/30 16:59:42 ERROR : ih8cjhftf26fc2m98gp703jcfo/cngr4qq0pcl4vthhcjf9sj8tm0/4d5ca11cusv8e75dbb8tt6hdrc/00cr9maj7s2u19gp9ei847829jv206s96qoocakkucfebrqhme8tdgicma9dsd6pvii7sma9dr8rc/rh21006ogllbpbaku1rm33fc71f2mpk4oq7rg9oh2f55mpk531uv3h5na2el4mgn7gnng3281vu48: Failed to copy: googleapi: Error 404: File not found: 13xiCI_GwoNFAyvCLugWd4fGa2Qc-njpw., notFound
2020/05/30 16:59:43 ERROR : ih8cjhftf26fc2m98gp703jcfo/cngr4qq0pcl4vthhcjf9sj8tm0/4d5ca11cusv8e75dbb8tt6hdrc/00cr9maj7s2u19gp9ei847829jv206s96qoocakkucfebrqhme8tdgicma9dsd6pvii7sma9dr8rc/smouph63afrgbarn5v79sndc20: Failed to copy: googleapi: Error 404: File not found: 16-BVhkVgmAKWo8pOJm4O-2Y6XvFqJ6UZ., notFound
2020/05/30 16:59:43 ERROR : ih8cjhftf26fc2m98gp703jcfo/cngr4qq0pcl4vthhcjf9sj8tm0/4d5ca11cusv8e75dbb8tt6hdrc/00cr9maj7s2u19gp9ei847829jv206s96qoocakkucfebrqhme8tdgicma9dsd6pvii7sma9dr8rc/ug3pcudfhu2cultlknkgrn2r6o: Failed to copy: googleapi: Error 404: File not found: 1USCz9UDCEqPJtcUWR-pT9b84b8uwSumv., notFound
2020/05/30 16:59:43 ERROR : ih8cjhftf26fc2m98gp703jcfo/cngr4qq0pcl4vthhcjf9sj8tm0/4d5ca11cusv8e75dbb8tt6hdrc/00cr9maj7s2u19gp9ei847829jv206s96qoocakkucfebrqhme8tdgicma9dsd6pvii7sma9dr8rc/uk40th91fgvv75unrfptmlkad48b0km3tbbljdbh638vmtla6ln6p32l9lvf75ouma7fnhpk56g9s: Failed to copy: googleapi: Error 404: File not found: 1B-a__pIqeibssYKSpuFw1ntNiCqqcfc9., notFound
2020/05/30 16:59:43 ERROR : ih8cjhftf26fc2m98gp703jcfo/cngr4qq0pcl4vthhcjf9sj8tm0/4d5ca11cusv8e75dbb8tt6hdrc/00cr9maj7s2u19gp9ei847829jv206s96qoocakkucfebrqhme8tdgicma9dsd6pvii7sma9dr8rc/ukpm8vnuia0qafale12vuvug2mqmljl54o8nht1l7h0bg6vfcmj0: Failed to copy: googleapi: Error 404: File not found: 1Jzquw3EV9S6mTiyVr5ixrfLRZ8DxE_5z., notFound

is that normal?

5

what is the command you are using?

6

rclone sync gdrive:/gdrive/crypt gdrive3:/gdrive3/crypt --drive-server-side-across-configs --create-empty-src-dirs --ignore-existing --log-file /root/upload.log

7

not sure what the problem is.

on the dest, do you see the files decrypted as expected?

8

yes they are decrypted correctly, or I only see directories, no files

I want to synchronize the whole gdrive to another gdrive

9

@calisro, what do you think?.

10

404s are permission errors.

The source account needs permissions on the destination to do a 'server side' copy.

11

how do i create this?

12

Fyi If you've copied them between crypts within the crypt the noonce will be different between the remotes and it'll think the files are different if you then tried to copy them at the drive level.

13

ok, but how can I copy or synchronize the gdrive with a good speed? or what's the best way?

14

More transfers and large chunk sizes. Or server-side copies. Why are you using a tps-limit at all?

15

I once read tps-limit where to use it. What would you say, the best command to synchronize the whole gdrive with a gdrive?

16

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.