Fast for single large files but slow on small files?

Help and Support
1

Thank you for this great software. One issue I am having, when transfering large files like 30G but transfer rate is 20-30Mbit per second. But if I upload small files like website files or regular ofis files those are smaller than 1 mb, upload rate is lower than 1Mbit. Why could this be happening? I am using amazon cloud drive.

Fast one:
Transferred: 5.333 GBytes (12.934 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 31m0.8s
Transferring:

  • vzdump-qemu-309-2017_01_17-02_29_09.vma.lzo: 56% done, 13.568 MBytes/s, ETA: 19m56s

Slow one:
Transferred: 297.375 MBytes (266.869 kBytes/s)
Errors: 0
Checks: 0
Transferred: 2265
Elapsed time: 19m1s
Transferring:

  •                          Dns/stal.com.dns: 100% done, 961 Bytes/s, ETA: 0s
  • …Control/WebSitesSecuredFoldersControl.ascx: 100% done, 0 Bytes/s, ETA: 0s
  • …ePanel/Server/LogParser/Web/W3SVC178.state: 100% done, 5 Bytes/s, ETA: 0s
  • …les/MSPControl/SettingsChargifyPolicy.ascx: 100% done, 12.682 kBytes/s, ETA: 0s

With slow upload in verbose mode, I see;

2017/01/19 21:48:34 Dns/stal.com.dns: Received error: HTTP code 429: “429 Too Many Requests”: response body: “{“logref”:“de6da1d8-de77-11e6-a52a-d55a5c27d05d”,“message”:“Rate exceeded”,“code”:”"}" - low level retry 2/10

Is there any way to fix this?

2

There is quite a bit of overhead per file. Lots of small files is the worst case for it. You can try increasing --transfers that should help.

The 429 rate limit errors are a fact of life with Amazon Drive - there isn’t anything rclone can do about it.

3

Yes I found out that after I post this. Can I increase --transfers to 300 or something ? Would it hurt amazon still?

4

I’d try doubling it (the default is 4) until the transfer rate stops going up.

I think 300 is probably too big and is likely to get you banned from Amazon.