Fast for single large files but slow on small files?

Thank you for this great software. One issue I am having, when transfering large files like 30G but transfer rate is 20-30Mbit per second. But if I upload small files like website files or regular ofis files those are smaller than 1 mb, upload rate is lower than 1Mbit. Why could this be happening? I am using amazon cloud drive.

Fast one:
Transferred: 5.333 GBytes (12.934 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 31m0.8s

  • vzdump-qemu-309-2017_01_17-02_29_09.vma.lzo: 56% done, 13.568 MBytes/s, ETA: 19m56s

Slow one:
Transferred: 297.375 MBytes (266.869 kBytes/s)
Errors: 0
Checks: 0
Transferred: 2265
Elapsed time: 19m1s

  •                          Dns/ 100% done, 961 Bytes/s, ETA: 0s
  • …Control/WebSitesSecuredFoldersControl.ascx: 100% done, 0 Bytes/s, ETA: 0s
  • …ePanel/Server/LogParser/Web/W3SVC178.state: 100% done, 5 Bytes/s, ETA: 0s
  • …les/MSPControl/SettingsChargifyPolicy.ascx: 100% done, 12.682 kBytes/s, ETA: 0s

With slow upload in verbose mode, I see;

2017/01/19 21:48:34 Dns/ Received error: HTTP code 429: “429 Too Many Requests”: response body: “{“logref”:“de6da1d8-de77-11e6-a52a-d55a5c27d05d”,“message”:“Rate exceeded”,“code”:”"}" - low level retry 2/10

Is there any way to fix this?

There is quite a bit of overhead per file. Lots of small files is the worst case for it. You can try increasing --transfers that should help.

The 429 rate limit errors are a fact of life with Amazon Drive - there isn’t anything rclone can do about it.

Yes I found out that after I post this. Can I increase --transfers to 300 or something ? Would it hurt amazon still?

I’d try doubling it (the default is 4) until the transfer rate stops going up.

I think 300 is probably too big and is likely to get you banned from Amazon.