I Rate ACD @ 0 and they are Rate Limiting my to 0 as well

Help and Support
1

Hi, I’m hoping someone can lend a hand with my “fiasco” … I’ve spent time researching on my own and I’m just stumped.

  • Scope of issue: Trying to migrate 62.2 TB off of ACD before it is deleted (will toss a bit of that, but need to review first)
  • ISP: 100/70 Mbps connection
  • FreeNAS AIO: 96TB Raw HDD Capacity

I used Cloudberry Explorer to upload these Linux ISOs; however, when attempting to recently reverse the process (ACD => Local) every file I attempted to move resulted in an error “Object or Element Not Found” (can’t recall exactly).

As an aside, the Linux ISO joke is funny and I just learned of it recently, but prior to learning how to appropriately perform offsite backup, I 7z’ed files with a p/w in a size of 8 GB and uploaded. Hey, at least I was sorta encrypting. At any rate, far from a best case scenario.

acd

Having read the news, I thought that rclone would’t be a feasible option, until I camne across this gem: Proxy for Amazon Cloud Drive

I managed to pull around 100GB off ACD and then RCLONE just stopped (more on that).

Log file (substituted more meaningful names where appropriate):
Version: rclone-v1.38-050-g4a1013f2-windows-amd64
rclone sync acd:directory\subdirectory \\freenas\pool\SMBShare\directory --acd-templink-threshold 0 --low-level-retries 1 --stats 10s --log-file=FILE -vv

rclone: Version “v1.38-050-g4a1013f2” starting with parameters [“rclone” “sync” “acd:Backups\Dir\SubDir” \\sullynas\Pool\SMBShare\Dir" “–acd-templink-threshold” “0” “–stats” “10s” “–log-file=FILE” “-vv”]
2017/11/02 12:02:39 DEBUG : amazon drive root ‘Backups/Dir/SubDir’: Token expired but no uploads in progress - doing nothing
2017/11/02 12:02:39 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2017/11/02 12:02:39 DEBUG : acd: Saved new token in config file
2017/11/02 12:02:40 INFO : amazon drive root ‘Backups …’: Modify window not supported
2017/11/02 12:02:41 DEBUG : FILENAME.7z.003: Downloading large object via tempLink
2017/11/02 12:02:41 DEBUG : FILENAME.002: Downloading large object via tempLink
2017/11/02 12:02:41 DEBUG : FILENAME.004: Downloading large object via tempLink
2017/11/02 12:02:41 DEBUG : FILENAME.7z.001: Downloading large object via tempLink
2017/11/02 12:02:41 DEBUG : pacer: Rate limited, sleeping for 593.167455ms (1 consecutive low level retries)
2017/11/02 12:02:41 DEBUG : pacer: low level retry 1/10 (error )
2017/11/02 12:02:41 DEBUG : pacer: Rate limited, sleeping for 1.424990005s (2 consecutive low level retries)
2017/11/02 12:02:41 DEBUG : pacer: low level retry 1/10 (error )
2017/11/02 12:02:41 DEBUG : pacer: Rate limited, sleeping for 1.081266286s (3 consecutive low level retries)
2017/11/02 12:02:41 DEBUG : pacer: low level retry 1/10 (error )
2017/11/02 12:02:41 DEBUG : pacer: Rate limited, sleeping for 4.003431565s (4 consecutive low level retries)
2017/11/02 12:02:41 DEBUG : pacer: low level retry 1/10 (error )
2017/11/02 12:02:41 DEBUG : pacer: Rate limited, sleeping for 1.016469993s (5 consecutive low level retries)
2017/11/02 12:02:41 DEBUG : pacer: low level retry 2/10 (error )
2017/11/02 12:02:41 DEBUG : pacer: Rate limited, sleeping for 23.634344457s (6 consecutive low level retries)
2017/11/02 12:02:41 DEBUG : pacer: low level retry 2/10 (error )
2017/11/02 12:02:42 DEBUG : pacer: Rate limited, sleeping for 14.698025498s (7 consecutive low level retries)
2017/11/02 12:02:42 DEBUG : pacer: low level retry 2/10 (error )

017/11/02 13:00:39 DEBUG : pacer: Rate limited, sleeping for 3m25.513408922s (35 consecutive low level retries)
2017/11/02 13:00:39 DEBUG : pacer: low level retry 9/10 (error <nil>)
2017/11/02 13:00:43 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:    55m41.1s
Transferring:
 * FILENAME.7z.001
 * FILENAME.7z.002
 * FILENAME.7z.003
 * FILENAME.7z.004

I think one of my issues is that a sub dir name on ACD = “Real Estate” and shows up in the log as xxxReal" “Estate” but that is far from explaining the rate limiting. Help please! And thank you.

2

I noticed that ACD has been behaving pretty erratic (based on my observation). Certain days, it can be good and certain days I’ll get what you were experiencing (rate limited). Not sure if this is going to help but if you were to lower the transfers using --transfers 1 will that do? If not I believe you may have to stop and retry a few hours later (at least that’s what I did and the following day I managed to transfer out “slowly” using low transfers).

3

Thanks so much for that reply. I will certainly give your suggestion a shot!

Stupid question and I’m sure there are more variables than I’m not considering (just curious for my understanding):

  • If transfers = X (flag undefined), will rclone take my available downlink of 100 Mbps and allocate across each transfer, such that if X were 10, I’m downloading at 10 Mbps per stream?
  • If transfers = 1 (flag defined), will rclone take my available downlink 100 Mbps such that I’m downloading at 100 Mbps?

Again, just curious as to how the underpinnings work, and in absence of being able to actually use my full downlink, I will certainly take 10 Mbps over 0 Mbps!

I’ll report back with my findings and again, thanks!

4

I kicked off two rclone commands to see how they run today and looks like I’m saturating my downlink :slight_smile:

(this is without setting transfers=1)

down

5

I think probably Nick would be a better person to reply you on that but based on my observation, the download will not split equally among each transfers. If transfers is not defined it will be 4 transfers and it won’t be 25Mbps each transfer (again based on my observation).
If however you specify transfers=1 I believe it will take up to your available bandwidth that is allowable from ACD

6

It is somewhat random exactly how the transfers will split. It depends on just how the TCP connections perform and the OS of the sender and the receiver.