B2 Error - couldn't list single level: Too Many Requests

[talkingpixels (Tom Scott)]

Hi, I’m running a Rclone (v1.36 on Ubuntu 14.06, 64-bit) to sync ~275GB of data from Dropbox to B2 (~200,000 files in ~35,000 directories), all goes smoothly until Rclone hits a directory with 2,386 child directories, after which the processes appears to abort.

Here’s the command I’m using
$ sudo -b rclone sync dropbox:/ backblaze:dropbox --transfers 10

And here’s the errors I’m getting…
2017/04/24 00:16:16 ERROR : B2 bucket dropbox: not deleting files as there were IO errors
2017/04/24 00:16:16 ERROR : Attempt 1/3 failed with 0 errors and: error reading source directory “”: couldn’t list single level: Too Many Requests
2017/04/24 00:16:18 ERROR : B2 bucket dropbox: not deleting files as there were IO errors
2017/04/24 00:16:18 ERROR : Attempt 2/3 failed with 0 errors and: error reading source directory “”: couldn’t list single level: Too Many Requests
2017/04/24 00:16:19 ERROR : B2 bucket dropbox: not deleting files as there were IO errors
2017/04/24 00:16:19 ERROR : Attempt 3/3 failed with 0 errors and: error reading source directory “”: couldn’t list single level: Too Many Requests
2017/04/24 00:16:19 Failed to sync: error reading source directory “”: couldn’t list single level: Too Many Requests

Any help gratefully appreciated

[ncw (Nick Craig-Wood)]

Possibly related to this issue: https://github.com/ncw/rclone/issues/1277

[talkingpixels (Tom Scott)]

I’m on a paid account and my cap limit isn’t even close to being hit, so I don’t think that’s the cause. As an aside, I have posted a separate reply about that issue here…

New sync method in v1.36 and Backblaze B2 Class C Transaction Cap

Hi, I'm having a similar issue. I have about ~275GB of data synced from Dropbox to B2 (~200,000 files in ~35,000 directories). I'd like to have a daily cron run to keep it up to date. However, each time I run rclone (without --old-sync-method) it uses 10,000+ transactions. Although the cost is negligible for a day, over a year this would nearly double the cost of me using B2. Keeping some kind of low-transaction method, without too higher price to pay in terms of speed and memory usage, would …

Unless I’m missing something, they appear to be separate issues.

[theevo]

Hi there! I’m encountering the same error also. Were you able resolve your issue, @talkingpixels?

[ncw (Nick Craig-Wood)]

Hi there! I’m encountering the same error also. Were you able resolve your issue,

Use a recent rclone, say the latest release and use the --fast-list flag. Provided you have enough memory to store the listing of your files this should fix the problem.