Anyone else noticing performance issues with IDrive E2 (S3 compatible cloud storage - IDrive® e2)?
Trying to list a bucket with about 54K objects using rclone lsd -R idrive: takes forever, sometimes over 20 minutes. I assume it's an IDrive E2 problem and not something rclone may be doing.
try --fast-list
and if that does not solve the issue, need to see the answers to the help and support template.
Sorry, I should have mentioned I tried both with and without --fast-list both are very slow.
With fast list listing 54k files should take about 54 transactions. If this were aws I'd expect that to take a second or two.
Try running with -vv ---dump headers to see the transactions - that might give a clue.
Some responses are just ridiculously slow, an example:
2022/07/08 16:13:01 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/08 16:13:01 DEBUG : HTTP REQUEST (req 0xc000524500)
2022/07/08 16:13:01 DEBUG : GET /...?delimiter=%2F&max-keys=1000&prefix=... HTTP/1.1^M
Host: h7e2.sj11.idrivee2-10.com^M
User-Agent: rclone/v1.58.1^M
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855^M
X-Amz-Date: 20220708T231301Z^M
Accept-Encoding: gzip^M
2022/07/08 16:14:07 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/08 16:14:07 DEBUG : HTTP RESPONSE (req 0xc000524500)
2022/07/08 16:14:07 DEBUG : HTTP/1.1 200 OK^M
Connection: close^M
Content-Security-Policy: block-all-mixed-content^M
Content-Type: application/xml^M
Date: Fri, 08 Jul 2022 23:14:07 GMT^M
Server: nginx^M
Strict-Transport-Security: max-age=31536000; includeSubDomains^M
Vary: Origin^M
Vary: Accept-Encoding^M
X-Amz-Request-Id: 16FFFD8FAD2DD7E3^M
X-Content-Type-Options: nosniff^M
X-Xss-Protection: 1; mode=block^M
That doesn't look like an rclone problem.
Do you have all 54k files in one directory? Maybe that is causing the slowdown?
It's actually a restic backup using rclone as the backend to talk to E2. It creates a folder structure from what I can tell data/00, data/01, ... data/ff, each one with 200-220 files. I sampled listing individual folders data/00 etc and each one seems to be relatively fast at 2-4s.
2-4s! If I tried that on AWS it will be back in 100ms.
Indeed. It's a new service, it appears they have work to do to make it better.
I signed up in Ireland region and I get loads of api timeouts. Have raised support tickets and they just say its been sorted. I think whatever they are using in AWS to host this is way under resourced (and I have told them that). I even tested with the AWS cli and its the same. If it does not improve soon I will switch to real S3 buckets (you can get 5Gb free via the AWS free tier and then just just pay for what you use above this).
hello and welcome to the forum,
there are a lot of S3 providers to choose from, each has its own issues.
AWS can be very expensive.
in my case, i use a combo of
--- wasabi, S3 clone known for hot storage. i believe wasabi has best api performance.
$5.99 for the first TiB and then prorated after that. to store recent backups
--- AWS S3 deep glacier.
$0.99/TiB/month. to store older backups
I don't think they use AWS based on:
104.200.149.252 registered to Total Server Solutions/Performive LLC which seems to be a managed cloud providerUnfortunately IDrive E2 is not ready for prime time, they seem to have major performance issues and bugs. You were lucky they even responded to your support ticket, I filed one over 10 days ago and no reply, I pinged them in the support chat and just getting the same pre-canned reply "we'll get back to you".
$4/1st year/TB is cheap enough to have another archive copy, but make sure it's not your only backup, I don't think they can be trusted with the safety of our data, at least not yet.
Backblaze B2 remains the better choice for now.
The Ireland IP shows up as being a Digiweb Ltd IP, and a Google shows them to be an ISP/hosting service in Dublin. I am just setting up syncing to an AWS S3 bucket. Will run the AWS one for a month and keep an eye on the billing to see how the costs work out on a fresh new Free Tier account. Lots of good advice on this thread so will look into these too!
amunro,
that was a good dig into idrive.e2
i would be curious to know what you learn about the various S3 providers?
and if any S3 provider can match wasabi in terms of the throttling of api calls/?
Based on above comments, what would everyone say needs to be specifically addressed by iDrive E2 in terms of performance (eg specific api calls and so on)?
Depending on the problems, we could still attempt to inform iDrive E2 of specific performance flaws to see if they are willing to work on it.
Unfortunately their support leaves a lot to be desired. I filed a ticket on July 5 and I have yet to get a reply from them, beyond the generic "we received it and we'll get back to you". I pinged them via chat and I get the same generic reply each time "our engineers will get back to you". This tells me all I need to know about the quality of E2, imagine having more serious issues, data loss, etc.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.