How to move data from one iDrive e2 location / bucket to another

[mar (Markus)]

INTRO

I am an absolut newbie when it comes to rclone, so that's only the most basic version of it. It was of help for me though, so I put the most important things together, it might be of help to someone else, especially as iDrive e2 support is pretty much non-existent. I am using iDrive e2 as an offsite backup location for WordPress sites. Those serve as additional backups, to have independent backups from the hosting provider.

---

ABOUT IDRIVE E2

An iDrive e2 Storage account can have different buckets on different locations. Because of a problem I could not solve, I was forced to migrate over from iDrive e2 Frankfurt to iDrive e2 Paris. In my particular case that meant, moving over all existing backups (folders, subfolders, files) as well.

The web interface from iDrive e2 does not support copying/moving folders and files from one bucket to another bucket. You can only copy/move individual files within a bucket, therefore it's pretty much useless for what I needed.

---

From how I see it there are only three options to move over data from one iDrive e2 location to another iDrive e2 location:

1. Download and upload again
2. Using a client, e.g. I used Forklift a macOS Finder replacement to connect to both iDrive locations and just drag & drop individual folders (not sure if that is actually server to server)
3. Using rclone

OPTION 1: Download and upload again

I think that's the easiest, and most unreliable option overall. Hardly any preparation needed, but I would guess large download/upload jobs directly from the laptop/pc could be problematic

OPTION 2: Using a client like Forklift

From my experience this worked, however it was unreliable, that means some files could not get copied, I needed to skip them and manually compare later on. I did only a very rough calculation on speed, I would not say it was actually slower than rclone, but I probably only moved 20GBs. However, I am not sure if this is a server to server job, or if that's download/upload in the end

OPTION 3: Using rclone to copy a folder including subfolders and files from iDrive e2 storage location Frankfurt to Paris

1. Install rclone
2. Setup remote locations
3. Prepare the copy command (and test it with a smaller folder first)
4. Run the rclone copy command

ad 1:
Install of rclone is described somewhere else in the documentation / community

ad 2:
Setting up iDrive e2 as new remote is best described here including screenshots: Rclone - IDrive® e2

• Run the command rclone config
• Select the option n) New remote and enter the name
• Select Amazon S3 Compliant Storage Providers in Option Storage
  • 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi \ (s3)
• Choose S3 provider as /IDrive e2 in Option provider
• Select Enter AWS credentials in the next step
• Enter access_key_id> and secret_access_key>
• Select Owner gets FULL_CONTROL. 1 | No one else has access rights (default).
• Edit advanced config → no
• Select "Yes this is ok"

When doing this for both iDrive e2 storage locations, it'll look like this.

ad 3:
The official iDrive e2 documentation only shows how to copy the entire content of the full bucket. The following command shows how it works for a subfolder including folder names that come with a space:

rclone copy --verbose idrive-frankfurt:bucketname/'folder with space'/subfolder idrive-paris:bucketname/'folder with space'/

---

Helpful resources:

Cloud Migration using IDrive® e2 and rclone

Rclone - IDrive® e2

[kapitainsky]

Thank you for this guide.

1. Do you know if transferring data between two different regions (e.g. like in your example Frankfurt <->Paris) counts as egress traffic?

2. Within the same region copy/move is done server-side. I am guessing that between different regions it is not?

[mar (Markus)]

1. Do you know if transferring data between two different regions (e.g. like in your example Frankfurt <->Paris) counts as egress traffic?

I didn't particularly research about this, I think from what I remember egress traffic is included up to 3x to what is stored

Within the same region copy/move is done server-side. I am guessing that between different regions it is not?

That's probably true, I was just not sure if my local client is needed or somewhere in between as well

[alneven]

What was the problem with Frankfurt?

[M_R]

I think Frankfurt is to slow. In my tests the bucket in germany was terrible slow.

[mar (Markus)]

It worked perfectly fine for my use case which is uploading daily backups at night. However, my hosting provider decided to give up on the hosting location in Germany, and focus on UK only. So my entire WHM account including all cPanel accounts got migrated over to another server/infrastructure in the UK. From then on I had daily problems.

After tons of investigation and testing, it was 100% clear the only problem that is left is somewhere between iDrive e2 Frankfurt and the new server in the UK (based on OVH infrastructure). The old server in Germany was based on Hetzner infrastructure.

As soon as I changed everything to iDrive e2 Paris, there was no more error. 100% success in uploading every single night.

It might be the fault of the UK server. It might be the fault of iDrive e2 Frankfurt. Both parties say everything works, but it didn't. Whitelisting didn't change anything. In the end after 50 hours testing, I was ready to let iDrive e2 Frankfurt go. Still waiting for the iDrive support by the way (worst ever unfortunately)

[alneven]

Thanks Markus!
it is good to know
as if I would go later on with iDrive e2, my server is also at Hetzner, and I would have added the bucket in Germany
but I guess I red here somewhere that someone else had also problems with the iDrive location in Germany, so not 100% sure what would make sense
speed is not a question, but if transactions fail that is bad

[mar (Markus)]

I read somewhere on reddit about issues with iDrive e2 Frankfurt in the last months, but then I read the same about iDrive e2 Paris. Difficult how to interpret that.

Fact is, for me it worked very reliable until that change of my hosting provider. I am not able to tell on which end the actual problem occurs. Either iDrive e2 has a problem with the new server/location/network. Or the other way round. Everyone tells me "all is good" which obviously is not.

Fact is also, the support of iDrive is an absolute nightmare and iDrive has a bad reputation for a reason.

I transferred all data now with rclone and it also failed partly and I got errors. I am now manually comparing the files and manually copy missing files from Frankfurt to Paris with Forklift.