How to move data from one iDrive e2 location / bucket to another

Howto Guides
1

INTRO

I am an absolut newbie when it comes to rclone, so that's only the most basic version of it. It was of help for me though, so I put the most important things together, it might be of help to someone else, especially as iDrive e2 support is pretty much non-existent. I am using iDrive e2 as an offsite backup location for WordPress sites. Those serve as additional backups, to have independent backups from the hosting provider.

ABOUT IDRIVE E2

An iDrive e2 Storage account can have different buckets on different locations. Because of a problem I could not solve, I was forced to migrate over from iDrive e2 Frankfurt to iDrive e2 Paris. In my particular case that meant, moving over all existing backups (folders, subfolders, files) as well.

The web interface from iDrive e2 does not support copying/moving folders and files from one bucket to another bucket. You can only copy/move individual files within a bucket, therefore it's pretty much useless for what I needed.

From how I see it there are only three options to move over data from one iDrive e2 location to another iDrive e2 location:

  1. Download and upload again
  2. Using a client, e.g. I used Forklift a macOS Finder replacement to connect to both iDrive locations and just drag & drop individual folders (not sure if that is actually server to server)
  3. Using rclone

OPTION 1: Download and upload again

I think that's the easiest, and most unreliable option overall. Hardly any preparation needed, but I would guess large download/upload jobs directly from the laptop/pc could be problematic

OPTION 2: Using a client like Forklift

From my experience this worked, however it was unreliable, that means some files could not get copied, I needed to skip them and manually compare later on. I did only a very rough calculation on speed, I would not say it was actually slower than rclone, but I probably only moved 20GBs. However, I am not sure if this is a server to server job, or if that's download/upload in the end

OPTION 3: Using rclone to copy a folder including subfolders and files from iDrive e2 storage location Frankfurt to Paris

  1. Install rclone
  2. Setup remote locations
  3. Prepare the copy command (and test it with a smaller folder first)
  4. Run the rclone copy command

ad 1:
Install of rclone is described somewhere else in the documentation / community

ad 2:
Setting up iDrive e2 as new remote is best described here including screenshots: Rclone - IDriveĀ® e2

  • Run the command rclone config
  • Select the option n) New remote and enter the name
  • Select Amazon S3 Compliant Storage Providers in Option Storage
    • 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi \ (s3)
  • Choose S3 provider as /IDrive e2 in Option provider
  • Select Enter AWS credentials in the next step
  • Enter access_key_id> and secret_access_key>
  • Select Owner gets FULL_CONTROL. 1 | No one else has access rights (default).
  • Edit advanced config ā†’ no
  • Select "Yes this is ok"

When doing this for both iDrive e2 storage locations, it'll look like this.
2023-09-10_002535@2x

ad 3:
The official iDrive e2 documentation only shows how to copy the entire content of the full bucket. The following command shows how it works for a subfolder including folder names that come with a space:

rclone copy --verbose idrive-frankfurt:bucketname/'folder with space'/subfolder idrive-paris:bucketname/'folder with space'/

Helpful resources:

1 Like
2

Thank you for this guide.

  1. Do you know if transferring data between two different regions (e.g. like in your example Frankfurt <->Paris) counts as egress traffic?

  2. Within the same region copy/move is done server-side. I am guessing that between different regions it is not?

3
  1. Do you know if transferring data between two different regions (e.g. like in your example Frankfurt <->Paris) counts as egress traffic?

I didn't particularly research about this, I think from what I remember egress traffic is included up to 3x to what is stored

Within the same region copy/move is done server-side. I am guessing that between different regions it is not?

That's probably true, I was just not sure if my local client is needed or somewhere in between as well

4

What was the problem with Frankfurt?

5

I think Frankfurt is to slow. In my tests the bucket in germany was terrible slow.

1 Like
6

It worked perfectly fine for my use case which is uploading daily backups at night. However, my hosting provider decided to give up on the hosting location in Germany, and focus on UK only. So my entire WHM account including all cPanel accounts got migrated over to another server/infrastructure in the UK. From then on I had daily problems.

After tons of investigation and testing, it was 100% clear the only problem that is left is somewhere between iDrive e2 Frankfurt and the new server in the UK (based on OVH infrastructure). The old server in Germany was based on Hetzner infrastructure.

As soon as I changed everything to iDrive e2 Paris, there was no more error. 100% success in uploading every single night.

It might be the fault of the UK server. It might be the fault of iDrive e2 Frankfurt. Both parties say everything works, but it didn't. Whitelisting didn't change anything. In the end after 50 hours testing, I was ready to let iDrive e2 Frankfurt go. Still waiting for the iDrive support by the way (worst ever unfortunately)

1 Like
7

Thanks Markus!
it is good to know
as if I would go later on with iDrive e2, my server is also at Hetzner, and I would have added the bucket in Germany
but I guess I red here somewhere that someone else had also problems with the iDrive location in Germany, so not 100% sure what would make sense
speed is not a question, but if transactions fail that is bad

8

I read somewhere on reddit about issues with iDrive e2 Frankfurt in the last months, but then I read the same about iDrive e2 Paris. Difficult how to interpret that.

Fact is, for me it worked very reliable until that change of my hosting provider. I am not able to tell on which end the actual problem occurs. Either iDrive e2 has a problem with the new server/location/network. Or the other way round. Everyone tells me "all is good" which obviously is not.

Fact is also, the support of iDrive is an absolute nightmare and iDrive has a bad reputation for a reason.

I transferred all data now with rclone and it also failed partly and I got errors. I am now manually comparing the files and manually copy missing files from Frankfurt to Paris with Forklift.

1 Like
9

Moving Backups Between iDrive e2 Locations: A Beginner's Guide with rclone
This guide is for complete beginners who want to migrate their WordPress backups stored on iDrive e2 from one location (e.g., Frankfurt) to another (e.g., Paris). Since the iDrive e2 web interface doesn't support inter-bucket transfers, we'll be using a powerful tool called rclone.

Before We Begin:

This guide assumes you're new to rclone.
Make sure you have a stable internet connection for the transfer.
The Problem:

iDrive e2 offers different storage locations (buckets) but lacks a built-in way to move data between them. This guide tackles this by utilizing rclone for a smooth migration.

The Options:

Download & Upload (Unreliable): Easy but time-consuming and prone to errors, especially for large backups.
Third-Party Client (Uncertain): Tools like Forklift might work, but reliability can be a concern. It's also unclear if the transfer happens server-to-server.
rclone (Recommended): This is the most efficient and reliable option. It allows server-to-server transfers and detailed progress monitoring.
Let's Use rclone!

Install rclone: Refer to the official rclone documentation for installation instructions.
Set Up Remote Locations: Here's how to configure rclone for your iDrive e2 buckets:
Run rclone config in your terminal.
Choose "n) New remote" and enter a name (e.g., "idrive-frankfurt").
Select "Amazon S3 Compliant Storage Providers."
Choose "/IDrive e2" as the S3 provider.
Enter your iDrive e2 access credentials (access_key_id and secret_access_key).
Repeat this process to set up a separate remote for your Paris location (e.g., "idrive-paris").
Moving Your Backups:

The official iDrive e2 documentation focuses on copying entire buckets. Here's the command for copying a specific folder (including subfolders) with spaces in the name:

rclone copy --verbose idrive-frankfurt:bucketname/'folder with space'/subfolder idrive-paris:bucketname/'folder with space'/subfolder
Explanation:

rclone copy: This initiates the copy operation.
--verbose: Provides detailed progress information during the transfer.
idrive-frankfurt: Source location name (replace with yours).
bucketname: The name of your bucket on the Frankfurt location.
'folder with space': The folder you want to copy (including spaces).
subfolder: The specific subfolder within the chosen folder (optional).
idrive-paris: Destination location name (replace with yours).
Important Notes:

Replace bucketname, folder with space, and subfolder names with your actual details.
Test the command with a smaller folder first to ensure everything works as expected.
This guide assumes you have the necessary permissions to access both iDrive e2 buckets.
Conclusion:

By following these steps, you can successfully migrate your WordPress backups between iDrive e2 locations using rclone. This method is reliable, efficient, and provides clear progress tracking.

10

This all happened before my ventures with iDrive e2... which gave me the same results. Frankfurt was going way too slowly way too often, so I moved everything to Paris. It went well for a while, but Paris too gave me one incident of slowness, which for me is one too many.

I have now moved to Jottacloud and Pcloud, they seem to be more consistently fast. Let's see how it goes.