Restic backup is *abysmally* slow

Help and Support
1

What is the problem you are having with rclone?

Hi, thank you for browsing the help section and helping others!

I have two crypt remotes, one is in a iDrive e2 bucket, the other in a Backblaze b2 bucket.

My goal: To create restic backups once a day of the entire iDrive e2 bucket to the restic repo located on my B2 bucket. The purpose of this is to create redundancy so if I lose access to iDrive E2 or something goes wrong, I still have all my files on another cloud provider.

Since Restic only allows backing up from file systems, I have to:

  • restic serve the repo using rclone serve restic backblaze_crypt:ResticRepos --addr http://localhost:8081, since it's located on an rclone crypt
  • rclone mount the iDrive E2 remote using this command: rclone mount idrive_crypt: Y: -o Threadcount=16 --vfs-cache-mode full --cache-dir "%USERPROFILE%\Desktop\rclone_cache" --dir-cache-time 168h --network-mode --transfers 128 --vfs-fast-fingerprint --vfs-refresh --s3-chunk-size 16M --s3-upload-cutoff 32M --s3-upload-concurrency 32

I then use restic backup: restic -r rest:http://127.0.0.1:8081/ backup "Y:" --read-concurrency 128 --compression max -vv --iexclude-file=restic_exclude.txt --verbose

The issue: The rclone mount is a significant source of bottleneck, as the fastest it can read files from iDrive E2 is 20 files per second. This means that using restic backup on a folder that contains 12,000 files (4GB total), it takes 10 minutes each time the restic backup command is ran. This is true even if there are no file changes. Obviously waiting 10 minutes for a daily backup is no big deal, however I cannot scale this setup to my entire ~1TB worth of files, because the daily backup would take more than 24h.

I tried about 10 different cloud providers, including non-S3 ones and 20 files read per second via mount is by far the fastest, so changing cloud providers won't help. Is there anything I can do to speed this up?

The only solution I see is sync the entire iDrive bucket to a local disk each day and then create the restic backup from that. This would be much faster, but it's not an option with my current setup and goals. I also tested backing up straight to Backblaze directly instead of serving a crypt remote, but found there was a negligible speed difference.

Run the command 'rclone version' and share the full output of the command.

rclone v1.70.3
- os/version: Microsoft Windows 10 Pro 22H2 22H2 (64 bit)
- os/kernel: 10.0.19045.6093 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.24.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

iDrive e2 and Backblaze b2

2

Forget rclone.
Forget mount.

Restic on its own allows to copy snapshots between repos:

Make sure that both repos are using the same chunker parameters to ensure deduplication (all is nicely covered in restic docs).

Both cloud providers you are using allow S3 access which is natively supported by restic.
I think B2 API is supported too with some caveats - check docs.

And IMO - even better is to run two separate backups. One to iDrive and one to B2.

3

I'm not trying to copy snapshots between repos. I want to create snapshots of the iDrive E2 bucket to a restic repo. In other words, restic backup an S3 bucket.

Whether the restic repo is on a local filesystem or the cloud doesn't matter, since the crucial bottleneck is the rclone mount speed.

4

Ok. I think i misunderstood your issue.

Instead of using your complicated setup why not to use rclone sync S3: B2: directly? And enable versioning on B2 bucket instead of using restic? It would be not only super fast but thanks to its simplicity much less prone to mistakes and errors.

With rclone you can then access B2 bucket state at specific date/time in the past if you need to roll things back in time.

To limit B2 bucket size you could also configure its lifecycle rules and keep only versions from last month for example.

5

Maybe... It would require some more tests to prove it. IMO the problem might be not rclone mount speed but its nature (slightly different behaviour than "local" filesystem) which makes restic to believe that all files potentially changed and have to be fully rescanned (so all data has to be downloaded from iDrive first every time you run backup). Otherwise given that you cache all dir data it should be much faster. But hard to guess without more testing.

Might be worth to try restic backup with option --ignore-inode. They are most likely not stable when using rclone mount.

6

I'm on windows so neither of those flags do anything unfortunately. But definitely a good tip for others so thank you.

I'm almost fully certain that the files are not being downloaded, at least not their full contents. I monitored the network activity of rclone during a backup and it was around 18k bytes sent and received per second.

You mentioned cache dir data. Interestingly, the VFS cache does not get populated with folders or files basically at all. After a test restic backup of 10242 files (4.568GB), my rclone_cache folder is 503MB in Size, 4 files, 12 folders. Weirdly, the Size on disk is 11.3MB (I have CompactOS enabled). As a potential explanation, I found this comment which states that dir structure and file metadata is RAM only. There is no persistent metadata cache, evidenced by a 4 year old feature request for it on github.

I created a debug level log file for rclone serve, rclone mount, and restic backup. I'm not sure what to include, as the 3 log files combined were 1.5GB, but I can copy-paste certain sections if you have any suggestions. Here is a snippet of the rclone mount log during a restic backup.

I also found this comment by @ncw, and apparently mount does not issue S3 HEAD requests in parallel when listing files. If I had to guess this seems to be the crux of the issue.

I tested --checksum as the documentation recommends it for S3->S3 transfers, although I'm not sure if it works for mount. Also toyed around with --read-only, --no-update-dir-modtime, --no-update-modtime, --use-server-modtime. None of them made a difference.

All in all, I am fairly confident that the rclone mount speed is extremely limited and is based on cloud provider. In my testing 2 files per second was the lowest, and 20 files per second was the highest. So for example if I try to delete 200 .md files in the mounted S3 remote, it takes ~10 seconds. Deleting 2000 .md files -> 100 seconds. Thus, any file operation such as restic backup will be bottlenecked by this limitation.

Please let me know if I'm missing something, but based on this, creating restic snapshots of rclone remotes is just not feasible for any non-negligible amount of files. I'll have to rethink my backup strategy :frowning:

7

Yes... This sometimes is the only option:) rclone is great but it has its limits.

Maybe others have some better ideas. Let's see

Myself I would consider direct sync and utilisation of B2 bucket versioning. This way I would have multiple versions of my source bucket. Even if source is wiped out by some evil actor and then synced (effectively deleting everything from the destination) I would be still able to restore all content before disaster. And to be very safe maybe it would be worth to add objects locking (supported by B2). Then even most clever malware would be helpless in trying to destroy precious data. But this is very different subject.

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.