Block-Level Deduplication + Compression Remote

Feature
1

Hello,

I've been a fan of swiftsync rclone for a lot of years, and I'm also a fan of Kopia.

Kopia is a CLI backup tool (also written in Go) that performs block-level deduplication and compression on the contents. Data is managed as snapshots, which is quite a different architecture compared to rclone.

I've always wanted an rclone remote that can deduplicate data to maximize my storage usage.

Recently I took inspiration from how kopia splits contents to make my own rclone remote called "dedup."

The dedup backend is a new rclone overlay backend that wraps another remote and provides block-level deduplication and compression. It uses content-defined chunking (rolling hash) to split files into variable-size chunks, hashes each chunk with keyed BLAKE2b-128, compresses with zstd, and stores only unique chunks on the underlying remote. A JSON manifest file per logical file records the chunk list needed to reconstruct it.

This is similar to Kopia's content-defined chunking, but without snapshot management — it's a transparent overlay. Files appear normally, and deduplication happens automatically behind the scenes.

Results

Original Data Set

Total size: 917.384 MiB

Dedup remote (no compression)

Total size: 690.422 MiB

Dedup remote + zstd 22

Total size: 426.619 MiB

For comparison, this is the current rclone compress remote with maximum zstd compression:

Total size: 671.503 MiB

I did wrap a compress remote with my dedup remote as a test but encountered very many file copy errors for some reason so I don't have real results for that test.

I rclone copy'd the data back to a new directory and ran a diff on the original directory against the newly downloaded one and they are identical.

I have admittedly done very little testing and consider it to be highly experimental / proof-of-concept. I'm unsure about data-loss risk with the dedup remote, but I thought it was a cool result and wanted to share.

Kind regards,
Matt

EDIT: Here's a screenshot of the same results as above.

image
image497×467 84.7 KB

EDIT2: Here's a (boring) video of me interacting with the dedup remote a little.
https://asciinema.org/a/tv01yDe8ppJVBXBo

1 Like
2

hi, that sounds like it might have potential, keep working on it.

how would your solution be different from
rclone_serve_restic
or
restic · Using rclone as a restic Backend

1 Like
3

Great question! Kopia also can use rclone as a backend such that your snapshots can sync to any target that rclone can interface with.

The defining difference between restic/borg/kopia/backuppc/duplicati/duplicacy and my dedup remote is snapshotting.
All those other solutions take a snapshot of a dataset and store it remotely.
If I delete some files on the local data set to free up space, the next snapshot does not include those deleted files.

Kopia has a way of mounting snapshots and you can even browse multiple snapshots simultaneously but it's a bit more cumbersome than interacting with rclone.

The benefit to the dedup remote is that I can rclone move files with no "layering" due to the snapshot management features that the other softwares have. Anything I upload is all visible in one place, essentially.

EDIT: I actually asked on the restic forums in 2020, about a feature like this: Flat storage (Snapshotless repository) - Getting Help - restic forum
The best solution was for local fs only, and doesn't help with deduplicating remote storage.

4

ok, good luck with it

1 Like
5

This is a fun idea and I have explored it too, though never even close to implementation.

Do you plan to keep a local cache of chunks that exists? I assume you will not re-upload a chunk that is already there but you still may want to reupload the manifest. Listing the remote must be very hard as you actually have to read each manifest as opposed to just the titles and metadata. Not insurmountable but also not great for many rclone opporations.

1 Like
6

I don't keep a local cache of chunks that exist, they are all stored on the remote. I skip re-uploading an existing chunk by comparing hashes- which isn't too slow thanks to Go's Goroutines.

Under the hood at the remote, the file names and organization structure remains the same, with a small modification. The "files" are actually just manifests that list the chunks they need to build the file. The chunks themselves live in .chunks on the remote, and are organized in a sort of content-addressed-storage scheme so they are quick to locate as well.

I haven't tested on large data sets yet.

1 Like