RClone and Windows Deduplication

Help and Support
1

Hi all,

My organization has been using rclone with great success. Our main use case is to sync files from a network share up to Google Drive. Some of the network share volumes we would like to eventually sync have been deduplicated using Windows' native deduplication process. My question for the group is: Does anybody know how rclone will deal with deduplicated Windows files? We have had some struggles with deduplicated files and our backup program, so I want to be 100% clear on what behavior I should expect from rclone. Thanks in advance for any assistance.

2

You'd probably have to define what a deduplicated windows file is for your scenario as everyone seems to have a unique definition of what a duplicate is or is not.

A use case / example is always the best.

3

Thanks for your reply. In this case, we are using native Windows deduplication on a volume, kicked off via the PowerShell cmdlet Enable-DedupVolume.

4

So you have a deduplicate volume and your question is:

SomeDir/FileA
AnotherDir/FileA

What happens? The more you can explain what you are asking, the easier it is to answer.

5

Apologies if I was unclear.

Let's say that SomeDir/FileA and AnotherDir/FileA are identical. Windows will "optimize" these files to save space and reduce storage costs. To an end user, you will never realize these files have been deduplicated because Windows handles it seamlessly in the background.

Now, if I were to copy both SomeDir/FileA and AnotherDir/FileA to Google Drive using rclone, I'm curious if the end result will yield two identical files, or if one file will still be "optimized" and thus unreadable by Google Drive.

Thanks again for your response.

6

I'd imagine dedupe happens at a lower level / block level rather than at a file level.

If you had:

and

and they are 'one' file deduped on the OS, it would appear two unique files, get copied twice.

7

hi, I looked into that years ago, decided did not fit my use-case.

windows will dedupe on block level, same as VSS snapshots, same as REFS checksumming.
rclone or any software, should perform the same, dedupe or not dedupe.

https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/understand
"Users and applications that access data on an optimized volume are completely unaware that the files they are accessing have been deduplicated."

which program?

https://www.veeam.com/blog/data-deduplication-windows-server-veeam.html
"The system preserves unique blocks only, moving them into a chunk store and leaving references (reparse points) for those blocks that are used more than once"

8

Thanks for your response! The resources you linked, along with your comment, are extremely helpful.

We use Acronis, which unfortunately does not handle it as well as Veeam does.

9

vbar(veeam backup and replication) offers a free version for commercial use.
and vagent, the client version, is also free.

these days, i only use acronis for bare-metal recovery to dissimilar hardware, not for backups.
vagent can do that, but not as well as acronis.

veeam+rclone is a complete solution.
i use rclone to copy recent backups to wasabi and move older backups to aws deep glacier.
the nice thing is that with veeam, each incremental/full backup run is saved to a singe file with a unique timestamp filename. makes it easy to use with rclone.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.