Seeking suggestions/recommendations for how to best backup data from several drives onto a 6 TB HDD

Off Topic
1

Hi there, so I have 4 SSD drives with a maximum total capacity of about 5,22 TB of which I wish to backup 3 of those drives to a newly installed 6 TB WD Blue.

I literally just put it in my system, formatted it to GPT. Now, Ive never really backed up data in any "proper"/systemized way in the past. I wish to setup a system where it automatically backs up my data once every 3rd day or so to my HDD & Im not really sure what software to use for this, hence I figured you guys might have some good advice. :heart:

I would like to setup some form of versioning system if possible, so that I always have at least 2 recovery points that I can go back to even if all my drives except for the ones holding my backups were to go toast (the purpose of several backups is that I figured I might also be able to use the drive to be able to recover files that I might delete by accident and so on, happens every once in a while that I override files by accident :stuck_out_tongue: ).

So Im here wondering what softwares you would recommend, how I can configure a system that can do multiple concurrent backups or setup some kind of versioning system that is not dependent on the data on my other drives, how I can protect my backup drive from falling a victim to ransomware (unwanted encryption, in case I install some no-no software by accident, not that I plan to, better safe than sorry).

As for multiple backups, the ideal would be if I could go back say up to a month or 2 and recover data the way it was from just a few hours/days ago (Im just looking at this from a wishing standpoint here, I do realize 6TB is just 6TB so Im not putting any expectations on this).

Whats most important though is that my backups should be accessible only with the data stored on the HDD entirely independently from my other drives, so if I were to unplug the HDD and put it in a new system, I want to be able to expect everything to be accessible from there as long as I havent physically damaged the drive in any way (might be okay if I need some kind of software to extract the data).

To recap, im open to any kind of advice :heart: Ive never setup any backup software before and dont really know whats good and bad. I hope I dont seem too lazy to just throw out the question like this but theres really a plethora of options out there & lots of different websites recommending different solutions, I could really use a guide.

Im a 3D artist as of recently and have been doing some software engineering since a year back, so Im not very frightened by getting my hands dirty if it can yield a better backup solution at the end. I want a system that I can use for years to come. Open source under public license is always preferred but Im open to anything!

2

I should add that Im on Windows 11 with an i9-12900K and 32 GB of ram (might buy more ram in the near future). I also need to be able to use the device whilst the backups are happening as I use the device for quite a lot of hours during the day and have my bed right next to it (so I wouldnt be able to stand having it go during the night).

Also how would you guys recommend compressing the data, should I compress it in the first place?
image

3

hi,
for drives to be used for backup, do not use perform a quick format.
take the time and let windows do a slow/full format

about compression, i have been using microsoft compression since dos 6.x back in the 1900s.
there are many good reasons to use it and few reasons not to.
but keep in mind, that ntfs compression is no way as good as 7zip or other dedicated compression tools.
many files should not be compressed, such as audio, video and i would assume your 3d artists files.
as nts compression, is done on the folder/file level, very easy to tweak what is compressed and what is not compressed.

rclone is not really a backup program, just a file copier plus some other features.
given your lack of technical experience, not sure rclone is a good fit and i would never use it for local to local.

given you lack of technical experience, you need a simple program with a easy gui and tech support.
i would recommend secondcopy, have used it on 50+ machines for over 15 years.
bulletproof.
it just copies files, including versioning. so you can take that backup drive, plug it into a new computer and the files are there, do not need to install secondcopy on that new computer, can use windows explorer.
and for each file copied, it does a checksum verification, can send summary emails after each backup completes.

4

Well, you're asking on an rclone forum....

But actually, while rclone can do some of this (including backups with --backup-dir), it really sounds like you want a dedicated backup solution

I can suggest some backup tools (restic, duplicacy, kopia, borg, etc) all with pros and cons and idiosyncrasies and gotchas. But based on:

I actually suggest a totally different, non-rclone route: Backblaze Personal. If you are on macOS or Windows, it is $7/month for unlimited cloud backups that hold for 30 days (or for $2/month more, 1 year). It is devoted to backups to the cloud.

(I have ZERO affiliation with them other than as a customer)

5

Thank you both for your anrsers :slight_smile: Lots of cool tools there.

Does any of the softwares you mentioned support versioning of some sort (having several backups of the same data with some form of data deduplication that works between the backups)? Kind of like git (but preferably faster since I will want to backup quite a lot of data)?

From a first look, restic and looks very interesting :heart:I like that it can be configured to communicate with rclone for remote backups, I can see that making use for me in the future.

Happy new Year btw!

6

if the source and dest drives are on the same machine, your proposed solution will not protect you from ransomware.

not a fan of dedupe for backups.
for versioning and to reduce the size of backups, i use veeam backup software.
veeam uses VSS snapshots and have chosen a stupid simple, forever forward incremental scheme.
i use veeam for a a bunch of servers/desktop with over 50TB of data stored on local, and two cloud providers - wasabi, s3 clone for hot storage and aws s3 deep glacier for cold storage at $1.00/TB/Month

7

For a reliable backup solution, consider using Veeam Backup software. It offers straightforward versioning and effective compression. As for compression, using tools like 7zip can really help save space, especially for files that can be compressed well. Before I was using a JPEG Compressor tool which is also a good compression tool.