I need help creating versioning with crypt to dated folders

Howto Guides
1

Greetings all,

I was wondering if there was some command line magic that I could use to set up a windows machine to automatically mount an encrypted drive, once per week, create a dated folder, within the encrypted folder, to perform a full copy of a shared directory hosted on a synology drive, then upon successful completion, dismounting the drive, to keep the drive "offline" in case ransomware hits the computer.

For instance, Synology is setup as S: on a windows machine / server. I'd like to copy the "business folder" from the Synology, to the encrypted rclone drive, that would automatically mount as drive Z:, once per week, on Saturday at say 9pm, and do a full copy of the data to a new dated, data directory like "Business_12-01-2019, incrementing per week, so I could retain at least 4 versions of a backup, per month.

The drive is a google business file sync drive with unlimited storage. I know this is not the fastest way to backup, but full weekly backups on a drive with unlimited storage seems like a no brainer to me. If there's no space limit, and the time to do the backup, no problem. If there's an option to do incremental copy's, I'll go with that as well.

I was thinking of using the Windows Scheduler to invoke the rclone on a weekly basis, that should be easy... getting rclone to automatically mount the Z: drive, and then create a new, dated directory, do the copy, and then once the job finishes, to dismount the encrypted drive would be where I'll need an assist.

This will not be the only backup I have, but I prefer redundancy, and offsite, offline drives, are much harder for malware to run amuck on, thus the automated mounting and dismount, and the saving a full copy, to a new directory, each time. Worse case scenario, I'd lost a week of data.

2

You don't actually need to mount the drive to do this. We could do the entire operation in script (which is the most efficient anyway), so then it would be absurdly unlikely that ransomware could touch the data. Or at least the ransomware would have to written for rclone, which I don't think is a thing I have ever heard of :stuck_out_tongue:

If you want full backup that's easy, but file-level incremental backup backup is possible too - where you only re-copy whatever changed since last time. It kind of seems wasteful to have a drive full of exactly identical duplicates but that's just me looking for efficiency out of an unlimited system heh...

Or a combo of both I suppose. Daily or even hourly incremental (with little work needed on each run) plus a big separate hardcopy for every week or month (and that could be run server-side even). We will need to discuss your needs in detail and what you feel you want out of it.

What you are asking for it not very hard. I should be able to help out - but tomorrow I think, because it is quite late. Please feel free to leave me a PM tomorrow to remind me.

But you need to know the magic word.... do you know it? ... :crazy_face:

3

Thank you for your kind and speedy reply.

If I don't need to mount the drive, then so much the better. The less aware the OS is of the drive, the less likely it would be for it to get a nasty malware infection.

Since this is possible, let's go with the most efficient, incremental backup possible, that will run on a schedule, every day, at say 9 pm. I'm assuming something like rclone -copy with some command arguments and some work with the windows scheduler would probably do this, but I'm not a rclone guru. I just got into using it a few weeks ago, thanks to the Archivist over at the eye, for my personal seedbox crap to "shared drives" I had purchased via .

I have a local, enterprise 8TB drive installed in my ESXI machine, as a local backup drive, for the Synology. The Synology is 8TB, total storage after the four drives have been formatted and raided. Backing up a raided Synology, to a single 8tb drive might seem crazy, but I was looking for redundancy and backup speed.

The initial idea was to use Iperius Backup or Veeam to do the local backup or if I wanted to cheap out, just using Windows Backup. Then I discovered Google File Stream drive, because of existing email and web hosting via Google, aka they proudly displayed a message when I logged in to my account that said, congratulations, you have unlimited storage... and I said, let's leverage this into the backup plan, because redundancy and offsite backup is awesome and malware doesn't give a hoot if the drive is a network connected drive, or a local drive... it will attack anything it can get to, so you are forced to pay a ransom. That was the reason I was initially hesitant about leaving the rclone drive mounted more than the time of the backup... but since it doesn't have to be mounted by the OS, so much the better, in this case.

It does make me kind of sad that no windows based backup program can actually see or work with the rclone mounted drive as "local storage" to actually do a backup. Aka, the drive mounted as Z: is not usable within any windows backup program I have tested thus far. The big boys want you to buy Google storage, instead of using the unlimited google file stream drive. I don't really want to have to pay, for another service, that basically does the same thing as Google File Stream and the price per gb is outrageous compared to the low, low price of 10 bucks a month for my "business" account with Google.

I was always told the magic word was PLEASE, so if you'd please lend me a hand in this endeavor, I would be honored.

If please isn't the magic word, then I'll have to guess and go with Abracadabra.

4

welcome to the forum,

@thestigma is correct, no need to mount.

i use:
veeam for to backup workstations, servers and virtual machines.
i perform full image backups for disater recovery and file backups as well.
rclone to sync veeam data to the cloud.
python to script rclone.

another way to do this is with daily incremental backups as i do,
i do not use rclone copy but instead i use rclone sync and the flag --backup-dir.
the most important thing is that flag which protects against ransomware and keeps older versions of changed files.

https://rclone.org/docs/#backup-dir-dir
" When using sync , copy or move any files which would have been overwritten or deleted are moved in their original hierarchy into this directory."

i do forever forward incremental backups each day and once a month full backups.
here is simplified snippet from my rclone command that i run each day.
each time it runs, rclone syncs the source to the dest and for any old files in the dest, rclone moves it to the archive folder.
each time rclone is run, i add the current date and time to the archive folder name, which gives me forever forward incremental backups.

rclone sync v:\en07.veaam\ wasabi:en07.veaam\backup\ --backup-dir=wasabi:en07.veaam\archive\20191102.130436\

i know that you are concerned about ransomware.
on the windows server, does the server needs access to the s: all the time, or just for backups?
if just for backups, then you can as i do,
before the backup starts i run net use s: \server\share
after the backup end, i run net use s: /delete /y
in that way, the server is only accessing the s: for a few minutes.

veeam backup files can be encrypted by veeam, no need to double encrypt

5

That is totally fixable.
A lot of backup solutions actually work out the box with a correct rclone setup. The rest require a little bit of workarounds.
I've actually done a fair amount of work into this exact thing recently as Canadian gentleman invited me into a project to create an rclone-based backup storage for the business market (Google Cloud app store). So this can be done robustly even if some certain programs like windows backup need a little bit of custom scripting.

I will take a look at making a v1 script for you later tonight. I recommend we use a solution similar to what asdffdsa as that is very flexible, efficient, and also creates a way to retrieve old files that got deleted along the way somewhere (either by accident or intentionally). if you ever need to go back and fetch something that is no longer in the up-to-date backup you can just go back to the archive and lok at the day you think you deleted it and restore it from there.

asdffdsa is our resident data-security paranoiac, so you know if it's good enough for him it's good enough :stuck_out_tongue:

6

here is the pseudo code i use
i use python to calculate the date and time but you can script that in batch file.
i use date and time, not just date. as you develop, test and debug the script. you want to include the timestamp.

set BackupName=en07.veaam
set Source=b:\%BackupName%
set DateTime=20191102.130436

sync %Source% wasabi:%BackupName%\backup\ --backup-dir=wasabi:%BackupName%\%archive%\%DateTime%\ --log-file=C:\data\rclone\logs\%BackupName%\%DateTime%\rclone.log

if you want to have the log emailed to you, you can use swithmail.

7

Thanks, I can reference that to make a similar system. This looks like I had imaged it would.
Time and date is pretty easy in batch as they are just system variables you can use (and maybe truncate a bit as needed - since we probably don't need to include more than down to the minute here).

@Charles_Hayes I will be assuming, unless you specify otherwise - that you will want me to make something for you (expect within the day/night). If you decide to use asdffdsa's template to make something yourself instead then I will wait. I don't mind helping, but I do not want waste time on doing redundant work.

8

well, the paranoid approach is that there is a chance that when testing and debugging the script there is a chance that you will run it twice within 60 seconds.

9

I always enact a file-lock on the script to avoid that - but thanks for reminding me as that is an important thing to have :slight_smile:

10

how do you do the file-lock?

11

Windows or Linux?

12

windows
.......

13

Notes. This uses OS-level exclusive access to the file.
This is a very very common strategy on Linux even for professionally coded software (like Linux itself), but it works equally well on Windows.

The benefit of this it will be active as long as the script runs
but it will unlock as soon as the script stops for ANY reason - including a crash or process-kill. Thus it is failsafe. Any other process hitting an active writelock will just fail (and retry later if they are scheduled to do so). If you want that script to "wait in line" then you can code a simple loop for it, but that's usually not necessary unless you are looking for some kind of efficiency optimimization.

Also note - the file-lock here is on the script itself (denoted by ~f0 , which means "the full pathname of this script). It does not have to be this (but it is often convenient and what you need). If you need multiple different scripts to never run at the same time then use a separate file somewhere that will then be the common lock for all the scripts in that group. Any file in any permanent location will do. I don't think it needs to exist beforehand... but only 90% sure on that.

Ask if there is anything you want me to elaborate on :slight_smile:

::Before we start, get a file-lock on this script to prevent concurrent instances
call :getLock
exit /b

:getLock
:: The CALL will fail if another process already has a write lock on the script
call :main 9>>"%~f0"
exit /b

:main
:: Body of script goes here. Only one process can ever get here
14

nice script, thanks,

15

One more note:

The reason there are 2 functions here rather than one (the first one seems pretty redundant right?) is a workaround for a batch-spesific quirk that may cause it to fail in certain circumstances if you omit it. (ie. the Linux version can be even simpler). But this double-call method makes it safe in batch in all circumstances. If you replicate this in python (which should work just fine) I don't think you will need it.

16

well, since i use vss for the source file and date.time including seconds, i do not need the file lock.

17

EDIT: Script updated to 1.3 to correct the 4 issues mentioned by asdfdsa later down. Now 1.3 for more fixes and improvements...

Well, since you knew the magic word (BOTH of them in fact!!) then I must oblige...
Here you go, all done! (now where do I send the bill? :stuck_out_tongue_winking_eye: )

Script has been tested, but of course I recommend you also test it thoroughly yourself and see that everything works as you expect before putting it into a live environment. Also check that you feel the formatting of time/date is to your liking. Note that " : " (normally in time formats) is an illegal character on google drive and most filesystems so I had to replace that. I opted for human readability over easy computer parsing.

Description:
Makes a "mirror backup" on the cloud of your the folder you designate as the the source. This includes deletion of files, ie. if you delete something locally then it gets removed from the "current backup" also.
However, these files are not lost. Instead of being deleted or overwritten they are moved to the "archive" and timestamped. Thus you have a full "back in time" archive for all file revisions ever made in case you should need to recover anything - either due to malware, errors, mistakes or dead harddrives.

The script does not clean up old archive data automatically, so you may want to remove the oldest stuff manually once a year or something (but it is not really required when your storage is unlimited, up to you...).

The system is efficient and does not require any duplicate data to be stored.
New entries in the archive only happen if anything actually changed. Otherwise nothing needs to be done and rclone will just exit after checking everything is up to date.

Usage:

  • Change the settings so they are correct for you (ask if needed but I commented it pretty robustly I think)
  • Use task scheduler to schedule the script to run as often as you wish. For example once every hour, but that is totally up to you. There is little to no waste of computer resources to run it often if you wish. Ask if you need help with that.

sidenote1:
If the files you want to backup&archive include files with weird or strict permissions that your own user can not normally access then you may want to run the script as SYSTEM account instead of yours. However, if you do then we may need to check that rclone uses the right config. I don't expect this will be needed, but I mention it for the sake of completeness.

sidenote2:
Google drive has a maximum upload limit of 750GB/day. That means that you may initially run into that limit and probably need several days to get up to speed. From there it should not be a problem as you most likely won't add or change 750GB pr day. Hitting the upload limit is the only circumstance under which rclone will fail do perform the backup&archive (which is not much we can do about really...)
It won't require any restart or interaction on your part though. It will keep running as scheduled even if that happened.

@asdffdsa Feel free to give this a once-over to see if I missed anything important

archivesync.bat

:: Archivesync v1.3 by Stigma, credit to asdfdsa(Jojo) for pseudocode assistance, debugging and emotional support ;)
@echo off

::Before we start, get a file-lock on this script to prevent concurrent instances
call :getLock
exit /b

:getLock
:: The CALL will fail if another process already has a write lock on the script
call :main 9>>"%~f0"
exit /b

:main
:: Body of script goes here. Only one process can ever get here

:: --------------SETTINGS START ----------------

:: set this to the folder where rclone.exe is located on your system
set "rclonepath=C:\rclone"

:: Set this to the folder (or driveletter) you want to protect with backup and archive
set "sourcepath=F:\testsource"

:: Set this the the folder (usually on a remote) you want to save the backup and archive to (does not necessarily have to be an rclone remote)
set "destpath=TD5C1:\scripttest\CurrentBackup"

:: Set this to the folder (usually on a remote) which will contain old "deleted" or "overwritten" revisions of files. I suggest keeping it next to your backup folder but it could be put anywhere.
set "archivepath=TD5C1:\scripttest\archive"

:: Set this path to where you want the logfile to be made.
set "logfilepath=F:\logfiles"

:: Set the detail of logging - from least verbose to most: ERROR or NOTICE or INFO or DEBUG (default, NOTICE, is usually sufficient)
:: see documentaion for more info : https://rclone.org/docs/#log-level-level
set "loglevel=INFO"

:: Set any other non-essential flags you want rclone to use. Can leave as empty set "flags=" if you want none
set "flags=--fast-list --progress --drive-chunk-size 64M"

::------------------SETTINGS END------------------
::----------------MAIN SCRIPT START --------------

::Various timestamp formatting in separate function (you can change timestamp there if you want).
call :FORMATTIME

::Make the logfile directory if it doesn't already exist
if not exist "%logfilepath%\" mkdir "%logfilepath%"

echo Archivesync is starting, stand by ...
echo rclone is gathering listing from the destination. This may take a minute if there is a lot of data.
echo:

:: Now let us sync. This makes a mirror of sourcepath to destpath (including removing files if required), and any files that get "overwritten" or "deleted" as a
:: result from destpath, will be moved into archive and and timestamped instead - effectively creating a full archive of all revisions of files you have ever had.
%rclonepath%\rclone sync "%sourcepath%" "%destpath%" %flags% --backup-dir="%archivepath%\%date%" --log-file="%logfilepath%\%date%.log" --log-level=%loglevel%
echo:

::If exit code of above command was anything but normal, display an error and pause
if not %ERRORLEVEL% equ 0 (
	echo rclone reported an error during the sync. Check that settings are correct. Check rclone logfile for spesific info about the error.
	exit /b 1
) else (
	echo Sync completed sucessfully!
	exit /b 0
)

::----------------MAIN SCRIPT END -----------------
::--------------HELPER FUNCTONS START--------------
:FORMATTIME

for /f "usebackq skip=1 tokens=1-6" %%g in (`wmic Path Win32_LocalTime Get Day^,Hour^,Minute^,Month^,Second^,Year ^| findstr /r /v "^$"`) do (
    set day=00%%g
    set hours=00%%h
    set minutes=00%%i
    set month=00%%j
    set seconds=00%%k
    set year=%%l
	)
set month=%month:~-2%
set day=%day:~-2%
set hh=%hours:~-2%
set mm=%minutes:~-2%
set ss=%seconds:~-2%

:: This can be easily modified to your liking if you prefer another timestamp format (for archive and logs) - credit to asdfdsa(Jojo)
set date=%year%.%month%.%day%_%hh%.%mm%
	
exit /b 0
::----------------HELPER FUNCTONS END--------------
1 Like
18

I want to thank each and every one of you for helping me with this.
You guys and gals / they / them's or whatever, are awesome people.
Now I've got something to work with, or at least a framework to build from.

Hopefully, with my limited knowledge of scripting, I'll learn something in the process without sporting the "Mr. Clean" look from pulling my hair out.

Also, if you get the commercial version of your backup going, let me know, and I'll purchase it. Hopefully, you'll sell me a lifetime license. :smiley:

19

Currently, the data I am planning on backing up is only 75 gb. So, I won't have to worry about the google backup limit.

I do have another PC I want to backup, that has about 4 TB of data on it, mainly cad files and architectural drawings, revit files, and other boring stuff that sucks up gobs of storage space. Is there a way to rate limit the rclone process in this case? I think the upload is like 10 to 20 mbps. I don't think I can push up more than about 300 gigs a day, if memory serves me correctly, so it may be a non issue.

My Plex server... dear god. It will be easier to just re-download and sync from my seedbox and then using my vps to push up to my rclone drive, than attempting to move 20TB of "files" from my connection to a crypted rclone drive. I'm not really THAT worried about files I can simply download again. I'm more worried about work files I actually spent time creating.

20

hah! thanks for that. Maybe you could betatest for us or something? Have a chat with @arnoldvilleneuve about that if interested. Testers are kind of something we will need very soon. We already have a working prototype going, but we have lots more planned in terms of alternate deployment options too.

As for scripting, feel free to ask - or just request a change if there's something you need. I have plenty of hair left so I can save you some of yours :wink:

You can run the same script there. Just make a second location for the second PC in the settings.
You can limit rclone's speed with --bwlimit 10M (megabytes/sec, not mbit). Setting to approx 8.5M will make it never hit a limit with 24/ uploading going on. Unfortunately we still lack a truly elegant way of detecting and dealing with hitting the quota limit. rclone will just keep trying and kind of stall, not that it matters since it won't be able to upload more anyway. it will work again as soon as it resets (sometime around midnight, but it varies from server to server).

If you want to mass-transfer easily and your local bandwidth is not quite suitable for that much data - I'd recommend using a Google Cloud microinstance VM (Linux based) to do the job for you. Importing data to google is free, and the free-use limits are more than enough for the task, so you can actually do it free (or spend a couple of dollars on a windows server for a week or two if you feel more comfortable in that environment). It's quite cheap really... even if you go above free-use limits. Besides, you get 300USD in free cedits to spend the first yet so yea... :stuck_out_tongue_winking_eye:

On a windows box you could literally just run the same script there and leave it running for 27 days - and those 20TB will be done on it's own... but of course it's not exactly hard to a much simpler Linux script for a simple mass-transfer if needed.