I need help creating versioning with crypt to dated folders

perhaps you are double quoting?
first for the sourcepath
set sourcepath="F:\testsource"
and then the rclone.exe command
rclone.exe sync "%sourcepath%"

if i do this echo "%sourcepath%"
i get this ""F:\testsource""

You are right, it's just sloppy on my part.
Wasn't generating any errors, but I've fixed it to be consistent and hopefully more correct now.
(changed script in earlier post)

yeah, most strange, i tested with and without the double quotes and zero errors in rclone??

so are we at 1.4 now?

Rclone too smart for my dumb coding :slight_smile:

we are at 1.30000001 I think with that :stuck_out_tongue:
But still worth pointing out. I need to get better about doing stuff the right way the first time. That just comes with experience I suppose.

one advantage that i have is i have been doing batch file before windows existed, before dos existed.

i have made so many mistakes over the years.
a wise man can learn from the mistakes of others.
a monekyman can only learn from its own mistakes.

there should not be any quotes in the set command, even if there is a space or multiple spaces in the path

try it out,

on the file system using windows explorer rename the source path to F:\t e s t s o u r c e
and change the script to
set sourcepath=F:\t e s t s o u r c e

and re-run the script and it will run just fine.

so how about 1.30000002?

easily enough tested in CMD:

C:\>set lol= a s d f g e w s d

C:\>echo %lol%
 a s d f g e w s d

C:\>set "lol=a b c d e f g"

C:\>echo %lol%
a b c d e f g

Good to know, but I suspect the format I use now (that I have just picked up along the way as a good practice) is probably more about clear readability. They produce the same result.

Without any quote-marks it would be very hard to read any variables that contained whitespaces at the end, or even be sure they were intentional and not just some formatting mistake.

Or are you aware of any problem that might occur from the way I was doing it now?

Didn't know you were a batch veteran... in that case I might have a problem for you that is irking me on another script... (PM)

well,
echo is not a good test and it seems that rclone is too forgiving about double quotation.

but try this my friend and see what happens

set path=f:\t e s t s o u r c e
dir %path%
dir "%path%"

this is what i do, for best readability is to use quotes for the set itself but not for the command.

set source="f:\t e s t s o u r c e"
rclone.exe ls %source%

According to stackoverflow veterans, this is the reason it is recommended as the best practice:

The recommended syntax is to use set "var=value" which does not assign the quotes to the variable value but provides protection against special characters. It also protects the assignment against the accidental trailing spaces.

yes, i agree, that is the best technical option for batch files.
it allows for special characters in both the variable name and the variable value.

i gave up .bat and .cmd batch script a long time ago.
i only use batch file when there is no other choice and have not needed batch complex scripts in many years.

now i use python to script rclone.
as we discussed in this post, using batch files, trying to have a simple date and time in various formats, in a buggy nightmare.
and there is no debugger for batch files.

Yea... I think I'm moving onto python for that reason.
There's nothing you can't do in python that you could do in batch right? (since you know them both)...
It looks really clean and simple. Just worried it might be too abstracted. I do want some control.

Have java and c++ experience previously, but those aren't necessarily the best for simple stuff like maintenance scripts. I suppose you could java it but...

1 Like

i am not an expert in python, i rank 5 out of 10, perhaps 4.

i use python to create on the fly .cmd batch files and then run those .cmd batch files via python.
and python runs on windows and linux.

but tomorrow, i will try to show you how you can use python for scripting.
good night

1 Like

python is simple syntax and is easy to read, does not have {} and ;
i get a context sensitive ide and a debugger.

for anything you want to, there is a library for it.
i mentioned that i have a 300+ line python script.

for example, i use keepass to store passwords, each time i want to run keepass, i run this script, it will backup keepass and then start keepass for me.
same thing with firefox, i run this script, tell it to backup firefox and then start firefox for me.

it reads config info from an .ini file, there is a section for each backup job.
for example,

[keepass]
SourceDir=C:\data\u\keepass
DoVShadow=true
DoFastCopy=true
DoZip=true
DoRcloneSyncZip=true
DoRcloneCheckZip=true
DoRcloneSyncFiles=true
DoPostOp=true
UseArchiveDir=true
SendEmailLogs=true
NetUse=true
PostOpCmd=C:\data\rclone\scripts\rr\postop\start_keepass.cmd

in this example, i want to backup keepass and i can do that in many ways

  1. enable VSS
  2. use fastcopy to copy the local source files to my local backup server.
  3. create a zip file of the local sources file and copy that to my local backup server.
  4. rclone sync that zip file to wasabi
  5. rclone check that zip file on wasabi
  6. rclone sync the source files to wasabi
  7. rclone check the source files on wasabi
  8. use --backup-dir for rclone commands
  9. after the backup has finished, analyze the logs files for errors and then email the following logs to me. fastcopy log, 7zip log, rclone log, vss log.
  10. Netuse - as a paranoid feature, the script creates an on-the-fly net use a: \\backupserver\backupfolder to the backup server, files are copied to that a: drive.
    and after the backup to the backup server net use a: /delete /y . in that way, if the source computer was infected with ransomware, it would not be to access the backup server.
    another paranoid feature, the username and password for net use are encrypted and cannot be seen from task manager.
    and you might think that you could modify the batch file that run rclone, to get the net use username and password, but this is not possible, yet another paranoid feature, as those batch files that run rclone, fastcopy, zip are encrypted.
    i decrypt the .cmd batch file on-the-fly, copy that batch file to the temp folder, execute the decrypted batch file, then the decrypted batch file is zeroed out, and then delete the decrypted batch file. and the username and passwords are written to the registry, then the .cmd batch file reads the passwords from registry, zeros out the registry entries and executes the .cmd file, then deletes the .cmd file
  11. as the last step, run the post-op command, which will launch keepass

the script can be run one-time or in daemon mode.
i use veeam agent to backup windows desktop/laptop computers and those backups can be large, 80GB+ for a full backup,
so on the laptop i have a batch file that starts veeam backup, which backups up to that backup server, but now i need to get the veeam backup files on the cloud, if i run rclone on the client, it can slow down and takes many hours, perhaps overnight and the end-user might have to take their laptop with them off-site.
on the backup server, the python script runs in daemon mode.
so after veeam have finished, the local script sends a signal to the python script running on the backup server, and then the backup server uses rclone to copy to wasabi.


RcloneDestFilesDir=             f'{RcloneDest}:{RcloneDestPrefix}\\{BackupName}\\rclone\\backup\\'
RcloneArchiveFilesDir=          f'{RcloneDest}:{RcloneDestPrefix}\\{BackupName}\\rclone\\archive\\{TheDateTime}\\'

RcloneSyncFilesCmd =            f'{ScriptsDir}\\rclone.exe  sync  "{SourceDir}"   {RcloneDestFilesDir} {RcloneSyncFilesCmdFlags}  {RcloneLogFlags}'
2 Likes

Yeah, it takes a minute or so, for Rclone Browser to actually mount the drive.
I thought I was going to have to mount the drive, to Z: before starting the process.
Oh well, it was working, with minor tweaking before.
Now it should run like a scalded dog...
I'll try the new batch, and see if things go faster.

Thanks a lot, to everyone in here who contributed to this code!

1 Like

@asdffdsa Thanks. I will take my time to absorb that. Let's not derail this thread more though. I will no doubt PM you some Python questions eventually.

There is one thing you can tell me right away though. What is your preferred IDE? It is always a pain to figure out which is best to start learning on. (PM me)

Here's my current issue.
Seems the file lock is not working, at least to me.

I've never had a single failure in thousands of tests of this exact code, so that is strange.
The only way it should not work is if you have 2 or more copies of the script file that you run - because the file-lock is on the script-file itself. So each file on it's own is MUTEXed, rather than the code inside it pr se. The solution to that - if that were the case here - would simply be to have one script that you reference from multiple places.

You can very easily test if this mutex is failing my manually launching the script, and then immediately after trying to launch more copies of it. In windows you will get an error-ding and a message. If you do then it's working as intended. No error-ding when running from CMD but you should get some appropriate error-message there too.


But what seems more likely here is that you are trying to copy from a live OS environment? If so then the problem is simple to explain. It is Windows and it's various processes that tends to use such files (rather than another instance fo the backup script). The NTuser.dat for example is a system file for lots of your user-settings. This script was not designed to deal with live-backup of in-use files such as system files.

If that is something you want I'd suggest that instead you:

  • Schedule WindowsBackup to automatically create a fresh backup-image every X days to some storage location where the backup script is protecting. These files can then be uploaded automatically when the backup is done. This is pretty simple to do.
    OR...
  • If you want a more advanced and flexible solution you have to make use of the shadowcopy function in windows that explicitly is designed to work around this problem of reading in-use files. In fact - I believe this is exactly the underlying system WindowsBackup makes use of to avoid the same issue.
    If you want to use this feature to be able to backup individual files rather than make a system-image then you can refer to asdffdsa's guide on the wiki for that as he figured out how to use this with rclone not too long ago:
    https://github.com/rclone/rclone/wiki/How-to-enable-VSS-for-rclone

@asdffdsa That is the correct link right?

Yeah, I've got other issues with the code as well.
For whatever reason, it can't create the logfile or the logfile directory.

Looks like the log-folder does not exist as far as the script can see ...

That probably means you entered the logfilepath incorrectly formatted.

Can yuo show me what your script has in this setting?

:: Set this path to where you want the logfile to be made.
set "logfilepath=F:\logfiles"

This line may also be correct, so show me this setting too:

set "archivepath=TD5C1:\scripttest\archive"

  1. when testing you can use flag --dry-run, will not make any changes to the dest.
  2. the log file needs to point to local storage, not cloud storage.