Backup folder per file change

What is the problem you are having with rclone?

The issue is that I basically was told to use inotify-tools in combination with rclone. But I have a gut feeling I am wasting my time with inotify-tools and can do this with rclone alone.

What I want the script to do is basically detect a change of a certain filetype stored locally, in this case a .rsf file type (a locally assigned text filetype I am using, don't ask). Once the change occurs, have it to make a new directory in my gdrive with a date-time stamp on it and start over again; detect, make new dir, upload, etc.

Basically what it does now is detect a change, make a new destination directory and from then on just keep copying everything to the destination, overwriting in the process.

Also, I m a n00b unlike you have seen before, I am honest to say that at times I bite off more than I can chew. Also English is not my native language, so if this question has been asked, I apologize for most likely searching with the wrong terms.

What is your rclone version (output from rclone version)

rclone v1.52.2

  • os/arch: linux/amd64
  • go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux System 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

#!/bin/bash

# Variable paths, easier to deal with

src="/home/charlie/.wine/drive_c/users/charlie/data/files/"
gdrive="gdrive:backup_files/test/$(date +%Y%m%d_%H%M%S)_$USER"

inotifywait -r -m -e modify -e delete -e create "$src" \
  | while read path _ file; do
	echo $path$file modified

       rclone --config .config/rclone/rclone.conf copy "$src" "$gdrive" && echo 'file copied'
   done

The rclone config contents with secrets removed.

[gdrive]
type = drive
scope = drive

hello and welcome to the forum,

i, also, am not a linux expert but i know basic scripting.

the issue is that you evaluate the variable gdrive one time, using the current date/time.
and re-use that same evaluation in the loop.

move the variable definition, inside the while loop.
that way, each time the while loop is executed, the variable gdrive will have the current date/time.

1 Like

I'm reading your message and immediately I am thinking of my words where I said I bite of more than I can chew. haha. I totally have no idea what you meant other than guessing I should perhaps

mkdir $(date +%Y%m%d_%H%M%S)_$USER

to create that folder and upload that but put this after the do function?

while read path _ file; do
    echo $path$file modified
    gdrive="gdrive:backup_files/test/$(date +%Y%m%d_%H%M%S)_$USER"
    rclone --config .config/rclone/rclone.conf copy "$src" "$gdrive" && echo 'file copied'
   done
1 Like

Wow... this is highly effective! Damn, it's just one line that had to change positions.

Thank you so much!!! :slight_smile:

1 Like

sure, glad to help. :upside_down_face:

1 Like

Perhaps another question. This time using this script also as a testcase to backup my save games. (I learned to make backups in the Windows era... ... the hard way)

Is there a way for rclone to wait with uploading until all changes have been detected and the game finished writing? With inotify I learned that, and this is how it's supposed to work, a save file is written two or three times after closing the game. Right now I am getting several backups and I can't seem to figure out how to let rclone wait with uploading until all files have completed writing.

Had I not added the sleep 45, I'd have many directories with just a few files in them rather than one directory with all the files. But even so, it still uploads to at least three newly made directories rather than one.

So my request, simply put: Wait until inotify to finish what it's doing, let's say wait two minutes, and then, only then, let rclone backup to gdrive. :slight_smile:

#!/bin/bash

wine "Games/Diablo II/Diablo II.exe" -act5 -vsync
clear

d2="/home/charlie/.wine/drive_c/users/charlie/Saved Games/Diablo II/"
timestamp="$(date +"%T")"

inotifywait -m -e modify -e delete "$d2" \
| while read path _ file; do

	if [[ "$file" =~ .*d2s$ ]]; then
	echo "$timestamp: $file"
	echo ""
	sleep 45
	gdrive="gdrive:ckwd/Diablo II LoD/Save/$(date +%Y%m%d_%H%M)_$USER"
	rclone --config .config/rclone/rclone.conf copy "$d2" "$gdrive"
	echo "$timestamp: All files copied"
	echo ""
	fi
done

seems to me that the game will complete its writing before exiting back to the command line.

1 Like

It doesn't seem to do that.

It writes away when you leave the game (as in, go to menu). It then writes away when closing the game entirely. it has to do with the key, ma0, ma1, etc, files. Even though I'm looking for d2s files only, these files have influences on the save games as they contain map data as well as other data connected to the d2s files.

so you are not exiting the game.
instead, just saving game progress and then continue playing the game?

1 Like

See what I wrote. it does on both occasions.

as an a side, you should move the variable timestamp inside the while loop else you will have the same problem as before.

each time you save a game, does the game create a new set of progress files or does the game overwrite the old set of progress files?

1 Like

lol thanks. Changed.

The game basically updates two files and adds depending on the act you play in.

edit
character.d2s is data is always written to.
character.key is only created upon char creation.
/edit

Basically it's only important to see what happens to d2s in this case.

the character.map, character.ma0, character.ma1, etc are additional files. So if I play for an hour and get my character through act 1 and decide to stop the game somewhere in act 2, it writes a bunch of these files.

Even though these files aren't really important as they are merely map files (overlay map), they do add to the bunch. :slight_smile:


Having said this. I am going to see what happens if I pay attention to the .key file instead of the ,d2s file. In the end the .key file is smaller in size than .d2s files (less writing away) so it could actually be the solution.


edit: That didn't work. lol

For me the question really only is to have rclone wait for a given amount of time, let's say one or two minutes, after the changes are written.

Been doing a lot of research and so far, in order to help others with similar situations, I am posting the solution here. For this particular case, inotify-tools is useless as it created a stream of backups per each file change and with this some file corruptions as well. I choose the safer method for this case.

The script basically checks if the game is a process (running) and only makes a backup if the game ends. This is a check I wanted in there to ensure it only did what I wanted it to do.
side-note: it also checks for internet connectivity to act on this in case the internet doesn't work.

Feel free to use and edit the script to your liking.

Special thanks to jojothehumanmonkey for his help, which got me more in bash scripting and paying attention. (timestamp now works properly as well). It's those tiny things that helps out. :slight_smile:

#!/bin/bash

# Set paths
d2=$(find ~/ -type f -name "*.d2s" ! -path "*/Documents/*" ! -path "*/Trash/*" -printf "%h\n" | sort -u)
dox=$(find ~/ -type d -name "Documents" ! -path "*/Trash/*" ! -path "*/.wine/*")
gdrive="gdrive:ckwd/Diablo II LoD/Save/$(date +%Y%m%d_%H%M%S)_$USER"
clear

# It will now search for the file Diablo II.exe and run it with wine.
find ~/ -name "Diablo II.exe" -print0 | xargs -0 -n 1 wine &
sleep 2

# Loop while looking for Game.exe to run in Wine.
# Diablo II.exe starts Game.exe, so that's what we'll look at.
while true
do
    if [[ -z "$(pgrep -a wine | pgrep -i Game.exe | awk '{print $1}')" ]];
    then
        break
    else
        sleep 5
    fi
done
clear

# Check for internet connection
# Backup accordingly
echo "Checking Internet Status"
wget -q --tries=5 --timeout=10 --spider google.com > /dev/null 2>&1
if [[ $? -eq 0 ]]; then
	echo "Online: Now backing up savegames to gdrive"
	rclone --config ~/.config/rclone/rclone.conf copy "$d2" "$gdrive"
else
	echo "Offline: Now backing up savegames to Documents"
	cp -r "$d2" "$dox/$(date +%Y%m%d_%H%M%S)_$USER"
fi

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.