Discriminate file deletions vs overwrites possible?

Help and Support
1

What is the problem you are having with rclone?

Is it possible to track file deletions with forward forever incremental backup strategy (rclone sync ... --backup-dir ...)?

When running the commands given below, then only one of the rclone runs creates a backup directory (the one right after file deletion). However the state that "the file was not present any more" during run 3 seemingly never gets recorded in a way that would allow to discern it from "file was overwritten before sync".

Or am I missing something and there is a possibility to represent the deletion (something akin to GNU tar's --listed-incremental) - i.e. without simultaneously checking a separate log file?

What is your rclone version (output from rclone version)

rclone v1.57.0

Which cloud storage system are you using? (eg Google Drive)

local storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

md ori bkp
rclone sync ori bkp --backup-dir=incr.`date '+%Y%m%d.%I%M%S'`
echo 1 > ori/1
rclone sync ori bkp --backup-dir=incr.`date '+%Y%m%d.%I%M%S'`
rm ori/1
rclone sync ori bkp --backup-dir=incr.`date '+%Y%m%d.%I%M%S'`
echo 2 > ori/1
rclone sync ori bkp --backup-dir=incr.`date '+%Y%m%d.%I%M%S'`

The rclone config contents with secrets removed.

no config

A log from the command with the -vv flag

General question not pertaining to specific run thus no log here
2

At this point the file system looks like

$ tree .
.
├── bkp
├── bpk
├── incr.20211201.010732
│   └── 1
└── ori

And now it looks like

tree .
.
├── bkp
│   └── 1
├── bpk
├── incr.20211201.010732
│   └── 1
└── ori
    └── 1

I guess the question is, how would you like to see that deletion represented? Rclone could optionally suffix the files with a .deleted or .modified modifier. Or it could write the deleted files into a different directory to the modified files?

Something like that?

3

Ay, precisely like that!

The .deleted vs. .modified suffix would be awesome. Thank you so much for considering!

4

How would you configure that? --backup-dir-extensions deleted,modified or something? Any ideas?

5

Maybe that's already too involved. Honestly I do not see a necessity to make the extensions configurable (may be mistaken however).

What would you think of a Boolean flag like

--backup-dir-tag-state    discern file deletion from modification by adding .deleted/.modified suffix

(Not sure honestly however, which to position last, if an additional --suffix is specified)

6

Maybe making this part of the existing --suffix mechanism might be the right thing to do.

I was thinking of introducing some special characters which could be used in --suffix and --backup-dir to enable formatting of times. Maybe there could be a special one for deleted vs modified...

7

Of course - that would rather be the right place!

In case you are looking for some ideas, you might want to have a look at what the borg guys do w.r.t. key words and formatting: borg create — Borg - Deduplicating Archiver 1.1.17 documentation
(Scroll down to the very bottom to see the examples)

They however just save a string somewhere and don't have to deal with file/directory naming restrictions. Thus they can use some fancy characters - which for rclone surely will have to be sanitized first.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.