As the subject says. To be more precise, I’m about to update a backup of my laptop on GDrive, and would like to do something similar to what I do when I backup that laptop to my local linux server: I first run a cp -rpl last_backup_dir new_backup_dir (which will “copy” all files as hard links, therefore not using up almost any space and doing very little I/O, and therefore being very fast) on the server, and then I do a rsync [other options] --whole-file" source_dir server:new_backup_dir on the laptop, so any files that rsync finds necessary to update, will be first deleted from new_backup_dir (therefore removing the link) and then copied from source[1].
I’m pretty sure that, by default, rclone will do exactly the same as rsync if presented with such a “hard-linked” remote destination; so my question is, how do I prepare such a destination efficiently (ie, the equivalent of the “cp -rpl” depicted above), specially on an encrypted Google Drive remote.
I was thinking about using GDrive’s Web interface to copy the whole last_backup_dir to a new_backup_dir on the (so far unfounded) rationale that the Web interface will do that in the most efficient way possible (ie, using GDrive’s internal equivalent of a hardlink), but as that remote is encrypted I would have to use first a rclone mkdir remote:new_backup_dir; rclone --crypt-show-mapping lsd remote: | grep new_backup_dir; rclone rmdir remote:new_backup_dir to determine the encrypted name to be used as a destination for the Web interface copy. Not a problem, but I wonder if there’s a less involved, and more amenable to command-line/shell-script automation, way to do that…
Cheers,
Durval.
[1] I’m aware that this is not exactly how rsync --whole-file works (it is actually more involved), but I’m keeping this simpl(ory) explanation here as it is easier to comprehend and sufficient for my purposes.
I don’t know whether drive can “hardlink” files, but it would be in google’s interest to do it if you do a server side copy.
So I’d start by doing
rclone copy gd:last_backup_dir gd:new_backup_dir
And see how that works out.
You could also use rclone’s new--backup-dir flag (in the beta), which probably does what you are trying to achieve, namely server side copies of files that are deleted or overwritten into a separate directory.
Ah! On the unencrypted remote, right? Great idea! Will do!
You could also use rclone’s new --backup-dir flag (in the beta), which probably does what you are trying to achieve, namely server side copies of files that are deleted or overwritten into a separate directory.
I’m aware of --backup-dir – if I’m understanding things correctly, the problem is that it does not do what I want, namely, give me a directory with a “snapshot” of my files at a certain moment in time. It does give me a “diff” between that moment in time and the next update, so in order to obtain the “snapshot” I want, I would have to copy the current directory and then copy all the “diffs” in between the current one and the point-in-time I want the snapshot to be, to a new directory – a much more involved and error-prone process. And then that directory will potentially have a lot of files that didn’t exist at the desired point-in-time (ie, were created at later updates), so I would have to implement some scheme to identify and delete them.
Though I plan at some point to make an rclone backup command which manages this sort of scheme automatically for you and hides the fact that all that gets stored are diffs…