Encrypted Google Drive: is there a efficient way to "link-copy" a large directory (ie, equivalent to "cp -rpl")?

Hello everyone (and specially @ncw),

As the subject says. To be more precise, I’m about to update a backup of my laptop on GDrive, and would like to do something similar to what I do when I backup that laptop to my local linux server: I first run a cp -rpl last_backup_dir new_backup_dir (which will “copy” all files as hard links, therefore not using up almost any space and doing very little I/O, and therefore being very fast) on the server, and then I do a rsync [other options] --whole-file" source_dir server:new_backup_dir on the laptop, so any files that rsync finds necessary to update, will be first deleted from new_backup_dir (therefore removing the link) and then copied from source[1].

I’m pretty sure that, by default, rclone will do exactly the same as rsync if presented with such a “hard-linked” remote destination; so my question is, how do I prepare such a destination efficiently (ie, the equivalent of the “cp -rpl” depicted above), specially on an encrypted Google Drive remote.

I was thinking about using GDrive’s Web interface to copy the whole last_backup_dir to a new_backup_dir on the (so far unfounded) rationale that the Web interface will do that in the most efficient way possible (ie, using GDrive’s internal equivalent of a hardlink), but as that remote is encrypted I would have to use first a rclone mkdir remote:new_backup_dir; rclone --crypt-show-mapping lsd remote: | grep new_backup_dir; rclone rmdir remote:new_backup_dir to determine the encrypted name to be used as a destination for the Web interface copy. Not a problem, but I wonder if there’s a less involved, and more amenable to command-line/shell-script automation, way to do that…

Cheers,

Durval.

[1] I’m aware that this is not exactly how rsync --whole-file works (it is actually more involved), but I’m keeping this simpl(ory) explanation here as it is easier to comprehend and sufficient for my purposes.

I don’t know whether drive can “hardlink” files, but it would be in google’s interest to do it if you do a server side copy.

So I’d start by doing

rclone copy gd:last_backup_dir gd:new_backup_dir

And see how that works out.

You could also use rclone’s new--backup-dir flag (in the beta), which probably does what you are trying to achieve, namely server side copies of files that are deleted or overwritten into a separate directory.

docs

Thanks for the info, @ncw.

So I’d start by doing

rclone copy gd:last_backup_dir gd:new_backup_dir

Ah! On the unencrypted remote, right? Great idea! Will do!

You could also use rclone’s new --backup-dir flag (in the beta), which probably does what you are trying to achieve, namely server side copies of files that are deleted or overwritten into a separate directory.

I’m aware of --backup-dir – if I’m understanding things correctly, the problem is that it does not do what I want, namely, give me a directory with a “snapshot” of my files at a certain moment in time. It does give me a “diff” between that moment in time and the next update, so in order to obtain the “snapshot” I want, I would have to copy the current directory and then copy all the “diffs” in between the current one and the point-in-time I want the snapshot to be, to a new directory – a much more involved and error-prone process. And then that directory will potentially have a lot of files that didn’t exist at the desired point-in-time (ie, were created at later updates), so I would have to implement some scheme to identify and delete them.

Am I correct?

Cheers,

Durval.

You are yes.

Though I plan at some point to make an rclone backup command which manages this sort of scheme automatically for you and hides the fact that all that gets stored are diffs…