"If if a file is being actively written to, rclone will abort the copy and based on some flags, maybe try again or fail out and produce an error." -- that isn't always true, the --no-check-updated option exists specifically to allow rclone to complete the successful upload of a file that's being appended to during the upload. It causes rclone to record the size of the file when it opened it and suppresses the final "did the file size change?" check at the end.
My case is I'm backing up the shards of an Apache CouchDB database. These files are append-only, and so any prefix of these files represents an earlier state of the database. To make a useful backup I need to record the size of each shard file of a given database and only upload that much of each file to my object storage backend. These files can be large, and we may apply an upload bandwidth limit too, so the upload can take a while. Should there be any issue with the upload (beyond rclone's retry limit), I want to be able to try again, and copy the same prefixes of those files. The hosting server might crash too, and I'd like to try again after it comes back. Hence the need to inform rclone about which prefix of the file I want to copy through some means.
Another way to do this, which might help illustrate my intention, would be an LVM snapshot, and then tell rclone to copy the files from the mounted snapshot volume. LVM takes care of hiding that the files are growing, and it's this same notion of a logical file size instead of the actual file size that I'm looking to add as an option. I'd like to achieve this without LVM (or equivalent).
Here's exactly how I'm successfully doing this today, but I suspect extended attributes are a bit too weird for general users, and there are cross-platform issues (that don't affect me personally);
diff --git a/backend/local/local.go b/backend/local/local.go
index 3c00d5389..37f8a6ab5 100644
@@ -1296,6 +1298,14 @@ func (o *Object) setMetadata(info os.FileInfo) {
o.size = int64(len(linkdst))
}
}
rclone_size, err := xattr.Get(o.path, "rclone_size")if err == nil {size, err := strconv.ParseInt(string(rclone_size), 10, 64)if err == nil {o.size = sizeo.fs.opt.NoCheckUpdated = true}}}
This is clearly hackish and not suitable to merge but it shows how I'm achieving my goal. I opened the topic here to see if there's general interest in the feature and, if so, what form that should take in terms of command line arguments, etc.