Hello, is there a way to only copy/update files that already exist at the destination?
Something similar to rsync's --existing :
--existing skip creating new files on receiver
Use case :
Two computers, A and B, sync to the same Remote.
Computer A makes a bunch of changes to existing files, that computer B needs to pull. Computer A also adds a new, large file, that needs to be synced to Remote for backup purposes, but is not required on computer B.
Remote is Google Drive.
Excluding the large file using --exclude is not an option, as practically, in the real world, it's not just a large file but rather an assortment of files under different directories that can't be actively monitored to be excluded and the file structure can't be changed to, say, have all excluded files in an "excluded" folder, for easy filtering.
A, B and Remote are synced.
A makes a bunch of changes to existing files.
A creates a new file.
A copies all changes to Remote.
Remote holds up to date versions of all files, including the new file.
B wants to update all files, it already has, but doesn't care about the new file.
rclone copy remote B, would copy both updated files and any new files. --update is also implied in copy, no?
If I was rsync-ing A to B directly, --existing would update all changed files that are older at the destination but not copy the new file.
Unclear is from my perception as of course it's clear to you
So you have 3 remotes? A B C ? What is the significance of "Remote?
So this means A copies to C.
Ok
Meaning you want to copy from C->B and ensure B is now in sync?
No update is not implied as normal flow, folks copy from A->B and A updates everything in B and you don't want to skip newer files as that would be a copy.
rclone isn't rsync so it's a bad compare on how things operate as they operate very differently.
Rclone copy/sync is really one way. You have to pick a source of truth and decide what you want to do with changes. If you have a method to exclude them, you can use exclude.
You can skip newer files on the destination or you can overwrite them.
I'd avoid comparing as they operate very different from each other as noted above.
Good post on some applications:
Which. may fit your use case.
There is a feature request out there for 2 way sync as well which might be it but I can't seem to locate the issue offhand.
So you have 3 remotes? A B C ? What is the significance of "Remote?
Remote signifies that it's actually a cloud service and not another computer.
So as per OP, A and B are two computers and Remote is Google Drive.
So this means A copies to C.
If by copies or syncs you are referring to the specific functionality of rclone copy or rclone sync, no.
A could be using either sync or copy, as it's not removing any files.
It doesn't actually matter what method is used, the point is that stuff changed and those changes are now in the Remote.
No update is not implied as normal flow
Indeed, it seems I've baked --update in my script and then though update was the default behavior.
I'd avoid comparing as they operate very different from each other as noted above.
I'm not comparing the two either. I'm using functionality already existing in one, widely-used application as an example of what I want to do.
There is a feature request out there for 2 way sync as well which might be it but I can't seem to locate the issue offhand.
I'm not sure if two-way sync actually reflects on what I need.
Wouldn't two-way sync also download non-locally-existing files, on B?
In Rclone, there are remotes, remotes can local or a cloud service as there's no difference in how things operate.
Copy makes A identical to B with deleting any extra.
Sync makes A completely identical to B with deleting things on B so it's a mirror copy.
I'm not sure exactly what those two in that respect as I don't use either one and haven't completely read through how they work. Perhaps @jwink3101 might have some ideas and can translate your need a bit better than I can as I'm still unfortunately not quite following the decision tree to for the three remotes and could to decide what gets copied or not.
Maybe this will help you follow the decision tree :
I have a work computer and a home computer.
I backup "everything" to Google Drive.
The home computer has a copy of all files including, say, my porn folder. It's private and has a big enough hard drive.
I need some of that stuff on the work computer without, say, the porn folder.
If I make a change to my .bashrc on the home computer I want that mirrored to the work computer, but doing an rclone copy --update would also copy the porn folder.
Yes, I can theoretically exclude the porn folder, using --exclude, but in reality it's not a porn folder, it's not even stuff in a specific folder, say archive. It's stuff that must exist in the same folder as other stuff and is not easily filtered with a pattern, say the extension or a filename prefix.
Having used rsync in the past, to similarly "synchronize" from a remote system I knew about the --existing option, and wondered if there's some way to replicate this behavior in rclone.
I can script around this by running rclone check first and building an exclusion filter based on the output, but was wondering if there was a "native" way of doing it.
Actually, I think @asdffdsa mentioned this in a different post.
I don't use rsync much but --existing is this right?
--existing skip creating new files on receiver
And rclone has:
--ignore-existing Skip all files that exist on destination
I feel that does the same thing re-reading your ask again as my previous statement I think was incorrect.
In rsync, sender is the source and receiver would be the destination which should translate to source destination in rclone so rsync A -> B and rclone copy A: B: with those flags I think would be identical.
You can test with --dry-run and validate as that I think is your answer.
If I'm not mistaken there is currently no such option in rclone. You could however use a two step process:
Check, which files are in source but not in target
Write that difference into a filter list
use this filterlist for the rclone copy
See this example:
md in out
echo 1 > out/1
sleep 2
echo 2 > in/1
echo 3 > in/2
rclone check in out --missing-on-dst skiplist
rclone copy in out --exclude-from skiplist -v
If I understood the OP correctly --ignore-existing would just achieve the opposite of what was intended: OP does want to update files that are in the target, and only these, not the ones missing from there.
Animosity022, not sure why you still think that what I want is to exclude existing files (on the destination), when the actual need is the opposite and, I believe, has been adequately described.
@brdlgwmpft thanks, yes, I've already thought of that but wanted to see if there's a built-in solution, either with a specific flag or some kind of built-in filter.
felix@gemini:~$ rsync --existing /home/felix/test/ /home/felix/test1/ -P -av
sending incremental file list
./
two
10 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/3)
sent 153 bytes received 38 bytes 382.00 bytes/sec
total size is 10 speedup is 0.05
felix@gemini:~$ cat test1/two
update me
felix@gemini:~$ ls test
one two
felix@gemini:~$ ls test1
two
That excludes or skips creating new files on the receiver side.
