I have a setup where I regularly copy files from a source to a destination. Sometimes the files in the destionation have been moved in the next copy round, triggering new tranfers of those files. Is there any way to force rclone to copy a file once, and only once, regardless if it have been removed from the destination?
Check out:
The files that have been moved out of the destination will be out of scope, so that wouldn't work. Thank you for the suggestion anyway!
This may be another option for you: --max-age (Docs: https://rclone.org/filtering/#max-age-don-t-transfer-any-file-older-than-this)
What would the logic be to validate a file has been copied if is not anywhere on the remote? How would you let rclone know that?
If destination allow, don't move file, hardlink them.
This way you can sync and have them elsewhere without taking more space.
Hard linking only works when you are on the same disk so you can't hard link from a local disk to a cloud provider.
It’s not for everyone but I often “hack” rclone to do what I want my deciding for myself what I want copied and using the files-from flag. Could something like that wirk
He didn't say what destination was, he might organise then on the same drive/volume, and that may be a local drive also.
Not enough information...
He’s asking on the rclone forum so not a stretch to connect the dots and assume he’s asking about copying to a cloud remote so I’d say there is plenty of info.
Did you consider the remote could be the source ?
But since you know all, I'll let connect the dots yourself...
Alright, I didn't give you enough information. The destination is a union of Google Drive and local disk (DrivePool, doesn't support hard links, over a smb mount). The source is SFTP.
I did but weighed that pretty low and the OP did confirm he actually had a rclone question so we are all good now and based on his response we have 100% that he cannot hard link.
Right so what is your criteria you are doing to determine if it has been copied if the file has been moved on the other side? That’s the key to the puzzle as that would help shape the solution.
It would be enough to skip previously copied filed based on size and file name.
If the file isn’t on the location or an unknown spot, how would you validate that?
Track renames would help if the file name changes or moves location but if you can’t find it m, I’m not sure how to validate if it was copied unless you keep a running filter file of exclusions.
It sounds like you are doing incremental backups?
And basically, you want a deduped incremental backup? Rclone’s commandline functions aren’t really set up to do that efficiently. Rclone mount, pointed to your storage, coupled with an incremental backup utility, however, works splendidly. I use Borg. Restic is also officially supported.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.