I have a situation where occasionally my uploads exceed the space available on my cloud storage. In that event, I will end up with (obviously) a destination that does not include all files in my source.
As of now, the best way I have come up with to determine what got uploaded and what didn't is just to mount the destination with rclone mount to ~/remote and do:
ls ~/local > local.txt
ls ~/remote > remote.txt
diff local.txt remote.txt
Then I can go through manually and figure out what still needs to be moved. It's tedious and I hate it.
It would be great if I could have rclone figure out what was left to be uploaded (i.e. at what point in my transfer I ran out of space) and then push only what was left to a different cloud storage bucket.
There is probably a way to just mount the source and destination, and pipe the output of diff to cp, but again, if I run out of space I'm just going to get read/write errors so that is not a clean solution.
What do you guys recommend I do differently to accomplish this (if anything)?
rclone lsf -R --files-only source: | sort > source
rclone lsf -R --files-only dest: | sort > dest
comm -23 source dest > filez
Will do that in a more automated way and
filez can be fed into an rclone with
rclone copy source: new_dest: --files-from filez
You can do that with
--compare-dest I think
rclone copy source: empty_dest: --compare-dest full_dest:
You must use the same remote as the destination of the sync. The compare directory must not overlap the destination directory.
So i think it shouldn't work, but try it.
It did work, although what you quote does imply that it should not. Curious.
Perhaps a better way to do this would have been to just create a union of as many total remotes as I would need, and then run rclone copy, and let rclone sort it out?
not really, because;
Only the last remote is used to write to and delete from, all other remotes are read-only.
maybe instead of union, you could mount your cloud remotes and make a union with mergerfs?
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.