Using rclone check to print/copy differences

Howto Guides
#1

I have a situation where occasionally my uploads exceed the space available on my cloud storage. In that event, I will end up with (obviously) a destination that does not include all files in my source.

As of now, the best way I have come up with to determine what got uploaded and what didn't is just to mount the destination with rclone mount to ~/remote and do:

ls ~/local > local.txt
ls ~/remote > remote.txt
diff local.txt remote.txt

Then I can go through manually and figure out what still needs to be moved. It's tedious and I hate it.

It would be great if I could have rclone figure out what was left to be uploaded (i.e. at what point in my transfer I ran out of space) and then push only what was left to a different cloud storage bucket.

There is probably a way to just mount the source and destination, and pipe the output of diff to cp, but again, if I run out of space I'm just going to get read/write errors so that is not a clean solution.

What do you guys recommend I do differently to accomplish this (if anything)?

#2

Using

rclone lsf -R --files-only source: | sort > source
rclone lsf -R --files-only dest: | sort > dest
comm -23 source dest > filez

Will do that in a more automated way and filez can be fed into an rclone with --files-from

rclone copy source: new_dest: --files-from filez

You can do that with --compare-dest I think

rclone copy source: empty_dest: --compare-dest full_dest:
Can rclone sync gdrive to multiple onedrive?
How to check different file more than 3dest?
#3

You must use the same remote as the destination of the sync. The compare directory must not overlap the destination directory.

So i think it shouldn't work, but try it.

#4

It did work, although what you quote does imply that it should not. Curious.

Perhaps a better way to do this would have been to just create a union of as many total remotes as I would need, and then run rclone copy, and let rclone sort it out?

#5

not really, because;

Only the last remote is used to write to and delete from, all other remotes are read-only.

maybe instead of union, you could mount your cloud remotes and make a union with mergerfs?

#6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.