Linux mounted remote and local folder - easy way to script comparison of folders/files

Help and Support
1

Hi,

I have terrible upload speed. However, I have access to a place where I can upload from occasionally (work)

What I'd like to do to start off with and populate my remote initially (many TB) is to have my home connection uploading - but then grab a listing of files that rclone locally hasn't touched yet, and copy those files to an external drive, which I'll bring to work and upload using copy.

What I'm trying to accomplish this is to create a script that will list files on the remote, and list files on the local storage, and compare the two. then give me an output of what is different, so i can select a specific folder or set of folders to copy to an external, bring to work, rclone copy, and my home setup will not have to use my terrible upload speed to finish copying.

I know theres a really good way of doing this. I don't know what that is. I've tried using rclone lsf or rclone ls piped to a cut command to show me the files there, and then trying ls locally, but i don't know what type of fancy linux foo to get the format of both to match, so that i can try to compare or diff.

any fancy scripters able to help me out?

2

Use rclone lsf -R on the remote and the local to get a list of files. Then use the comm utility

something like this

rclone lsf -R --files-only remote:path | sort > remote-files
rclone lsf -R --files-only /local/path | sort > local-files
comm -23 local-files remote-files > local-only-files

You can then feed local-only-files into rclone with the --files-from local-only-files flag.

Use this once to transfer to external storage

rclone copy --files-from local-files-only /local/path /mnt/external-storage/path

And again to copy those files to the remote

rclone copy --files-from local-files-only /mnt/external-storage/path remote:path

All untested :wink:

2 Likes
3

So if I ever need to make a file-list for use with --files-from then lsf is the compatible format to use?
I will keep that in mind for later, because it's something I have been wondering.

I think this would be highly relevant to add to the doc's description of --files-from because it seems like a pretty obvious question that every potential user will have to ask to use the feature.

4

untested but perfect.
thank you much! i guess i didn't even think of the fact that rclone could parse the local directories and output their files also. kept trying to use local bash commands to get the formatting right was a nightmare. this is much easier.
output looks exactly correct - grabbed a few new 10tb last week during the sale, will test tonight!
thanks again!

1 Like
5

Yes rclone lsf and --files-from go together nicely!

This is mentioned in the docs of rclone lsf but not in the docs of --files-from.

@thestigma Do you want to try an experiment?

This is the file that needs editing: https://github.com/rclone/rclone/blob/master/docs/content/filtering.md

If you login to github then you will get an edit button. I think you should be able to edit the doc and make a pull request all from the browser...

image

1 Like
6

Ooooh. I like experiments. They are like surprises - but with more SCIENCEEE!!

(was it really so simple that I missed the obvious edit button before, or did you actually do anything different aside from navigating to the file?)

request added

EDIT: whoopsie! I guess there was one more button to actually make the pull request. Now it should be in there.
I'm going to assume I don't need to provide every bit of documentation asked for to add 2 lines here - but tell me if I am being too sloppy :stuck_out_tongue:

7

These very simple scripts do something similar. Currently they do a differential move but easy to change to copy.

The scripts create intermediate lists -- what is on remote1 but not on remote 2, and vice versa. These are simple to use in --files-from

8

I've replied on the pull request - thank you!

No you are good!

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.