Wanted something like tar/rsync "--one-file-system" option for rclone, so we went ahead and implemented it ourselves

Hello @ncw and everyone,

Subject says it all. Me and a friend (hello MaxD13!) spent a nice afternoon/early evening hacking around in rclone to implement an option to avoid rclone crossing filesystem boundaries in local filesystems while copying.

The results of our labor can be found here, in diff format against the latest git master (or at least what the latest git master was, around 15:00 GMT yesterday): http://pastebin.com/2y66Vetf

I’ve done some testing here and it seems to be working. This was our first ever attempt at doing anything with Go (neither of us had so much as looked at any Go code before), so please be gentle with the criticism :wink:

Seriously, we can see there’s quite a few things missing:

  1. Windows doesn’t support .Dev fields (which makes sense as it has no concept of mountpoints either) , therefore this code will almost surely fails spectacularly on Windows (at compile time or at debug time? haven’t tried). We need to test whether the code is being (compiled, executed) under Windows and disable that option appropriately.,

  2. Documentation for the new option (apart from the very basic “usage” line, which I can see also got incorporated into the .md files produced by “rclone gendocs”)

Finally, bringing this all up into github (I think this would need me to somehow “sync” back our modifications up to github and then sending @ncw a “pull request”, but I’m not sure on how to do any of this).

@ncw: in another post (in a github issue, I think) you mentioned you’d be willing to “guide” someone in implementing some feature in rclone; would you be willing to guide us in doing 1) and 2) above, and whatever else you think needs to be done to get this new feature into rclone? We’d love to contribute back to the community, and would be able to do it much faster if we somehow were able to get over the remaining Go and Github hurdles…

EDIT: got a friend to tutor me with git, so this is taken care of. Getting my stuff ready here regarding github, then will send you the pull request.

Cheers,

Durval.

Hey that doesn’t look too bad at all :slight_smile: If I was doing a core feature like this I’d write a test for it which might need some ingenuity.

I’m in the car (not driving :wink: at the moment so I can’t comment much more at the moment!

Assuming you started from a git checkout then you can follow a tutorial like this: https://yangsu.github.io/pull-request-tutorial/ to create a pull request.

It takes a while the first few times. Once you get the PR in then it will be tested with various code quality tools and the unit tests will run on Windows, Mac and Linux which will give you some feedback.

I can give more help but I need to know more about your setup, how you got the code, your git knowledge level etc!

Great feature look forward to integrating it :slight_smile:

Hi Nick,

Thanks for the kind words, and glad you liked the feature.

Right now I’m reading the git PR tutorial you pointed, and getting help from a friend in order to try and send you the PR.

Will let you know if I hit any stumbling blocks.

Cheers,

Durval.

:slight_smile:

Thinking about this further the new flag should probably be defined in local/local.go as it only applies to the local file system .

PS I’d rather have patches than no PR so if it is all too much then just send me the patches and I’ll integrate them :slight_smile:

Hello Nick,

Sorry for the delay in responding; too much happening here in “real life” right now :confounded:

In order to not delay this any longer, I’ve reimplemented with the modification you suggested (ie, declare the flag in local/local.go instead of in fs/config.go, so everything is now restricted to this file).

I can’t attach the patch here as the forum software rejects the .patch extension (it seems to accept only graphic file extensions), so I pasted it here: http://pastebin.com/zGPifxfh

Again, sorry for not having the leisure to go through the PR process right now. Hope the feature is useful and makes rclone into even better software than it already is.

Cheers,

Durval.

I know what you mean about real life!

I’ve put the patch in a branch. I can see I’ll have to work on it a bit to get it to compile under Windows, but I’ll do that - no problems.

I usually take an email address from contributors for the authors file in rclone (the git PR process would supply one normally) - if you email me at nick@craig-wood.com from the address you want in the contributors file I’ll put it in.

Thank you very much for your contribution :slight_smile:

Hi Nick.

Thanks! I feel really bad about leaving this matter in your lap… :-/

Thanks for the credit, Nick! Just emailed you with the address.

Cheers,

Durval.

I’ve merged that now - you can see your changes here:

It will be in the next beta (uploaded 15-30 mins from now)

http://beta.rclone.org/v1.33-98-gb35123b/

1 Like

Thanks for the follow-up, Nick!