Split the functionality into the rclone suite

[shadejinx]

As a marvel at how functional rclone has become over the last couple years, I’m also starting to wonder if it’s getting too complicated. The command-line options have ballooned in size making the barrier to entry for simple tasks very high.

Is there any appetite for splitting rclone up into separate functional binaries that can all use a common --rc server for communication? My original thought is to create 5 binaries:

• rclone master sync binary
• rcmount which is just the mount, cache and vfs-cache
• rcserve which is the ftp/sftp/REST server
• rcutil provides the hashing, directory browsing, etc functionality
• rccontrol which provides the background control framework for more advanced users, basically the --rc functionality but for all rcsuite binaries.

I understand that there would have to be duplicate effort to maintain multiple binaries, however I think the long term supportability will be simplified as large monolithic binaries are harder to troubleshoot as they get more complex, systemic bugs become harder to fix and intertwined dependencies get harder to identify as new features are added.

[calisro (Rob)]

On the negative side, someone like me who uses every one of those would need to maintain multiple binaries and consistent versioning between each.

For the most part, you really don’t need to use most of the switches. The defaults are good for 90% of the operations.

[Ajki]

On the negative side, someone like me who uses every one of those would need to maintain multiple binaries and consistent versioning between each.

I agree, if anything maybe just rclonesync that have copy and move for the ones that wanna use that, however functionality should be in master as well.

[ncw (Nick Craig-Wood)]

As a marvel at how functional rclone has become over the last couple years, I’m also starting to wonder if it’s getting too complicated. The command-line options have ballooned in size making the barrier to entry for simple tasks very high.

Yes, I’ve noted the same thing about the command line options! I’ve attempted to address that in recent releases by making the default help text shorter and providing searching for the flags rclone help flags b2 will show you all the flags with b2 in them for instance.

I also split the backend command line flags from the other command line flags which makes it a little more manageable, unfortunately this split isn’t visible in the online docs - I haven’t worked out how to do that!

Is there any appetite for splitting rclone up into separate functional binaries that can all use a common --rc server for communication? My original thought is to create 5 binaries:

rclone is very modular internally. You can commend out any backends you don’t want in backends/all/all.go and any command you don’t want in cmd/all/all.go and recompile for a slimmed down rclone.

So in theory it would be easy enough to make specialized binaries, maybe something like the way caddy will compile a version with just the plugins you want.

So making specialized binaries would be a bit of work, but once done it wouldn’t be a maintenance burden particularly.

I think my preference would be to keep everything in one binary though and help the user with the complexity of the command line options.

One thing I’d like to do is reduce the number of global options rclone has. There are lots of options which don’t apply everywhere

These should only apply to sync

  --delete-after                         When synchronizing, delete files on destination after transferring (default)
  --delete-before                        When synchronizing, delete files on destination before transferring
  --delete-during                        When synchronizing, delete files during transfer
  --delete-excluded                      Delete files on dest excluded from sync

And these should only apply to rclone commands which list things

  --exclude stringArray                  Exclude files matching pattern
  --exclude-from stringArray             Read exclude patterns from file
  --exclude-if-present string            Exclude directories if filename is present
  --fast-list                            Use recursive list if available. Uses more memory but fewer transactions.
  --files-from stringArray               Read list of source-file names from file

-f, --filter stringArray Add a file-filtering rule
–filter-from stringArray Read filtering patterns from a file
–ignore-case Ignore case in filters (case insensitive)
–include stringArray Include files matching pattern
–include-from stringArray Read include patterns from file

So I think grouping the existing global flags into say listing, syncing etc and applying these groups to the relevant commands rather than globally would help a lot, so when you did rclone sync -h you would only get options which were relevant to sync.