I am creating a new thread for this as it is a larger separate change from the upload handler.
My project was going to wrap rclone in this functionality but I figure everyone would benefit if it were integrated.
@ncw What do you think about depreciating the current rclone remote:path scheme for the RFC3986 standard?
The standard defines the scheme as scheme ":" hier-part [ "?" query ] [ "#" fragment ]
Pre-configured rclone remotes would be named as the authority with a rclone:
scheme:
rclone://[remote]/path?query
Unconfigured remotes can be specified using the backend name as the scheme:
http://user@host:port/path?query
s3://token@host:port/path?query
ftp://user@host:port/path?query
The query portion contains key=value
pairs that override or specify options for the remote.
dropbox://client_id:client_secret@/path?chunk-size=1000
If the host is omitted it will default to the official domain name of the service. Preconfigured rclone remotes can specify if they allow overrides to avoid security implications.
I don't think it is currently possible to transfer between two different s3 endpoints as the --s3-endpoint
argument is set globally. This specification would allow such a transfer.
Command line compatibility is maintained by having the default scheme being file:
. url paths have to be url encoded to support file and directory names that are not RFC3986 compliant. To avoid requiring command line users from having to url encode their local path parameters, only urls without a scheme are automatically encoded.
rclone copy './foo/b?ar' 'rclone://myremote/'
maps to
rclone copy 'file:./foo/b%3Far' rclone://myremote/
Windows path drive letters will be ambiguous with a scheme, this will require that backend names be longer than a single letter to disambiguate. Will it be necessary to support multi letter drive names? I don't think that occurs often in the wild. The presence of \
in the path along with an unknown scheme might also be a good criteria to disambiguate.
This is how the paths would map:
c:\my\files\file -> file:/c/my/files/file
Many backends already specify a url scheme for their resources. I think a uniform scheme is more important than trying to forward any of the quirks of the backend schemes.
For backwards compatibility this can all be disabled unless a global --urls
argument is passed to rclone.
I expect that the following changes will be required:
- modify all of the /cmd/cmd.go newfs*() functions
- modify /fs/cache/cache.go to key on scheme://host
- modify /fs/fs.go ParseRemote() to accept url.URL