Make Docs for mortals not only rclone gurus

Feature
1

It is continuation of my comment and ncw reply from Rclone v1.63.0 release - #7 by kapitainsky, more appropriate to move it here:

Yes I understand automation part - otherwise it can easily become not manageable. Rclone today has 700+ flags and I can see on the forum that people struggle to find things which are pretty much obvious like -u, --update for rclone copy. Well obvious for somebody who knows something about rclone. But for new user? impossible. Almost. It requires careful reading of 16k words single web page- Documentation

2

This is not criticism but rather observation. I do understand massive legacy of years of development and adding new features.

I do not have easy solution for this problem neither. But hopefully some clever ideas are out there.

It would make life so much easier - not just for newbies - if things are structured in more human way. What I mean by this is that e.g. if I want to use rclone sync I can easily see all flags applicable to syncing.

3

You have a very valid point, but I doubt there is an easy solution.

Here is however a few tricks I find useful, and sometimes try to nudge in my forum replies.

I have found it very helpful to skim the list of non-backend flags to get an impression of the possibilities, it is "just" one hundred flags if ignoring all the --rc options.

With that basic understanding I then use simple text search on the webpage to find the exact flag when needed, e.g. "rename".

The list of backend flags is huge indeed, but very helpful if you want to quickly check the possible flags for a specific backend (search for e.g. "onedrive") or find all the backends supporting a given flag (search for e.g. "pacer").

I just wish the list of flags had links to the corresponding section in the general docs or the relevant backend docs.

1 Like
4

Thank you. I do not have good answer neither yet. But I think that without trying to change it we will always have the same "old good" docs:)

1 Like
5

and your tips how to navigate all chaos are valid. But I more think about new users - rclone is cool - install - config - sync command. And what? Sync help page is just bla bla - you have one flag mentioned there. With small link below - global flags:)

6

Perhaps having something like https://tldr.inbrowser.app/ for the various commands with multiple usage examples and different flags may be helpful?

1 Like
7

Another approach may be to have an LLM-based chatbot like GitHub - hwchase17/chat-langchain which uses the docs to answer user queries.

1 Like
8

What we could do is add to the metadata for each command which (global) flags are important to that command.

The doc generator could then include these flags in the help page for an individual command.

That would then bring to the front some of the more important flags and the user could then peruse the full list if desired.

Another idea

Ideally we'd cut down the number of global (non backend) flags and try to group them and attach them to individual commands.

Groups might be filter, copy, sync say.

rclone ls would only need the filter group, but rclone copy would filter and copy and rclone sync would need them all.

This would then put the flags directly on the command page.

Whether we can do this without breaking backwards compatibility, I'm not sure.

The idea of using an LLM trained on the rclone docs is one I've had before. It could be trained on high quality forum posts too.

1 Like
9

I like this idea. Simple start but the key here is that by creating such metadata we will get structured information which later can be used in more creative ways. Also that it does not require manual docs editing.

10

IMHO I think having a way to post various Example command lines would be of great help.

I have been using rclone almost as long as it came out, and I literally use it everyday and not via scripts.

Even with my usage I still find trying to figure out the proper command sequence can get a bit confusing.

So maybe an easy way people can share cmd line examples base on the type of cli.

Maybe just different Threads based on cmd structures where people can write short description of what they are doing and the cmd that made it happen

Rclone is so advanced and so many more options etc then things like wget I think this could certainly help.

It also could mean fewer Help and Support tickets since if people search google using

search phrase site:rclone.org

should allow them to find it faster.

I am "assuming" people are wise enough to use google dorks for finding solutions though.

11

What we could do is add to the metadata for each command which (global) flags are important to that command.
The doc generator could then include these flags in the help page for an individual command.

That would then bring to the front some of the more important flags and the user could then peruse the full list if desired.

Just a thought, what about something like a Wiki Page for each command where even users could post what they did or their tricks etc?

Not saying has to wiki page itself, but something like it

12

Danswer may be worth a try. It still requires an OpenAI API Key for now but I believe they are planning to add support for local models soon.

13

Can we start with this for the beginning? I am sure that at some stage some LLM based solution can help. But more structured data we have then it is only easier to add more clever ways to use it. Without intelligent input no AI can help us.

As long as there is framework to start adding metadata I am happy to contribute myself into doing it. It even does not have to be used from day one.

It does not have to be perfect from day 1. We can rectify and improve it organically.

And I know most people prefer to spend time on new features:) not on boring docs. This is why step by step approach IMHO is the best way - let's start building metadata.

14

OK here is my proposal.

I invent some way of tagging global flags (not backend flags) to be part of a small number of groups. This tagging will be in the source code and you will need to supply it when defining a flag so we won't forget it.

Something like this to put the flag into the Sync group. You won't be able to make up groups there, they will be predefined.

flags.Int64VarP(flagSet, &ci.MaxDelete, "max-delete", "", -1, "When synchronizing, limit the number of deletes", "Sync")

We then decide which of these groups we want to show in in each rclone command by adding a bit of metadata, something like:

        Annotations: map[string]string{
                "versionIntroduced": "v1.52",
                "flagsGroups": "Copy,Sync,List",
        },

(you can see there the metadata which makes the first version available tags in the docs)

We can also break up the flags page into these groups to make it more digestible.

Open questions - do flags need to be part of more than one group?

Here is my initial classification - comments welcome!

Groups of flags

Copy

Flags used for sync, copy, move, copyto, moveto, rcat. Basically anything which can copy a file using operations.Copy.

      --backup-dir string                           Make backups into hierarchy based in DIR
      --check-first                                 Do all the checks before starting transfers
  -c, --checksum                                    Skip based on checksum (if available) & size, not mod-time & size
      --compare-dest stringArray                    Include additional comma separated server-side paths during comparison
      --copy-dest stringArray                       Implies --compare-dest but also copies files from paths into destination
      --cutoff-mode string                          Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
      --ignore-case-sync                            Ignore case when synchronizing
      --ignore-checksum                             Skip post copy check of checksums
      --ignore-existing                             Skip all files that exist on destination
      --ignore-size                                 Ignore size when skipping use mod-time or checksum
  -I, --ignore-times                                Don't skip files that match size and time - transfer all files
      --immutable                                   Do not modify files, fail if existing files have been modified
      --inplace                                     Download directly to destination file instead of atomic download to temp/rename
      --max-backlog int                             Maximum number of objects in sync or check backlog (default 10000)
      --max-delete int                              When synchronizing, limit the number of deletes (default -1)
      --max-delete-size SizeSuffix                  When synchronizing, limit the total size of deletes (default off)
      --max-duration Duration                       Maximum duration rclone will transfer data for (default 0s)
  -M, --metadata                                    If set, preserve metadata when copying objects
      --modify-window Duration                      Max time diff to be considered the same (default 1ns)
      --multi-thread-cutoff SizeSuffix              Use multi-thread downloads for files above this size (default 250Mi)
      --multi-thread-streams int                    Max number of streams to use for multi-thread downloads (default 4)
      --multi-thread-write-buffer-size SizeSuffix   In memory buffer size for writing when in multi-thread mode (default 128Ki)
      --no-check-dest                               Don't check the destination, copy regardless
      --no-traverse                                 Don't traverse destination file system on copy
      --no-unicode-normalization                    Don't normalize unicode characters in filenames
      --no-update-modtime                           Don't update destination mod-time if files identical
      --order-by string                             Instructions on how to order the transfers, e.g. 'size,descending'
      --refresh-times                               Refresh the modtime of remote files
      --server-side-across-configs                  Allow server-side operations (e.g. copy) to work across different configs
      --size-only                                   Skip based on size only, not mod-time or checksum
      --streaming-upload-cutoff SizeSuffix          Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
      --suffix string                               Suffix to add to changed files
      --suffix-keep-extension                       Preserve the extension when using --suffix
  -u, --update                                      Skip files that are newer on the destination

Sync only

Flags just used for rclone sync

      --delete-after                                When synchronizing, delete files on destination after transferring (default)
      --delete-before                               When synchronizing, delete files on destination before transferring
      --delete-during                               When synchronizing, delete files during transfer
      --delete-excluded                             Delete files on dest excluded from sync
      --ignore-errors                               Delete even if there are I/O errors
      --track-renames                               When synchronizing, track file renames and do a server-side move if possible
      --track-renames-strategy string               Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash")

Networking

General networking and HTTP stuff

      --bind string                                 Local address to bind to for outgoing connections, IPv4, IPv6 or name
      --bwlimit BwTimetable                         Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
      --bwlimit-file BwTimetable                    Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
      --ca-cert stringArray                         CA certificate used to verify servers
      --client-cert string                          Client SSL certificate (PEM) for mutual TLS auth
      --client-key string                           Client SSL private key (PEM) for mutual TLS auth
      --contimeout Duration                         Connect timeout (default 1m0s)
      --disable-http-keep-alives                    Disable HTTP keep-alives and use each connection once.
      --disable-http2                               Disable HTTP/2 in the global transport
      --dscp string                                 Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21
      --expect-continue-timeout Duration            Timeout when using expect / 100-continue in HTTP (default 1s)
      --header stringArray                          Set HTTP header for all transactions
      --header-download stringArray                 Set HTTP header for download transactions
      --header-upload stringArray                   Set HTTP header for upload transactions
      --no-check-certificate                        Do not verify the server SSL certificate (insecure)
      --no-gzip-encoding                            Don't set Accept-Encoding: gzip
      --timeout Duration                            IO idle timeout (default 5m0s)
      --tpslimit float                              Limit HTTP transactions per second to this
      --tpslimit-burst int                          Max burst of transactions for --tpslimit (default 1)
      --use-cookies                                 Enable session cookiejar
      --user-agent string                           Set the user-agent to a specified string (default "rclone/v1.64.0-beta.7130.3f588d9bb.fix-smb-about")

Performance

Could mark some of the flags as important for performance

      --buffer-size SizeSuffix                      In memory buffer size when reading files for each --transfer (default 16Mi)
      --checkers int                                Number of checkers to run in parallel (default 8)

Config

General config that didn't go elsewhere.

Some of these are more important, eg --dry-run, --checkers, --transfers so maybe there should be an important config section.

      --ask-password                                Allow prompt for password for encrypted configuration (default true)
      --auto-confirm                                If enabled, do not request console confirmation
      --cache-dir string                            Directory rclone will use for caching (default "/home/ncw/.cache/rclone")
      --color string                                When to show colors (and other ANSI codes) AUTO|NEVER|ALWAYS (default "AUTO")
      --config string                               Config file (default "/home/ncw/.rclone.conf")
      --default-time Time                           Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
      --disable string                              Disable a comma separated list of features (use --disable help to see a list)
  -n, --dry-run                                     Do a trial run with no permanent changes
      --error-on-no-transfer                        Sets exit code 9 if no files are transferred, useful in scripts
      --fs-cache-expire-duration Duration           Cache remotes for this long (0 to disable caching) (default 5m0s)
      --fs-cache-expire-interval Duration           Interval to check for expired remotes (default 1m0s)
      --human-readable                              Print numbers in a human-readable format, sizes with suffix Ki|Mi|Gi|Ti|Pi
  -i, --interactive                                 Enable interactive mode
      --kv-lock-time Duration                       Maximum time to keep key-value database locked by process (default 1s)
      --low-level-retries int                       Number of low level retries to do (default 10)
      --no-console                                  Hide console window (supported on Windows only)
      --password-command SpaceSepList               Command for supplying password for encrypted configuration
      --retries int                                 Retry operations this many times if they fail (default 3)
      --retries-sleep Duration                      Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)
      --temp-dir string                             Directory rclone will use for temporary files (default "/tmp")
      --transfers int                               Number of file transfers to run in parallel (default 4)
      --use-mmap                                    Use mmap allocator (see docs)
      --use-server-modtime                          Use server modified time instead of object metadata

Debugging

      --dump DumpFlags                              List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
      --memprofile string                           Write memory profile to file
      --cpuprofile string                           Write cpu profile to file

Filter

Filters apply to most rclone commands except rclone purge.

      --exclude stringArray                         Exclude files matching pattern
      --exclude-from stringArray                    Read file exclude patterns from file (use - to read from stdin)
      --exclude-if-present stringArray              Exclude directories if filename is present
      --files-from stringArray                      Read list of source-file names from file (use - to read from stdin)
      --files-from-raw stringArray                  Read list of source-file names from file without any processing of lines (use - to read from stdin)
  -f, --filter stringArray                          Add a file filtering rule
      --filter-from stringArray                     Read file filtering patterns from a file (use - to read from stdin)
      --ignore-case                                 Ignore case in filters (case insensitive)
      --include stringArray                         Include files matching pattern
      --include-from stringArray                    Read file include patterns from file (use - to read from stdin)
      --max-age Duration                            Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
      --max-depth int                               If set limits the recursion depth to this (default -1)
      --max-size SizeSuffix                         Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
      --max-transfer SizeSuffix                     Maximum size of data to transfer (default off)
      --min-age Duration                            Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
      --min-size SizeSuffix                         Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)

Listing

Maybe should be elsewhere?

      --fast-list                                   Use recursive list if available; uses more memory but fewer transactions

Logging

Logging and statistics stuff

      --log-file string                             Log everything to this file
      --log-format string                           Comma separated list of log format options (default "date,time")
      --log-level string                            Log level DEBUG|INFO|NOTICE|ERROR (default "NOTICE")
      --log-systemd                                 Activate systemd integration for the logger
      --max-stats-groups int                        Maximum number of stats groups to keep in memory, on max oldest is discarded (default 1000)
  -P, --progress                                    Show progress during transfer
      --progress-terminal-title                     Show progress on the terminal title (requires -P/--progress)
  -q, --quiet                                       Print as little stuff as possible
      --stats Duration                              Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s)
      --stats-file-name-length int                  Max file name length in stats (0 for no limit) (default 45)
      --stats-log-level string                      Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default "INFO")
      --stats-one-line                              Make the stats fit on one line
      --stats-one-line-date                         Enable --stats-one-line and add current date/time prefix
      --stats-one-line-date-format string           Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format
      --stats-unit string                           Show data rate in stats as either 'bits' or 'bytes' per second (default "bytes")
      --syslog                                      Use Syslog for logging
      --syslog-facility string                      Facility for syslog, e.g. KERN,USER,... (default "DAEMON")
      --use-json-log                                Use json log format
  -v, --verbose count                               Print lots more stuff (repeat for more)

Metadata

  -M, --metadata                                    If set, preserve metadata when copying objects
      --metadata-exclude stringArray                Exclude metadatas matching pattern
      --metadata-exclude-from stringArray           Read metadata exclude patterns from file (use - to read from stdin)
      --metadata-filter stringArray                 Add a metadata filtering rule
      --metadata-filter-from stringArray            Read metadata filtering patterns from a file (use - to read from stdin)
      --metadata-include stringArray                Include metadatas matching pattern
      --metadata-include-from stringArray           Read metadata include patterns from file (use - to read from stdin)
      --metadata-set stringArray                    Add metadata key=value when uploading

rc

      --rc                                          Enable the remote control server
      --rc-addr stringArray                         IPaddress:Port or :Port to bind server to (default [localhost:5572])
      --rc-allow-origin string                      Set the allowed origin for CORS
      --rc-baseurl string                           Prefix for URLs - leave blank for root
      --rc-cert string                              TLS PEM key (concatenation of certificate and CA certificate)
      --rc-client-ca string                         Client certificate authority to verify clients with
      --rc-enable-metrics                           Enable prometheus metrics on /metrics
      --rc-files string                             Path to local files to serve on the HTTP server
      --rc-htpasswd string                          A htpasswd file - if not provided no authentication is done
      --rc-job-expire-duration Duration             Expire finished async jobs older than this value (default 1m0s)
      --rc-job-expire-interval Duration             Interval to check for expired async jobs (default 10s)
      --rc-key string                               TLS PEM Private key
      --rc-max-header-bytes int                     Maximum size of request header (default 4096)
      --rc-min-tls-version string                   Minimum TLS version that is acceptable (default "tls1.0")
      --rc-no-auth                                  Don't require auth for certain methods
      --rc-pass string                              Password for authentication
      --rc-realm string                             Realm for authentication
      --rc-salt string                              Password hashing salt (default "dlPL2MqE")
      --rc-serve                                    Enable the serving of remote objects
      --rc-server-read-timeout Duration             Timeout for server reading data (default 1h0m0s)
      --rc-server-write-timeout Duration            Timeout for server writing data (default 1h0m0s)
      --rc-template string                          User-specified template
      --rc-user string                              User name for authentication
      --rc-web-fetch-url string                     URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest")
      --rc-web-gui                                  Launch WebGUI on localhost
      --rc-web-gui-force-update                     Force update to latest version of web gui
      --rc-web-gui-no-open-browser                  Don't open the browser automatically
      --rc-web-gui-update                           Check and update to latest version of web gui

Backend Flags

Most of the backend flags aren't generally relevant but these might be.

  -L, --copy-links                                          Follow symlinks and copy the pointed to item
  -l, --links                                               Translate symlinks to/from regular files with a '.rclonelink' extension
2 Likes
15

PS It would be nice if we could link the flags to their documentation pages. This isn't straightforward unfortunately as

  • not all the flags are on the same page (some are on docs, some on filtering, some on rc, however those are probably relatively easy to tell apart.)
  • we'd need to add anchors to the headings that we can make from the flag name. For instance the --verbose flag currently has the anchor v-vv-verbose. This would be a lot of manual work!

We could potentially write a linter to make sure there is a doc for every flag which might be a good idea as I know we've missed some!

1 Like
16

Overall idea looks promising - I have to think a bit more to understand it fully:)

IMO these flags from CONFIG belong rather in PERFORMANCE:

      --fs-cache-expire-duration Duration           Cache remotes for this long (0 to disable caching) (default 5m0s)
      --fs-cache-expire-interval Duration           Interval to check for expired remotes (default 1m0s)
      --low-level-retries int                       Number of low level retries to do (default 10)
      --retries int                                 Retry operations this many times if they fail (default 3)
      --retries-sleep Duration                      Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)
      --transfers int                               Number of file transfers to run in parallel (default 4)
      --use-mmap                                    Use mmap allocator (see docs)

the same with these flags from NETWORKING

      --bwlimit BwTimetable                         Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
      --bwlimit-file BwTimetable                    Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
      --tpslimit float                              Limit HTTP transactions per second to this
      --tpslimit-burst int                          Max burst of transactions for --tpslimit (default 1)

and this one in PERFORMANCE too

      --fast-list                                   Use recursive list if available; uses more memory but fewer transactions
17 
      --max-transfer SizeSuffix                     Maximum size of data to transfer (default off)

and

      --cutoff-mode string                          Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")

should be together - in COPY

18

Could we include anchor when putting flag into group?

flags.Int64VarP(flagSet, &ci.MaxDelete, "max-delete", "", -1, "When synchronizing, limit the number of deletes", "anchor", "Sync")
19

That's probably a good idea. We can put flags into multiple sections if we decide that is useful.

:+1:

We could, but I'd prefer to make it automatically from the long name of the flag then it is one less thing to worry about.

1 Like
21

This one I think is the perfect low hanging fruit to make everybody's life much easier - 700 flags alphabetical list is simply useless unless I know what I am looking for.

E.g. I saw few times question how to change file name length in stats. With groups I just look at LOGGING group and can quickly find --stats-file-name-length among other 19 flags.