Make Docs for mortals not only rclone gurus

Feature
22

OK here is the first attempt. This page was generated by rclone and is a replacement for flags.md

I categorized things mostly like before, but there are likely to be mistakes as I was going through them very quickly.

Let me know what you think.

Flags can be in multiple groups.

Next step getting the right flags groups onto the command docs pages.

Global Flags

This describes the global flags available to every rclone command
split into groups.

Copy

Flags for anything which can Copy a file.

      --check-first                                 Do all the checks before starting transfers
  -c, --checksum                                    Skip based on checksum (if available) & size, not mod-time & size
      --compare-dest stringArray                    Include additional comma separated server-side paths during comparison
      --copy-dest stringArray                       Implies --compare-dest but also copies files from paths into destination
      --cutoff-mode string                          Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
      --ignore-case-sync                            Ignore case when synchronizing
      --ignore-checksum                             Skip post copy check of checksums
      --ignore-existing                             Skip all files that exist on destination
      --ignore-size                                 Ignore size when skipping use mod-time or checksum
  -I, --ignore-times                                Don't skip files that match size and time - transfer all files
      --immutable                                   Do not modify files, fail if existing files have been modified
      --inplace                                     Download directly to destination file instead of atomic download to temp/rename
      --max-backlog int                             Maximum number of objects in sync or check backlog (default 10000)
      --max-duration Duration                       Maximum duration rclone will transfer data for (default 0s)
      --max-transfer SizeSuffix                     Maximum size of data to transfer (default off)
  -M, --metadata                                    If set, preserve metadata when copying objects
      --modify-window Duration                      Max time diff to be considered the same (default 1ns)
      --multi-thread-cutoff SizeSuffix              Use multi-thread downloads for files above this size (default 250Mi)
      --multi-thread-streams int                    Max number of streams to use for multi-thread downloads (default 4)
      --multi-thread-write-buffer-size SizeSuffix   In memory buffer size for writing when in multi-thread mode (default 128Ki)
      --no-check-dest                               Don't check the destination, copy regardless
      --no-traverse                                 Don't traverse destination file system on copy
      --no-update-modtime                           Don't update destination mod-time if files identical
      --order-by string                             Instructions on how to order the transfers, e.g. 'size,descending'
      --refresh-times                               Refresh the modtime of remote files
      --server-side-across-configs                  Allow server-side operations (e.g. copy) to work across different configs
      --size-only                                   Skip based on size only, not mod-time or checksum
      --streaming-upload-cutoff SizeSuffix          Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
  -u, --update                                      Skip files that are newer on the destination

Sync

Flags just used for rclone sync.

      --backup-dir string               Make backups into hierarchy based in DIR
      --delete-after                    When synchronizing, delete files on destination after transferring (default)
      --delete-before                   When synchronizing, delete files on destination before transferring
      --delete-during                   When synchronizing, delete files during transfer
      --ignore-errors                   Delete even if there are I/O errors
      --max-delete int                  When synchronizing, limit the number of deletes (default -1)
      --max-delete-size SizeSuffix      When synchronizing, limit the total size of deletes (default off)
      --suffix string                   Suffix to add to changed files
      --suffix-keep-extension           Preserve the extension when using --suffix
      --track-renames                   When synchronizing, track file renames and do a server-side move if possible
      --track-renames-strategy string   Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash")

Important

Important flags useful for most commands.

  -n, --dry-run         Do a trial run with no permanent changes
  -i, --interactive     Enable interactive mode
  -v, --verbose count   Print lots more stuff (repeat for more)

Check

Flags used for rclone check.

      --max-backlog int   Maximum number of objects in sync or check backlog (default 10000)

Networking

General networking and HTTP stuff.

      --bind string                        Local address to bind to for outgoing connections, IPv4, IPv6 or name
      --bwlimit BwTimetable                Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
      --bwlimit-file BwTimetable           Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
      --ca-cert stringArray                CA certificate used to verify servers
      --client-cert string                 Client SSL certificate (PEM) for mutual TLS auth
      --client-key string                  Client SSL private key (PEM) for mutual TLS auth
      --contimeout Duration                Connect timeout (default 1m0s)
      --disable-http-keep-alives           Disable HTTP keep-alives and use each connection once.
      --disable-http2                      Disable HTTP/2 in the global transport
      --dscp string                        Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21
      --expect-continue-timeout Duration   Timeout when using expect / 100-continue in HTTP (default 1s)
      --header stringArray                 Set HTTP header for all transactions
      --header-download stringArray        Set HTTP header for download transactions
      --header-upload stringArray          Set HTTP header for upload transactions
      --no-check-certificate               Do not verify the server SSL certificate (insecure)
      --no-gzip-encoding                   Don't set Accept-Encoding: gzip
      --timeout Duration                   IO idle timeout (default 5m0s)
      --tpslimit float                     Limit HTTP transactions per second to this
      --tpslimit-burst int                 Max burst of transactions for --tpslimit (default 1)
      --use-cookies                        Enable session cookiejar
      --user-agent string                  Set the user-agent to a specified string (default "rclone/v1.64.0-beta.7144.ed3757555.fix-flag-groups")

Performance

Flags helpful for increasing performance.

      --buffer-size SizeSuffix   In memory buffer size when reading files for each --transfer (default 16Mi)
      --checkers int             Number of checkers to run in parallel (default 8)
      --transfers int            Number of file transfers to run in parallel (default 4)

Config

General configuration of rclone.

      --ask-password                        Allow prompt for password for encrypted configuration (default true)
      --auto-confirm                        If enabled, do not request console confirmation
      --cache-dir string                    Directory rclone will use for caching (default "$HOME/.cache/rclone")
      --color string                        When to show colors (and other ANSI codes) AUTO|NEVER|ALWAYS (default "AUTO")
      --config string                       Config file (default "$HOME/.config/rclone/rclone.conf")
      --default-time Time                   Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
      --disable string                      Disable a comma separated list of features (use --disable help to see a list)
  -n, --dry-run                             Do a trial run with no permanent changes
      --error-on-no-transfer                Sets exit code 9 if no files are transferred, useful in scripts
      --fs-cache-expire-duration Duration   Cache remotes for this long (0 to disable caching) (default 5m0s)
      --fs-cache-expire-interval Duration   Interval to check for expired remotes (default 1m0s)
      --human-readable                      Print numbers in a human-readable format, sizes with suffix Ki|Mi|Gi|Ti|Pi
  -i, --interactive                         Enable interactive mode
      --kv-lock-time Duration               Maximum time to keep key-value database locked by process (default 1s)
      --low-level-retries int               Number of low level retries to do (default 10)
      --no-console                          Hide console window (supported on Windows only)
      --no-unicode-normalization            Don't normalize unicode characters in filenames
      --password-command SpaceSepList       Command for supplying password for encrypted configuration
      --retries int                         Retry operations this many times if they fail (default 3)
      --retries-sleep Duration              Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)
      --temp-dir string                     Directory rclone will use for temporary files (default "/tmp")
      --use-mmap                            Use mmap allocator (see docs)
      --use-server-modtime                  Use server modified time instead of object metadata

Debugging

Flags for developers.

      --cpuprofile string   Write cpu profile to file
      --dump DumpFlags      List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
      --dump-bodies         Dump HTTP headers and bodies - may contain sensitive info
      --dump-headers        Dump HTTP headers - may contain sensitive info
      --memprofile string   Write memory profile to file

Filter

Flags for filtering directory listings.

      --delete-excluded                     Delete files on dest excluded from sync
      --exclude stringArray                 Exclude files matching pattern
      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
      --exclude-if-present stringArray      Exclude directories if filename is present
      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
  -f, --filter stringArray                  Add a file filtering rule
      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
      --ignore-case                         Ignore case in filters (case insensitive)
      --include stringArray                 Include files matching pattern
      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
      --metadata-exclude stringArray        Exclude metadatas matching pattern
      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
      --metadata-filter stringArray         Add a metadata filtering rule
      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
      --metadata-include stringArray        Include metadatas matching pattern
      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)

Listing

Flags for listing directories.

      --default-time Time   Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
      --fast-list           Use recursive list if available; uses more memory but fewer transactions
      --max-depth int       If set limits the recursion depth to this (default -1)

Logging

Logging and statistics.

      --log-file string                     Log everything to this file
      --log-format string                   Comma separated list of log format options (default "date,time")
      --log-level string                    Log level DEBUG|INFO|NOTICE|ERROR (default "NOTICE")
      --log-systemd                         Activate systemd integration for the logger
      --max-stats-groups int                Maximum number of stats groups to keep in memory, on max oldest is discarded (default 1000)
  -P, --progress                            Show progress during transfer
      --progress-terminal-title             Show progress on the terminal title (requires -P/--progress)
  -q, --quiet                               Print as little stuff as possible
      --stats Duration                      Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s)
      --stats-file-name-length int          Max file name length in stats (0 for no limit) (default 45)
      --stats-log-level string              Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default "INFO")
      --stats-one-line                      Make the stats fit on one line
      --stats-one-line-date                 Enable --stats-one-line and add current date/time prefix
      --stats-one-line-date-format string   Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format
      --stats-unit string                   Show data rate in stats as either 'bits' or 'bytes' per second (default "bytes")
      --syslog                              Use Syslog for logging
      --syslog-facility string              Facility for syslog, e.g. KERN,USER,... (default "DAEMON")
      --use-json-log                        Use json log format
  -v, --verbose count                       Print lots more stuff (repeat for more)

Metadata

Flags to control metadata.

  -M, --metadata                   If set, preserve metadata when copying objects
      --metadata-set stringArray   Add metadata key=value when uploading

RC

Flags to control the Remote Control API.

      --rc                                 Enable the remote control server
      --rc-addr stringArray                IPaddress:Port or :Port to bind server to (default [localhost:5572])
      --rc-allow-origin string             Set the allowed origin for CORS
      --rc-baseurl string                  Prefix for URLs - leave blank for root
      --rc-cert string                     TLS PEM key (concatenation of certificate and CA certificate)
      --rc-client-ca string                Client certificate authority to verify clients with
      --rc-enable-metrics                  Enable prometheus metrics on /metrics
      --rc-files string                    Path to local files to serve on the HTTP server
      --rc-htpasswd string                 A htpasswd file - if not provided no authentication is done
      --rc-job-expire-duration Duration    Expire finished async jobs older than this value (default 1m0s)
      --rc-job-expire-interval Duration    Interval to check for expired async jobs (default 10s)
      --rc-key string                      TLS PEM Private key
      --rc-max-header-bytes int            Maximum size of request header (default 4096)
      --rc-min-tls-version string          Minimum TLS version that is acceptable (default "tls1.0")
      --rc-no-auth                         Don't require auth for certain methods
      --rc-pass string                     Password for authentication
      --rc-realm string                    Realm for authentication
      --rc-salt string                     Password hashing salt (default "dlPL2MqE")
      --rc-serve                           Enable the serving of remote objects
      --rc-server-read-timeout Duration    Timeout for server reading data (default 1h0m0s)
      --rc-server-write-timeout Duration   Timeout for server writing data (default 1h0m0s)
      --rc-template string                 User-specified template
      --rc-user string                     User name for authentication
      --rc-web-fetch-url string            URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest")
      --rc-web-gui                         Launch WebGUI on localhost
      --rc-web-gui-force-update            Force update to latest version of web gui
      --rc-web-gui-no-open-browser         Don't open the browser automatically
      --rc-web-gui-update                  Check and update to latest version of web gui

Backend

Backend only flags. These can be set in the config file also.

(omitted for brevity!)

1 Like
23

This looks so much better than monster list before! Definitely this is going to make life of many much easier.

24

I have made the first implementation of this change.

Check out https://tip.rclone.org/

In particular Global Flags which has a categorised list of flags and the commands. Here is cat rclone cat which has a new section for relevant global flags

image

Here the Filter and Listing options.

Which category each flag is in (and it can be in more than one) is defined at the time the flag is created, and which flag groups each comment shows is defined in an annotation on the command.

The docs are all built automatically from those.

Hopefully this will be easier to use!

I could do with help checking this though

  1. Are the flags in flags.md in the right sections.
  2. Should there be any more / less sections
  3. Do the commands have the right groups of flags? (checkout sync which is probably the most complicated)

Any comments and or PRs welcome :slight_smile:

Cc: @albertony and @Ole who are both good at docs!

1 Like
25

Thank you. Looks so much better!

I will go thought details slowly but one comment I have already is regarding backend flags section.

It is still LARGE monster:) Maybe when auto generating this section it could be split by first word in a flag e.g.:

azureblob
      --azureblob-access-tier string                        Access tier of blob: hot, cool or archive
      --azureblob-account string                            Azure Storage Account Name
      --azureblob-archive-tier-delete                       Delete archive tier blobs before overwriting
      --azureblob-chunk-size SizeSuffix                     Upload chunk size (default 4Mi)

box
      --box-access-token string                             Box App Primary Access Token
      --box-auth-url string                                 Auth server URL
      --box-box-config-file string                          Box App config.json location
      --box-box-sub-type string                              (default "user")

and two odd links flags there added to local section:

local
      --local-case-insensitive                              Force the filesystem to report itself as case insensitive
      --local-case-sensitive                                Force the filesystem to report itself as case sensitive
      --local-encoding MultiEncoder                         The encoding for the backend (default Slash,Dot)
...
      -l, --links                                               Translate symlinks to/from regular files with a '.rclonelink' extension
      -L, --copy-links                                          Follow symlinks and copy the pointed to item
26

My second general comment.

Global flags now start looking tidy, logical and much easier to use. But what is missing IMO is to have ability to click at link and see any flag detailed description. They are mostly on Usage page. What would be required to achieve it? I am happy to help with tiding things if needed. Does not have to be in the first release of Docs but would be very nice to have at some stage.

Small things I think that can be useful and clarify matters I will do through PRs. If not sure will discuss here.

27

Great!

It is there more for completeness. The flags are all documented in the backend pages. I'll have a think about splitting them up, unfortunately it isn't as easy as you might think due to the rather complicated way the flags get generated!

--link and --copy-link are actually local backend flags! They don't affect any other backends.

Yes this would be good. What we would need for that is to make sure we've used consistent anchors in the docs pages they we could link to them. We'd also have to think of a nice way of rendering the flags in markdown since they are currently rendered fixed width which can't have links

So currently

    --flag1    flag1 description
    --flag2    flag2 description

Maybe a table?

flag description
--flag1 flag1 description
--flag2 flag2 description

Or maybe

flag description
--flag1 flag1 description
--flag2 flag2 description

The work to do this would be

  • make sure each flag in docs.md has a consistent anchor so the headings need to be something like
### --option STRING {#option}
  • the docs generator needs to know where the docs for all the flags are. Most are in /docs but the filter flags are in /filtering and the rc flags in /rc - this is easy to work out from the group.
  • then the docs generator needs to format the flags.

Open question - where are all the flags documented?

28

none of the above looks good...

The first one does make flags names no easy to read when you have many of them
The second maybe IMO looks very bad

What about something like for remotes?

image
image846×290 19.8 KB

extra blue icon - with label details,description or more

29

Nice idea, so something more like this? Where (docs) is the nice button shape? (I realised we need a colum for short opts too).

short long description more
-F --flag1 flag 1 description (docs)
--flag2 flag 2 descriptin (docs)
30

Definitely this is much better way to present it.

I am only wondering how it will look like for flags with very long description, e.g.:

      --sharefile-encoding MultiEncoder                     The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot)

If we would have link to full flag description maybe we can trim flags description like this? Because as it is formatted now:

image
image2684×1436 476 KB

in this case second from the bottom flag:

--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)

would push the last column outside viewing field.

So maybe to make it description length proof:

more short long description
(docs) -F --flag1 flag 1 description
(docs) --flag2 flag 2 descriptin
31

I think what should happen if using tables is that the long lines wrap into two lines which would work quite well. I'll have to experiment.

You are right though we don't want the buttons being pushed off the page.

32

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.