Using concurrency and transfer settings per remote / backend section in config file

Help and Support
1

What is the problem you are having with rclone?

I'm trying to set how much concurrent files can be uploaded for specific remote. Remote is S3 Compatible - Wasabi. First part was Using backend flags in remote's configuration in config file .

What is your rclone version (output from rclone version)

Latest for now

root@server~ # rclone --version
rclone v1.57.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-91-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Wasabi S3 compatible

The command you were trying to run (eg rclone copy /tmp remote:tmp)

(values was reducted)

rclone --progress copy /data/sites/somesite1 wasabis3-sites-backup:data-sites-backup/tests/data-sites

The rclone config contents with secrets removed.

root@server:/root/tmp# cat /root/.config/rclone/rclone.conf
[wasabis3-sites-backup]
type = s3
provider = Wasabi
env_auth = false
access_key_id = FZZZZZZZZZZZz
secret_access_key = HZYYYYYYYYYYYY
endpoint = s3.eu-central-1.wasabisys.com
#chunk_size = 24M
no_check_bucket = true
#upload_concurrency = 1
upload_concurrency = 5
s3_upload_concurrency = 7
transfers = 9

A log from the command with the -vv flag

env vars dump:

root@server ~ # env|fgrep -i RCLONE_|wc -l
0

Beginning of the log to show no env vars set

root@server ~ # rclone -vv --progress copy /data/sites/somesite1 wasabis3-sites-backup:data-sites-backup/tests/data-sites|head -n 15
2021/12/15 22:23:13 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "--progress" "copy" "/data/sites/somesite1" "wasabis3-sites-backup:data-sites-backup/tests/data-sites"]
2021/12/15 22:23:13 DEBUG : Creating backend with remote "/data/sites/somesite1"
2021/12/15 22:23:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"                                                                                                                                                       2021/12/15 22:23:13 DEBUG : Creating backend with remote "wasabis3-sites-backup:data-sites-backup/tests/data-sites"
Transferred:              0 B / 0 B, -, 0 B/s, ETA -                                                                                                                                                                                        Elapsed time:         1.9sTransferred:                    0 B / 0 B, -, 0 B/s, ETA -
Checks:                 0 / 18, 0%                                                                                                                                                                                                          Elapsed time:         2.4s

Then I do Ctrl+C (note ^C on the last line)

Checking:
 *           wordpress/wp-admin/async-upload.php: checking
 *                wordpress/wp-admin/credits.php: checking
 *      wordpress/wp-admin/custom-background.php: checking
 *          wordpress/wp-admin/custom-header.php: checking
 *              wordpress/wp-admin/customize.php: checking
 *          wordpress/wp-admin/edit-comments.php: checking
 *     wordpress/wp-admin/edit-form-advanced.php: checking
 *       wordpress/wp-admin/edit-form-blocks.php: checking
Transferring:
 * wordpress/wp-content/languages/admin-ru_RU.po: 29% /855.394Ki, 127.958Ki/s, 4s
 * wordpress/wp-admin/inc…des/plugin-install.php:100% /33.741Ki, 16.859Ki/s, 0s
 *        wordpress/wp-admin/includes/plugin.php:100% /86.313Ki, 43.128Ki/s, 0s
 *          wordpress/wp-admin/includes/post.php:100% /73.129Ki, 36.542Ki/s, 0s^C

so it uses default 4 transfers. I couldn't find "config option name" for transfers though - Documentation mentions nothing. Sample with config option name - Amazon S3 , which are --s3-chunk-size and chunk-size accordingly.

What else I've tested:

  • setting export RCLONE_TRANSFERS=10 - is detected on startup with -vv and shown/used correctly if set.

expected results

  1. config options except auth settings are shown on -vv, if option uknown then at least warning should be shown
  2. setting transfers per remote should work
2

That's quite a bit of overhead for a typo so not super sure I agree with that as you cannot prevent everything/check for everything.

There are different flags and not every flag is consider a backend flag. Transfers a higher level setting and not a backend flag so it's not able to set in that fashion.

Documentation (rclone.org) gives a pretty good overview of the different between a global flag/backend specific flag. There are quite a number of flags that you can't set on a specific remote with an environment variable.

If you have any suggestions to cleaning up the documentation to be more clear / specific, that's always welcome.

3

I'm more interested to be able to set such settings per remote..say my local S3 and S3 on another side of the Earth could have quite different configurations.
Having ability to do this, makes sense for me and ensures working with remotes is uniform for all programs without any extra configuration and not depend on any environment variables.

4

I'm not debating or saying your reasons are not correct as for your use case, they are 100% valid.

I'd assume you have different scripts/processes and you can set per process and just adjust that way.

If you find it very useful for your use case and have the ability / time, pull requests are always appreciated.

5

I can open some bountsource for this, as technically I'm not a dev guy.

It's not impossible of course, but ruins idea of self contained configuration of remote - some flags set, some not, no warnings on startup and no mention in doc of applicable via ENV only . Inconsistency.

Technically, to my poor understanding, if it's parsed from ENV and applied per-remote, means such logic already exists.

6

I'm in the same boat :slight_smile:

I'm sure there is a lot as there are more flags / features / hidden things that I don't even know exist so trying to maintain and keep things consistent with such a big product / open source requires help from other folks to adjust / fix things like you are saying.

I'd rephrase slightly as the environment variable is the same as running --transfers 10 on a command line. They both are for that command / iteration. They are global flags applied at a top level. Transfers doesn't really go per remote as it's for an operation (copy or sync). If you try to set per remote, who wins if one is bigger or smaller?

Conceptually, certain flags are more global as the design and apply overall.

The environment variable could persist by using export or be a one time thing but not exporting it and being set via a script.

I tend to promote my scripts through a path and as you said, different values for different things as those are usually configurable things that go with a particular 'set'. Those values for me tend to live in configuration files for each tier or in the scripts themselves.

Many ways to get a job done and all are very specific to each person's use case / environment / OS / etc.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.