What is the problem you are having with rclone?
I'm trying to configure per-remote backend flags in config, in particual --s3-chunk-size
and --s3-upload-concurrency
What is your rclone version (output from rclone version
)
checked with:
rclone v1.50.2 - Ubuntu 20.04
rclone v1.56.2 - release from Github page
Which cloud storage system are you using? (eg Google Drive)
Wasabi S3 in this example
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone -vv -P copy /var/lib/mysql/mysql-bin.001098 wasabis3:company-mysql-backup/blabla/
The rclone config contents with secrets removed.
RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE = 6M
RCLONE_CONFIG_WASABIS3_S3_UPLOAD_CONCURRENCY = 1
[wasabis3]
type = s3
provider = Wasabi
env_auth = false
access_key_id = BXXXXXXXXX
secret_access_key = eZZZZZZZZZZZZZZZZZ
endpoint = s3.eu-central-1.wasabisys.com
A log from the command with the -vv
flag
./rclone -vv -P copy /var/lib/mysql/mysql-bin.001098 wasabis3:mycompany-mysql-backup/blabla/
2021/10/08 21:03:01 DEBUG : rclone: Version "v1.56.2" starting with parameters ["./rclone" "-vv" "-P" "copy" "/var/lib/mysql/mysql-bin.001098" "wasabis3:mycompany-mysql-backup/blabla/"]
2021/10/08 21:03:01 DEBUG : Creating backend with remote "/var/lib/mysql/mysql-bin.001098"
2021/10/08 21:03:01 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/10/08 21:03:01 DEBUG : fs cache: adding new entry for parent of "/var/lib/mysql/mysql-bin.001098", "/var/lib/mysql"
2021/10/08 21:03:01 DEBUG : Creating backend with remote "wasabis3:mycompany-mysql-backup/blabla/"
2021/10/08 21:03:01 DEBUG : fs cache: renaming cache item "wasabis3:mycompany-mysql-backup/blabla/" to be canonical "wasabis3:mycompany-mysql-backup/blabla"
2021-10-08 21:03:02 DEBUG : mysql-bin.001098: Need to transfer - File not found at Destination
2021-10-08 21:03:02 INFO : S3 bucket mycompany-mysql-backup path blabla: Bucket "mycompany-mysql-backup" created with ACL "private"
2021-10-08 21:03:04 DEBUG : mysql-bin.001098: multipart upload starting chunk 1 size 5Mi offset 0/1.002Gi
2021-10-08 21:03:04 DEBUG : mysql-bin.001098: multipart upload starting chunk 2 size 5Mi offset 5Mi/1.002Gi
2021-10-08 21:03:04 DEBUG : mysql-bin.001098: multipart upload starting chunk 3 size 5Mi offset 10Mi/1.002Gi
So RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE = 6M is ignored.
I've tried to put it inside remote [wasabis3]
section, i.e.
[wasabis3]
type = s3
provider = Wasabi
env_auth = false
access_key_id = BXXXXXXXXX
secret_access_key = eZZZZZZZZZZZZZZZZZ
endpoint = s3.eu-central-1.wasabisys.com
RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE = 6M
RCLONE_CONFIG_WASABIS3_S3_UPLOAD_CONCURRENCY = 1
But had no difference.
From my PoV, documentation is very blurry on it, without config file snippet example. What I could understand, that in config parsing and environment variables parsing is done via the same mechanics, so variable name export RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE should be correct for my case.
Tried to set it via ENV, no luck as well
root@node3 ~/tmp/rclone-v1.56.2-linux-amd64 # env|fgrep -i RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE
RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE=6M
Still chunk size of 5M was used.
I guess I do misread something, need a working config sample/direction on doing it right.