I already have chunk_size set in rclone.conf, but when I specify it once again on the command line as --drive-chunk-size, I get drastically increased transfer rates. Is this really taken from the config file? How can I tell?
Run the command 'rclone version' and share the full output of the command.
rclone v1.58.1
os/version: oracle 6.10 (64 bit)
os/kernel: 4.1.12-124.48.6.el6uek.x86_64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.17.9
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
crypt on top of Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
yeah, I am indeed transferring a whole bunch of small files, but some big ones too, its a mix of logs
the transfer rate goes down when it stumbles upon a bunch of small files, but it also goes up when it gets to big ones. I've been running without --drive-chunk-size for hours and have never seen the transfer rate go nowhere near the transfer rates with the option, its a crazy coincidence, if it is a coincidence.
my question was actually: is there a way to display somehow what is the end configuration for a remote, including all the ways one can override the config, rclone.conf, env variables, cmdline options and whatnot
in the -vvv log i can see that the config was overridden, but not exactly what was overridden and with what
If I'm not mistaken, anything from an environment variable/config happens first and then it applies the command line flags you specify.
So in my example, you can see I ran with the flag 16M for my chunk size which overrides my environment/config file as that 'wins'.
felix@gemini:~$ rclone copy /etc/hosts DB: -vvv --dropbox-chunk-size 16M
2022/05/04 11:10:18 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/05/04 11:10:18 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "copy" "/etc/hosts" "DB:" "-vvv" "--dropbox-chunk-size" "16M"]
2022/05/04 11:10:18 DEBUG : Creating backend with remote "/etc/hosts"
2022/05/04 11:10:18 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/05/04 11:10:18 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2022/05/04 11:10:18 DEBUG : Creating backend with remote "DB:"
2022/05/04 11:10:18 DEBUG : DB: detected overridden config - adding "{QEqRK}" suffix to name
2022/05/04 11:10:18 DEBUG : fs cache: renaming cache item "DB:" to be canonical "DB{QEqRK}:"
2022/05/04 11:10:18 DEBUG : hosts: Need to transfer - File not found at Destination
2022/05/04 11:10:19 DEBUG : hosts: Uploading chunk 1/1
2022/05/04 11:10:20 DEBUG : hosts: Uploading chunk 2/1
2022/05/04 11:10:20 DEBUG : Dropbox root '': Adding "/hosts" to batch
2022/05/04 11:10:21 DEBUG : Dropbox root '': Batch idle for 500ms so committing
2022/05/04 11:10:21 DEBUG : Dropbox root '': Committing sync batch length 1 starting with: /hosts
2022/05/04 11:10:22 DEBUG : Dropbox root '': Upload batch completed in 142.478935ms
2022/05/04 11:10:22 DEBUG : Dropbox root '': Committed sync batch length 1 starting with: /hosts
2022/05/04 11:10:22 DEBUG : hosts: dropbox = 36600f2d623ef48807551ee091ef25a9563094245b344b55654c77361add095b OK
2022/05/04 11:10:22 INFO : hosts: Copied (new)
2022/05/04 11:10:22 INFO :
Transferred: 236 B / 236 B, 100%, 78 B/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3.9s
2022/05/04 11:10:22 DEBUG : 10 go routines active
2022/05/04 11:10:22 INFO : Dropbox root '': Commiting uploads - please wait...
Your config file with the value you is fine so no need to specify it on the command line as well. The overridden part is a flag you specified has overrode some value in your config file or environment variables. So if I remove the flag, the override part goes away.
felix@gemini:~$ rclone delete DB:hosts
felix@gemini:~$ rclone copy /etc/hosts DB: -vvv
2022/05/04 11:12:24 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/05/04 11:12:24 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "copy" "/etc/hosts" "DB:" "-vvv"]
2022/05/04 11:12:24 DEBUG : Creating backend with remote "/etc/hosts"
2022/05/04 11:12:24 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/05/04 11:12:24 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2022/05/04 11:12:24 DEBUG : Creating backend with remote "DB:"
2022/05/04 11:12:24 DEBUG : hosts: Need to transfer - File not found at Destination
2022/05/04 11:12:25 DEBUG : hosts: Uploading chunk 1/1
2022/05/04 11:12:26 DEBUG : hosts: Uploading chunk 2/1
2022/05/04 11:12:26 DEBUG : Dropbox root '': Adding "/hosts" to batch
2022/05/04 11:12:27 DEBUG : Dropbox root '': Batch idle for 500ms so committing
2022/05/04 11:12:27 DEBUG : Dropbox root '': Committing sync batch length 1 starting with: /hosts
2022/05/04 11:12:28 DEBUG : Dropbox root '': Upload batch completed in 150.108357ms
2022/05/04 11:12:28 DEBUG : Dropbox root '': Committed sync batch length 1 starting with: /hosts
2022/05/04 11:12:28 DEBUG : hosts: dropbox = 36600f2d623ef48807551ee091ef25a9563094245b344b55654c77361add095b OK
2022/05/04 11:12:28 INFO : hosts: Copied (new)
2022/05/04 11:12:28 INFO :
Transferred: 236 B / 236 B, 100%, 78 B/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3.7s
2022/05/04 11:12:28 DEBUG : 10 go routines active
2022/05/04 11:12:28 INFO : Dropbox root '': Commiting uploads - please wait...
You can dump headers/requests if you want to inspect the transactions to see what's being passed specifically.
--dump headers,requests
I used Google Drive/rclone for quite some time so I'm very sure it works as I did quite a bit of testing with that particular flag before I moved on to Dropbox.
Again, you've shared a snippet of a log that doesn't say much, if you are moving lots of small files, Google stinks for that. You have a huge number of transfers/checkers which is probably slowing you down, but again, without sharing a full debug log, I can't be sure of what your issue is as that's why we ask for it so we have all the details and don't have to guess.
I used rclone --log-level DEBUG --log-file /tmp/rclone-without-cmdline.log, but there is nothing interesting in there, just a bunch of
filename: : md5 = 600373a637dc6dd70a4c4018046532ab OK
filename: Copied (new)
and not much else, except at the beginning when it is figuring out what to copy, where it is full of
filename: Size and modification time the same (differ by 0s, within tolerance 1ms)
filename: Unchanged skipping
I am a bit hesitant about sharing the logs as I'll have to anonymize the filenames as they have quite a bit of metadata in them that I'm not supposed to share
