What is the problem you are having with rclone?
Copy performance to gdrive is unexpectedly low on fast connection, looking for anything to improve it. I'm currently at a symmetric fiber connection, and I'm able speedtest consistently around 400Mbit/sec, and that's sustainable as far as I can tell, though I'm currently doing it over wifi, and probably won't be able to try it wired. I wanted to copy some larger files, and a limit at 30MByte/sec seemed reasonable, but I can't seem to actually get a sustained transfer speed above ~2-3MByte/sec though that's often on the optimistic side even. I've run it several times over the last 3 days and it doesn't appear to be a transient problem.
I do get what looks like an initial burst around 30MB/sec, though it jumps around a lot when as I retry, so I'm not certain that it is hitting that and then throttling, which is sort of what that looks like. The connection isn't supposed to be throttled though, so I'm working from the basis that is true.
Machine I'm doing this from is a quad core ryzen laptop with 32GB/memory, and the files being uploaded are on external SSD connected over USB3. I don't see any resource limits being hit when I run the transfer.
So right now, mostly checking to see I didn't mess up my command or if there's something I should do differently.
Thanks!
Run the command 'rclone version' and share the full output of the command.
rclone v1.57.0
- os/version: Microsoft Windows 10 Home 2009 (64 bit)
- os/kernel: 10.0.19043.1586 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy 'A:/#stage' 'gd:/#stage' --ignore-existing --verbose --transfers 4 --checkers 8 --bwlimit 30M --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --drive-chunk-size 512M --stats 1s --stats-file-name-length 0 --drive-pacer-burst 2000 --drive-pacer-min-sleep 5ms --user-agent ******* -P
The rclone config contents with secrets removed.
[gd]
type = drive
client_id = xxxx.apps.googleusercontent.com
client_secret = xxxx
scope = drive
root_folder_id = xxxx
token = {"access_token":"xxxx","token_type":"Bearer","refresh_token":"1//xxxx","expiry":"2022-04-06T10:34:58.7318335-04:00"}
[secO]
type = crypt
remote = gd:/secO
filename_encryption = obfuscate
password =
A log from the command with the -vv
flag
Transcript started, output file is rclone.txt
PS C:\scripts> .\copystuff.ps1
2022/04/06 10:50:55 INFO : Starting bandwidth limiter at 30Mi Byte/s
2022/04/06 10:50:55 INFO : Starting transaction limiter: max 10 transactions/s with burst 1
2022/04/06 10:50:55 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\rclone\\rclone.exe" "copy" "A:/#stage" "secO:/#stage" "--ignore-existing" "--verbose" "--tpslimit"
"10" "--bwlimit" "30M" "--contimeout" "60s" "--timeout" "300s" "--retries" "3" "--low-level-retries" "10" "--drive-chunk-size" "512M" "--stats" "1s" "--stats-file-name-length" "0" "-
-drive-pacer-burst" "2000" "--drive-pacer-min-sleep" "5ms" "--user-agent" "*******" "-P" "-vv"]
2022/04/06 10:50:55 DEBUG : Creating backend with remote "A:/#stage"
2022/04/06 10:50:55 DEBUG : Using config file from "C:\\Users\\Chiefmas\\AppData\\Roaming\\rclone\\rclone.conf"
2022/04/06 10:50:55 DEBUG : fs cache: renaming cache item "A:/#stage" to be canonical "//?/A:/#stage"
2022/04/06 10:50:55 DEBUG : Creating backend with remote "secO:/#stage"
2022/04/06 10:50:56 DEBUG : Creating backend with remote "gd:/secO/55.#yzgmk"
2022/04/06 10:50:56 DEBUG : gd: detected overridden config - adding "{sPcu0}" suffix to name
2022/04/06 10:50:56 DEBUG : fs cache: renaming cache item "gd:/secO/55.#yzgmk" to be canonical "gd{sPcu0}:secO/55.#yzgmk"
2022/04/06 10:50:56 DEBUG : fs cache: switching user supplied name "gd:/secO/55.#yzgmk" for canonical name "gd{sPcu0}:secO/55.#yzgmk"
2022-04-06 10:50:56 DEBUG : Encrypted drive 'secO:/#stage': Waiting for checks to finish
2022-04-06 10:50:56 DEBUG : Encrypted drive 'secO:/#stage': Waiting for transfers to finish
2022-04-06 10:50:57 DEBUG : 172.pQxO txOP (3199) (6h PzxK LC 57JJ)/40.iJqH mqHI (6422) (9a IsqD Ev 80CC).CAL: Sending chunk 0 length 536870912
Transferred: 313.702 MiB / 73.217 GiB, 0%, 2.607 MiB/s, ETA 7h57m17s
Transferred: 0 / 1, 0%
Elapsed time: 1m58.7s
Transferring:
* Huge File (1977) (4K scan of 35mm)/Huge File (1977) (4K scan of 35mm).mkv: 0% /73.217Gi, 2.607Mi/s, 7h57m16s
PS C:\scripts> TerminatingError(): "The pipeline has been stopped."
>> TerminatingError(): "The pipeline has been stopped."
>> TerminatingError(): "The pipeline has been stopped."
PS C:\scripts> Stop-Transcript
**********************
PowerShell transcript end
End time: 20220406105258
**********************