Still Very Slow Uploads to Dropbox

alexkrohn · October 10, 2023, 8:16pm

What is the problem you are having with rclone?

Uploads to Dropbox are still very slow, despite trying the solutions outlined here.

Speeds uploading to Dropbox cap around 2-4 MiB/s. Speeds downloading from Dropbox or AWS are near 100 MiB/s.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

- os/version: ubuntu 22.04 (64 bit)

- os/kernel: 5.15.0-84-generic (x86_64)

- os/type: linux

- os/arch: amd64

- go/version: go1.21.1

- go/linking: static

- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox. Speeds are similar when going from my local machine to Dropbox or from AWS to Dropbox.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone move -Pvc ./ dropbox-remote:20231010_BarnOwl3RAD --include="Bird*" --bind 0.0.0.0 --checkers 12 --tpslimit 12
2023/10/10 19:14:00 INFO  : Starting transaction limiter: max 12 transactions/s with burst 1

Transferred:   	    6.879 GiB / 65.897 GiB, 10%, 1.912 MiB/s, ETA 8h46m55s
Checks:                56 / 60, 93%
Deleted:               28 (files), 0 (dirs)
Renamed:               28
Transferred:           28 / 325, 9%
Elapsed time:      55m8.7s
Checking:

Transferring:
 *                        Bird12_R1_001.fastq.gz: 74% /373.596Mi, 449.041Ki/s, 3m35s
 *                        Bird12_R2_001.fastq.gz: 74% /373.822Mi, 533.292Ki/s, 3m3s
 *                        Bird13_R1_001.fastq.gz: 54% /427.302Mi, 544.191Ki/s, 6m4s
 *                        Bird13_R2_001.fastq.gz: 34% /436.036Mi, 419.675Ki/s, 11m38s

Note that transfer speeds are siilar whether --bind uses 0.0.0.0 or my actual local ipV4.

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[aws-tbc]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-east-1

[dropbox-remote]
type = dropbox
token = XXX
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the `-vv` flag

rclone move -Pvc ./ dropbox-remote:20231010_BarnOwl3RAD --include="Bird*" --bind 0.0.0.0 --checkers 12 --tpslimit 12 -vv
2023/10/10 20:11:52 INFO  : Starting transaction limiter: max 12 transactions/s with burst 1
2023/10/10 20:11:52 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "move" "-Pvc" "./" "dropbox-remote:20231010_BarnOwl3RAD" "--in
clude=Bird*" "--bind" "0.0.0.0" "--checkers" "12" "--tpslimit" "12" "-vv"]
2023/10/10 20:11:52 DEBUG : Creating backend with remote "./"
2023/10/10 20:11:52 DEBUG : Using config file from "/home/tangled/.config/rclone/rclone.conf"
2023/10/10 20:11:52 DEBUG : fs cache: renaming cache item "./" to be canonical "/home/tangled/oct2023-sequencing-30-811641238"
2023/10/10 20:11:52 DEBUG : Creating backend with remote "dropbox-remote:20231010_BarnOwl3RAD"
2023/10/10 20:11:53 DEBUG : Dropbox root '': Using root namespace "2971962001"
2023/10/10 20:11:53 DEBUG : fs cache: renaming cache item "dropbox-remote:20231010_BarnOwl3RAD" to be canonical "dropbox-remote:TBCShareable/genomics-backup/20231010_BarnOwl3RAD"
2023-10-10 20:11:53 DEBUG : Bird12_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird12_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird13_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird13_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird13_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird14_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird78_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird78_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird78_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird79_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird79_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird79_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird79_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird80_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird80_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird80_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird80_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird81_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird81_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird81_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird81_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird82_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird82_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird82_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird82_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird83_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird83_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird83_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird83_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird85_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird85_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird85_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird85_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird86_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird86_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird86_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird86_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird87_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird87_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird87_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird87_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird88_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird88_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird88_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird88_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird89_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird89_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird89_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird89_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird90_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird90_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird90_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird90_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird92_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird92_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird92_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird92_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird93_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird93_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird93_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird93_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird94_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird94_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird94_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird94_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird95_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird95_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird95_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird95_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird96_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird96_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird96_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird96_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird97_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird97_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird97_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird97_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird98_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird98_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird98_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird98_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird99_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird99_R1_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird99_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Bird99_R2_001.fastq.gz.md5: Need to transfer - File not found at Destination
2023-10-10 20:11:53 DEBUG : Dropbox root 'TBCShareable/genomics-backup/20231010_BarnOwl3RAD': Waiting for checks to finish
2023-10-10 20:11:53 DEBUG : Dropbox root 'TBCShareable/genomics-backup/20231010_BarnOwl3RAD': Waiting for transfers to finish
2023-10-10 20:11:54 DEBUG : Bird12_R2_001.fastq.gz: Uploading chunk 1/8
2023-10-10 20:11:54 DEBUG : Bird13_R1_001.fastq.gz: Uploading chunk 1/9
2023-10-10 20:11:54 DEBUG : Bird13_R2_001.fastq.gz: Uploading chunk 1/10
2023-10-10 20:11:54 DEBUG : Bird12_R1_001.fastq.gz: Uploading chunk 1/8

Animosity022 · October 10, 2023, 9:16pm

You'd want to make your own client ID/secret.

You'd want to share a little more debug as not much to see in 2.5 seconds.

alexkrohn · October 11, 2023, 12:55pm

Hi @Animosity022 . Can you explain what it means to make your Client ID secret?

The command finished properly without any evidence of problems or warnings. There were dozens of files, but the debugs look pretty similar. Here's the last one as an example:

2023-10-11 01:32:54 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Adding "20231010_BarnOwl3RAD/Bird99_R2_001.fastq.gz.md5" to batch
2023-10-11 01:32:54 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Batch idle for 500ms so committing
2023-10-11 01:32:54 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Committing sync batch length 1 starting with: 20231010_BarnOwl3RAD/Bird99_R2_001.fastq.gz.md5
2023-10-11 01:32:55 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Committed sync batch length 1 starting with: 20231010_BarnOwl3RAD/Bird99_R2_001.fastq.gz.md5
2023-10-11 01:32:55 DEBUG : Bird99_R2_001.fastq.gz.md5: dropbox = 9d816189c7246d04672fc458d441bd98e911d32f8893691c582518ea04d2e964 OK
2023-10-11 01:32:55 INFO  : Bird99_R2_001.fastq.gz.md5: Copied (new)
2023-10-11 01:32:55 INFO  : Bird99_R2_001.fastq.gz.md5: Deleted
2023-10-11 01:33:08 DEBUG : Bird98_R2_001.fastq.gz: Uploading chunk 11/11
2023-10-11 01:33:08 DEBUG : Bird99_R2_001.fastq.gz: Uploading chunk 9/14
2023-10-11 01:33:12 DEBUG : Bird99_R1_001.fastq.gz: Uploading chunk 9/13
2023-10-11 01:33:33 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Adding "20231010_BarnOwl3RAD/Bird98_R2_001.fastq.gz" to batch
2023-10-11 01:33:34 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Batch idle for 500ms so committing
2023-10-11 01:33:34 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Committing sync batch length 1 starting with: 20231010_BarnOwl3RAD/Bird98_R2_001.fastq.gz
2023-10-11 01:33:34 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Committed sync batch length 1 starting with: 20231010_BarnOwl3RAD/Bird98_R2_001.fastq.gz
2023-10-11 01:33:34 DEBUG : Bird98_R2_001.fastq.gz: dropbox = 89b119679980bd8905ffb7b7c1dbdc57ccc5fee37981023ac262767adf32be41 OK
2023-10-11 01:33:34 INFO  : Bird98_R2_001.fastq.gz: Copied (new)
2023-10-11 01:33:34 INFO  : Bird98_R2_001.fastq.gz: Deleted
2023-10-11 01:33:38 DEBUG : Bird99_R1_001.fastq.gz: Uploading chunk 10/13
2023-10-11 01:33:41 DEBUG : Bird99_R2_001.fastq.gz: Uploading chunk 10/14
2023-10-11 01:34:00 DEBUG : Bird99_R2_001.fastq.gz: Uploading chunk 11/14
2023-10-11 01:34:04 DEBUG : Bird99_R1_001.fastq.gz: Uploading chunk 11/13
2023-10-11 01:34:22 DEBUG : Bird99_R1_001.fastq.gz: Uploading chunk 12/13
2023-10-11 01:34:25 DEBUG : Bird99_R2_001.fastq.gz: Uploading chunk 12/14
2023-10-11 01:34:41 DEBUG : Bird99_R1_001.fastq.gz: Uploading chunk 13/13
2023-10-11 01:34:48 DEBUG : Bird99_R2_001.fastq.gz: Uploading chunk 13/14
2023-10-11 01:34:57 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Adding "20231010_BarnOwl3RAD/Bird99_R1_001.fastq.gz" to batch
2023-10-11 01:34:57 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Batch idle for 500ms so committing
2023-10-11 01:34:57 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Committing sync batch length 1 starting with: 20231010_BarnOwl3RAD/Bird99_R1_001.fastq.gz
2023-10-11 01:34:58 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Committed sync batch length 1 starting with: 20231010_BarnOwl3RAD/Bird99_R1_001.fastq.gz
2023-10-11 01:34:58 DEBUG : Bird99_R1_001.fastq.gz: dropbox = 13dae534b00a0540024d2ef933bcdb1ca9d14e88b151873ecb75358f3d4cdf8d OK
2023-10-11 01:34:58 INFO  : Bird99_R1_001.fastq.gz: Copied (new)
2023-10-11 01:34:58 INFO  : Bird99_R1_001.fastq.gz: Deleted
2023-10-11 01:35:04 DEBUG : Bird99_R2_001.fastq.gz: Uploading chunk 14/14
2023-10-11 01:35:06 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Adding "20231010_BarnOwl3RAD/Bird99_R2_001.fastq.gz" to batch
2023-10-11 01:35:06 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Batch idle for 500ms so committing
2023-10-11 01:35:06 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Committing sync batch length 1 starting with: 20231010_BarnOwl3RAD/Bird99_R2_001.fastq.gz
2023-10-11 01:35:07 DEBUG : Dropbox root '20231010_BarnOwl3RAD': Committed sync batch length 1 starting with: 20231010_BarnOwl3RAD/Bird99_R2_001.fastq.gz
2023-10-11 01:35:07 DEBUG : Bird99_R2_001.fastq.gz: dropbox = 49f5388ef703e12e6b8df278951e8bfba491aca42074ab9a487c97eacde8b995 OK
2023-10-11 01:35:07 INFO  : Bird99_R2_001.fastq.gz: Copied (new)
2023-10-11 01:35:07 INFO  : Bird99_R2_001.fastq.gz: Deleted
Transferred:       59.936 GiB / 59.936 GiB, 100%, 3.632 MiB/s, ETA 0s
Checks:               594 / 594, 100%
Deleted:              297 (files), 0 (dirs)
Renamed:              297
Transferred:          297 / 297, 100%
Elapsed time:   5h23m14.9s
2023/10/11 01:35:07 INFO  :
Transferred:       59.936 GiB / 59.936 GiB, 100%, 3.632 MiB/s, ETA 0s
Checks:               594 / 594, 100%
Deleted:              297 (files), 0 (dirs)
Renamed:              297
Transferred:          297 / 297, 100%
Elapsed time:   5h23m14.9s

2023/10/11 01:35:07 DEBUG : 12 go routines active
2023/10/11 01:35:07 INFO  : Dropbox root '20231010_BarnOwl3RAD': Committing uploads - please wait...

As you can see, the upload speed stayed around 3.6 MiB/s the whole time, and the 56 GiB upload took 5 hours.

(I also changed the file paths in the example and debug code to try to make it more anonymous. This shouldn't affect the issue, but apologies if some of the paths aren't the same. The upload completed without a problem and all the files are in the right place.)

Animosity022 · October 11, 2023, 1:01pm

Create your own client ID and secret as documented:

and for single file transfers, increase the chunk size:

Anything around 128-150ish seems to be a sweet spot for me. You'd have to test and see what works for you.

alexkrohn · October 11, 2023, 1:31pm

Hmm. I wasn't able to create my own client ID, I followed the instructions and copied the App key and App secret keys to the client_id and client_secret lines, but got this error:

rclone authorize "dropbox" "<unique key here>"
2023/10/11 09:26:10 NOTICE: Make sure your Redirect URL is set to "http://localhost:53682/" in your custom config.
2023/10/11 09:26:10 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=xerca0UwgeZB2jbaigEtDQ
2023/10/11 09:26:10 NOTICE: Log in and authorize rclone for access
2023/10/11 09:26:10 NOTICE: Waiting for code...
2023/10/11 09:26:14 NOTICE: Got code
Error: failed to get token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error": "invalid_client: Invalid client_id or client_secret"}
Usage:
  rclone authorize [flags]

Flags:
      --auth-no-open-browser   Do not automatically open auth link in default browser
  -h, --help                   help for authorize
      --template string        The path to a custom Go template for generating HTML responses

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2023/10/11 09:26:15 Fatal error: failed to get token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error": "invalid_client: Invalid client_id or client_secret"}

I can increase the chunk size for single file transfers, but I usually do multiple files at once. Any advice there?

Animosity022 · October 11, 2023, 1:33pm

Not sure as you didn't follow something. When you configure it, you to use rclone config. Perhaps try creating a brand new remote and if that doesn't work, you'd really have to detail your steps as the docs work.

Test and see what works for your setup. The default transfers is 4 at a time.

alexkrohn · October 11, 2023, 6:47pm

No dice. I was able to login with the client_id/secret, and try increasing chunk sizes between 128 and 150. Increasing the chunk sizes slowed down the transfer rate to < 1 MiB/s. Copying via the client_id/secret had no effect -- the transfer rates remained around 3-4 MiB/sec.

I could also confirm that downloads were still very fast from dropbox using the client_id/secret authentication.

Animosity022 · October 11, 2023, 6:49pm

You'd want to share any changes you've made and a log file.

The latest config, command, debug log, etc.

alexkrohn · October 11, 2023, 8:54pm

OK. Is this what you'd like?

With the client-id enabled remote was 16m at 3.1 MiB/sec

rclone copy -Pv -vv ./ dropbox-secret:/test
2023/10/11 19:01:49 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "-Pv" "-vv" "./" "dropbox-secret:/test"]
2023/10/11 19:01:49 DEBUG : Creating backend with remote "./"
2023/10/11 19:01:49 DEBUG : Using config file from "/home/tangled/.config/rclone/rclone.conf"
2023/10/11 19:01:49 DEBUG : fs cache: renaming cache item "./" to be canonical "/home/tangled/oct2023-sequencing-30-811641238/test-upload"
2023/10/11 19:01:49 DEBUG : Creating backend with remote "dropbox-secret:/test"
2023/10/11 19:01:50 DEBUG : Dropbox root '': Using root namespace "2971962001"
2023/10/11 19:01:50 DEBUG : fs cache: renaming cache item "dropbox-secret:/test" to be canonical "dropbox-secret:test"
2023-10-11 19:01:50 DEBUG : LISE0053_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-11 19:01:50 DEBUG : LISE0053_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-11 19:01:50 DEBUG : LISE0054_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-11 19:01:50 DEBUG : LISE0054_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-11 19:01:50 DEBUG : LISE0055_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-11 19:01:50 DEBUG : LISE0055_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-11 19:01:50 DEBUG : LISE0056_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-11 19:01:50 DEBUG : LISE0056_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023-10-11 19:01:50 DEBUG : Dropbox root 'test': Waiting for checks to finish
2023-10-11 19:01:50 DEBUG : Dropbox root 'test': Waiting for transfers to finish
2023-10-11 19:01:51 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 1/12
2023-10-11 19:01:51 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 1/11
2023-10-11 19:01:51 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 1/12
2023-10-11 19:01:51 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 1/12
2023-10-11 19:02:32 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 2/12
2023-10-11 19:02:33 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 2/12
2023-10-11 19:02:35 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 2/11
2023-10-11 19:02:36 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 2/12
2023-10-11 19:03:14 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 3/12
2023-10-11 19:03:17 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 3/12
2023-10-11 19:03:23 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 3/11
2023-10-11 19:03:25 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 3/12
2023-10-11 19:03:57 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 4/12
2023-10-11 19:03:59 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 4/12
2023-10-11 19:04:09 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 4/12
2023-10-11 19:04:13 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 4/11
2023-10-11 19:04:39 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 5/12
2023-10-11 19:04:43 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 5/12
2023-10-11 19:04:53 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 5/12
2023-10-11 19:04:55 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 5/11
2023-10-11 19:05:23 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 6/12
2023-10-11 19:05:23 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 6/12
2023-10-11 19:05:38 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 6/12
2023-10-11 19:05:46 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 6/11
2023-10-11 19:06:07 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 7/12
2023-10-11 19:06:10 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 7/12
2023-10-11 19:06:18 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 7/12
2023-10-11 19:06:34 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 7/11
2023-10-11 19:06:49 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 8/12
2023-10-11 19:06:58 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 8/12
2023-10-11 19:07:01 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 8/12
2023-10-11 19:07:16 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 8/11
2023-10-11 19:07:37 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 9/12
2023-10-11 19:07:43 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 9/12
2023-10-11 19:07:45 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 9/12
2023-10-11 19:08:00 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 9/11
2023-10-11 19:08:14 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 10/12
2023-10-11 19:08:20 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 10/12
2023-10-11 19:08:38 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 10/12
2023-10-11 19:08:43 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 10/11
2023-10-11 19:09:04 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 11/12
2023-10-11 19:09:04 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 11/12
2023-10-11 19:09:20 DEBUG : LISE0054_R2_001.fastq.gz: Uploading chunk 11/11
2023-10-11 19:09:29 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 11/12
2023-10-11 19:09:45 DEBUG : LISE0054_R1_001.fastq.gz: Uploading chunk 12/12
2023-10-11 19:09:47 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 12/12
2023-10-11 19:09:50 DEBUG : Dropbox root 'test': Adding "/test/LISE0054_R1_001.fastq.gz" to batch
2023-10-11 19:09:51 DEBUG : Dropbox root 'test': Batch idle for 500ms so committing
2023-10-11 19:09:51 DEBUG : Dropbox root 'test': Committing sync batch length 1 starting with: /test/LISE0054_R1_001.fastq.gz
2023-10-11 19:09:52 DEBUG : Dropbox root 'test': Committed sync batch length 1 starting with: /test/LISE0054_R1_001.fastq.gz
2023-10-11 19:09:52 DEBUG : LISE0054_R1_001.fastq.gz: dropbox = ba9cc1389180f3bcacf7b73c7526ce5c893815e6f6631c945acd6ba5fafebf5e OK
2023-10-11 19:09:52 INFO  : LISE0054_R1_001.fastq.gz: Copied (new)
2023-10-11 19:09:52 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 1/12
2023-10-11 19:10:06 DEBUG : Dropbox root 'test': Adding "/test/LISE0054_R2_001.fastq.gz" to batch
2023-10-11 19:10:06 DEBUG : Dropbox root 'test': Adding "/test/LISE0053_R2_001.fastq.gz" to batch
2023-10-11 19:10:07 DEBUG : Dropbox root 'test': Batch idle for 500ms so committing
2023-10-11 19:10:07 DEBUG : Dropbox root 'test': Committing sync batch length 2 starting with: /test/LISE0054_R2_001.fastq.gz
2023-10-11 19:10:08 DEBUG : Dropbox root 'test': Committed sync batch length 2 starting with: /test/LISE0054_R2_001.fastq.gz
2023-10-11 19:10:08 DEBUG : LISE0054_R2_001.fastq.gz: dropbox = 09ddd70e2968e05cf7da223d9e39de721b6a5b8be3d163be626018ee2261408f OK
2023-10-11 19:10:08 INFO  : LISE0054_R2_001.fastq.gz: Copied (new)
2023-10-11 19:10:08 DEBUG : LISE0053_R2_001.fastq.gz: dropbox = cc0c101c36a651ca215ab7ad2e5183c3d6be47794bb1234183c1f794306deae0 OK
2023-10-11 19:10:08 INFO  : LISE0053_R2_001.fastq.gz: Copied (new)
2023-10-11 19:10:08 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 1/12
2023-10-11 19:10:08 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 1/12
2023-10-11 19:10:10 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 12/12
2023-10-11 19:10:32 DEBUG : Dropbox root 'test': Adding "/test/LISE0053_R1_001.fastq.gz" to batch
2023-10-11 19:10:33 DEBUG : Dropbox root 'test': Batch idle for 500ms so committing
2023-10-11 19:10:33 DEBUG : Dropbox root 'test': Committing sync batch length 1 starting with: /test/LISE0053_R1_001.fastq.gz
2023-10-11 19:10:34 DEBUG : Dropbox root 'test': Committed sync batch length 1 starting with: /test/LISE0053_R1_001.fastq.gz
2023-10-11 19:10:34 DEBUG : LISE0053_R1_001.fastq.gz: dropbox = ba4d7944d3b1269505e39939fdddc9b6a8561682766a1a2e1d76d63f0e6af5ab OK
2023-10-11 19:10:34 INFO  : LISE0053_R1_001.fastq.gz: Copied (new)
2023-10-11 19:10:34 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 1/12
2023-10-11 19:10:35 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 2/12
2023-10-11 19:10:48 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 2/12
2023-10-11 19:10:54 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 2/12
2023-10-11 19:11:19 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 3/12
2023-10-11 19:11:20 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 2/12
2023-10-11 19:11:32 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 3/12
2023-10-11 19:11:35 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 3/12
2023-10-11 19:12:02 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 4/12
2023-10-11 19:12:09 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 4/12
2023-10-11 19:12:13 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 3/12
2023-10-11 19:12:20 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 4/12
2023-10-11 19:12:49 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 5/12
2023-10-11 19:12:50 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 5/12
2023-10-11 19:13:06 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 5/12
2023-10-11 19:13:06 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 4/12
2023-10-11 19:13:28 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 6/12
2023-10-11 19:13:32 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 6/12
2023-10-11 19:13:54 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 6/12
2023-10-11 19:14:00 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 5/12
2023-10-11 19:14:05 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 7/12
2023-10-11 19:14:11 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 7/12
2023-10-11 19:14:41 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 7/12
2023-10-11 19:14:49 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 8/12
2023-10-11 19:14:49 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 8/12
2023-10-11 19:14:50 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 6/12
2023-10-11 19:15:27 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 8/12
2023-10-11 19:15:31 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 7/12
2023-10-11 19:15:31 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 9/12
2023-10-11 19:15:31 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 9/12
2023-10-11 19:16:08 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 10/12
2023-10-11 19:16:14 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 8/12
2023-10-11 19:16:17 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 10/12
2023-10-11 19:16:21 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 9/12
2023-10-11 19:16:50 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 11/12
2023-10-11 19:16:58 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 11/12
2023-10-11 19:17:02 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 9/12
2023-10-11 19:17:06 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 10/12
2023-10-11 19:17:21 DEBUG : LISE0056_R1_001.fastq.gz: Uploading chunk 12/12
2023-10-11 19:17:37 DEBUG : Dropbox root 'test': Adding "/test/LISE0056_R1_001.fastq.gz" to batch
2023-10-11 19:17:37 DEBUG : Dropbox root 'test': Batch idle for 500ms so committing
2023-10-11 19:17:37 DEBUG : Dropbox root 'test': Committing sync batch length 1 starting with: /test/LISE0056_R1_001.fastq.gz
2023-10-11 19:17:38 DEBUG : Dropbox root 'test': Committed sync batch length 1 starting with: /test/LISE0056_R1_001.fastq.gz
2023-10-11 19:17:38 DEBUG : LISE0056_R1_001.fastq.gz: dropbox = 233409999d068a8d14a13142c8a3d146b6a818dc6848fad1bbc95027508e4fe1 OK
2023-10-11 19:17:38 INFO  : LISE0056_R1_001.fastq.gz: Copied (new)
2023-10-11 19:17:40 DEBUG : LISE0055_R1_001.fastq.gz: Uploading chunk 12/12
2023-10-11 19:17:47 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 10/12
2023-10-11 19:17:49 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 11/12
2023-10-11 19:18:10 DEBUG : Dropbox root 'test': Adding "/test/LISE0055_R1_001.fastq.gz" to batch
2023-10-11 19:18:11 DEBUG : Dropbox root 'test': Batch idle for 500ms so committing
2023-10-11 19:18:11 DEBUG : Dropbox root 'test': Committing sync batch length 1 starting with: /test/LISE0055_R1_001.fastq.gz
2023-10-11 19:18:11 DEBUG : Dropbox root 'test': Committed sync batch length 1 starting with: /test/LISE0055_R1_001.fastq.gz
2023-10-11 19:18:11 DEBUG : LISE0055_R1_001.fastq.gz: dropbox = d3917048c3790d51bdff6d8caa0c13910a2916b92134c3f604c36a97f64516b4 OK
2023-10-11 19:18:11 INFO  : LISE0055_R1_001.fastq.gz: Copied (new)
2023-10-11 19:18:17 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 11/12
2023-10-11 19:18:18 DEBUG : LISE0055_R2_001.fastq.gz: Uploading chunk 12/12
2023-10-11 19:18:37 DEBUG : Dropbox root 'test': Adding "/test/LISE0055_R2_001.fastq.gz" to batch
2023-10-11 19:18:37 DEBUG : Dropbox root 'test': Batch idle for 500ms so committing
2023-10-11 19:18:37 DEBUG : Dropbox root 'test': Committing sync batch length 1 starting with: /test/LISE0055_R2_001.fastq.gz
2023-10-11 19:18:38 DEBUG : Dropbox root 'test': Committed sync batch length 1 starting with: /test/LISE0055_R2_001.fastq.gz
2023-10-11 19:18:38 DEBUG : LISE0055_R2_001.fastq.gz: dropbox = e784c1ccb5c0d977a2c86e57c3b4f48ded87963b387fee53e138d0ae18c3832a OK
2023-10-11 19:18:38 INFO  : LISE0055_R2_001.fastq.gz: Copied (new)
2023-10-11 19:18:40 DEBUG : LISE0056_R2_001.fastq.gz: Uploading chunk 12/12
2023-10-11 19:18:46 DEBUG : Dropbox root 'test': Adding "/test/LISE0056_R2_001.fastq.gz" to batch
2023-10-11 19:18:47 DEBUG : Dropbox root 'test': Batch idle for 500ms so committing
2023-10-11 19:18:47 DEBUG : Dropbox root 'test': Committing sync batch length 1 starting with: /test/LISE0056_R2_001.fastq.gz
2023-10-11 19:18:48 DEBUG : Dropbox root 'test': Committed sync batch length 1 starting with: /test/LISE0056_R2_001.fastq.gz
2023-10-11 19:18:48 DEBUG : LISE0056_R2_001.fastq.gz: dropbox = c013e0f242a52b4f733574295b337b17a859471791aa29f2f6c8c0009b7f24e2 OK
2023-10-11 19:18:48 INFO  : LISE0056_R2_001.fastq.gz: Copied (new)
Transferred:   	    4.289 GiB / 4.289 GiB, 100%, 3.150 MiB/s, ETA 0s
Transferred:            8 / 8, 100%
Elapsed time:     16m58.4s
2023/10/11 19:18:48 INFO  : 
Transferred:   	    4.289 GiB / 4.289 GiB, 100%, 3.150 MiB/s, ETA 0s
Transferred:            8 / 8, 100%
Elapsed time:     16m58.4s

2023/10/11 19:18:48 DEBUG : 14 go routines active
2023/10/11 19:18:48 INFO  : Dropbox root 'test': Committing uploads - please wait...

With chunk size of 40 or 128 it gets slowed way down (I stopped the log early because this 4 GB transfer would take 1.5 hours).

2023/10/11 20:52:21 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "-Pv" "-vv" "./" "dropbox-secret:/test" "--dropbox-chunk-size" "40" "--log-file=dropbox-secret-with-chunk4"]
2023/10/11 20:52:21 DEBUG : Creating backend with remote "./"
2023/10/11 20:52:21 DEBUG : Using config file from "/home/tangled/.config/rclone/rclone.conf"
2023/10/11 20:52:21 DEBUG : fs cache: renaming cache item "./" to be canonical "/home/tangled/oct2023-sequencing-30-811641238/test-upload"
2023/10/11 20:52:21 DEBUG : Creating backend with remote "dropbox-secret:/test"
2023/10/11 20:52:21 DEBUG : dropbox-secret: detected overridden config - adding "{l7syH}" suffix to name
2023/10/11 20:52:22 DEBUG : Dropbox root '': Using root namespace "2971962001"
2023/10/11 20:52:22 DEBUG : fs cache: renaming cache item "dropbox-secret:/test" to be canonical "dropbox-secret{l7syH}:test"
2023/10/11 20:52:23 DEBUG : dropbox-secret-with-128chunks.txt: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : dropbox-secret-with-chunk4: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : LISE0053_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : LISE0053_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : LISE0054_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : LISE0054_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : LISE0055_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : LISE0055_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : LISE0056_R1_001.fastq.gz: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : LISE0056_R2_001.fastq.gz: Need to transfer - File not found at Destination
2023/10/11 20:52:23 DEBUG : Dropbox root 'test': Waiting for checks to finish
2023/10/11 20:52:23 DEBUG : Dropbox root 'test': Waiting for transfers to finish
2023/10/11 20:52:23 DEBUG : dropbox-secret-with-128chunks.txt: Uploading chunk 1/28
2023/10/11 20:52:23 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 1/14219
2023/10/11 20:52:23 DEBUG : dropbox-secret-with-chunk4: Uploading chunk 1/1
2023/10/11 20:52:23 DEBUG : pacer: low level retry 1/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:23 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2023/10/11 20:52:23 DEBUG : pacer: Reducing sleep to 15ms
2023/10/11 20:52:23 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 1/14110
2023/10/11 20:52:23 DEBUG : pacer: low level retry 2/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:23 DEBUG : pacer: Rate limited, increasing sleep to 30ms
2023/10/11 20:52:23 DEBUG : pacer: low level retry 3/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:23 DEBUG : pacer: Rate limited, increasing sleep to 60ms
2023/10/11 20:52:23 DEBUG : pacer: low level retry 4/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:23 DEBUG : pacer: Rate limited, increasing sleep to 120ms
2023/10/11 20:52:23 DEBUG : pacer: low level retry 5/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:23 DEBUG : pacer: Rate limited, increasing sleep to 240ms
2023/10/11 20:52:23 DEBUG : pacer: low level retry 6/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:23 DEBUG : pacer: Rate limited, increasing sleep to 480ms
2023/10/11 20:52:23 DEBUG : pacer: Reducing sleep to 360ms
2023/10/11 20:52:23 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 2/14219
2023/10/11 20:52:23 DEBUG : pacer: Reducing sleep to 270ms
2023/10/11 20:52:23 DEBUG : dropbox-secret-with-128chunks.txt: Uploading chunk 2/28
2023/10/11 20:52:24 DEBUG : pacer: Reducing sleep to 202.5ms
2023/10/11 20:52:24 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 2/14110
2023/10/11 20:52:24 DEBUG : pacer: low level retry 7/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:24 DEBUG : pacer: Rate limited, increasing sleep to 405ms
2023/10/11 20:52:24 DEBUG : pacer: Reducing sleep to 303.75ms
2023/10/11 20:52:24 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 3/14219
2023/10/11 20:52:25 DEBUG : pacer: Reducing sleep to 227.8125ms
2023/10/11 20:52:25 DEBUG : dropbox-secret-with-128chunks.txt: Uploading chunk 3/28
2023/10/11 20:52:25 DEBUG : pacer: low level retry 8/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:25 DEBUG : pacer: Rate limited, increasing sleep to 455.625ms
2023/10/11 20:52:25 DEBUG : pacer: Reducing sleep to 341.71875ms
2023/10/11 20:52:25 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 3/14110
2023/10/11 20:52:26 DEBUG : pacer: Reducing sleep to 256.289062ms
2023/10/11 20:52:26 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 4/14219
2023/10/11 20:52:26 DEBUG : pacer: low level retry 9/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:26 DEBUG : pacer: Rate limited, increasing sleep to 512.578124ms
2023/10/11 20:52:26 DEBUG : pacer: Reducing sleep to 384.433593ms
2023/10/11 20:52:26 DEBUG : dropbox-secret-with-128chunks.txt: Uploading chunk 4/28
2023/10/11 20:52:27 DEBUG : pacer: Reducing sleep to 288.325194ms
2023/10/11 20:52:27 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 5/14219
2023/10/11 20:52:27 DEBUG : pacer: low level retry 10/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328))
2023/10/11 20:52:27 DEBUG : pacer: Rate limited, increasing sleep to 576.650388ms
2023/10/11 20:52:27 DEBUG : dropbox-secret-with-chunk4: Received error: upload failed: Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 2328) - low level retry 1/10
2023/10/11 20:52:28 DEBUG : pacer: Reducing sleep to 432.487791ms
2023/10/11 20:52:28 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 4/14110
2023/10/11 20:52:28 DEBUG : pacer: Reducing sleep to 324.365843ms
2023/10/11 20:52:28 DEBUG : dropbox-secret-with-128chunks.txt: Uploading chunk 5/28
2023/10/11 20:52:29 DEBUG : pacer: Reducing sleep to 243.274382ms
2023/10/11 20:52:29 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 6/14219
2023/10/11 20:52:29 DEBUG : pacer: Reducing sleep to 182.455786ms
2023/10/11 20:52:29 DEBUG : dropbox-secret-with-chunk4: Uploading chunk 1/1
2023/10/11 20:52:29 DEBUG : pacer: Reducing sleep to 136.841839ms
2023/10/11 20:52:29 DEBUG : LISE0053_R2_001.fastq.gz: Uploading chunk 5/14110
2023/10/11 20:52:29 DEBUG : pacer: low level retry 1/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 7598))
2023/10/11 20:52:29 DEBUG : pacer: Rate limited, increasing sleep to 273.683678ms
2023/10/11 20:52:30 DEBUG : pacer: Reducing sleep to 205.262758ms
2023/10/11 20:52:30 DEBUG : LISE0053_R1_001.fastq.gz: Uploading chunk 7/14219
2023/10/11 20:52:30 DEBUG : pacer: low level retry 2/10 (error Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 890 to 7598))
2023/10/11 20:52:30 DEBUG : pacer: Rate limited, increasing sleep to 410.525516ms
2023/10/11 20:52:30 DEBUG : pacer: Reducing sleep to 307.894137ms

Transferred:   	      960 KiB / 4.290 GiB, 0%, 76.658 KiB/s, ETA 16h17m53s
Transferred:            0 / 10, 0%
Elapsed time:        14.0s
Transferring:
 *             dropbox-secret-with-128chunks.txt: 25% /1.071Mi, 23.333Ki/s, 35s
 *                    dropbox-secret-with-chunk4:  0% /890, 0/s, -
 *                      LISE0053_R1_001.fastq.gz:  0% /555.392Mi, 29.995Ki/s, 5h15m48s
 *                      LISE0053_R2_001.fastq.gz:  0% /551.162Mi, 23.331Ki/s, 6h42m56s

Similar behavior occurs with chunk sizes in the non-client_id/secret remote.

Any other suggestions?

Animosity022 · October 11, 2023, 9:06pm

Did you update your config file with the client ID/secret?

Animosity022 · October 11, 2023, 9:11pm

I generally get ~30MB/s on a single file transfer.

[felix@gemini tmp]$ fallocate -l 10G testfile
[felix@gemini tmp]$ rclone copy testfile DB: -P -v
Transferred:   	    2.175 GiB / 10 GiB, 22%, 33.365 MiB/s, ETA 4m
Transferred:            0 / 1, 0%
Elapsed time:      1m18.0s
Transferring:
 *                                      testfile: 21% /10Gi, 33.368Mi/s, 4m0s

[felix@gemini tmp]$ rclone config redacted DB:
[DB]
type = dropbox
client_id = XXX
client_secret = XXX
token = XXX
chunk_size = 64Mi
server_side_across_configs = true
### Double check the config for sensitive info before posting publicly

I have generally good peering to dropbox and gigabit internet / network guy so I can be sure I've got no other traffic going on for my testing.

Doing 4 transfers at a time, I can max out my gigabit.

alexkrohn · October 12, 2023, 12:56pm

Yeah, this is with the client ID/secret setup:

rclone config redacted dropbox-secret
[dropbox-secret]
type = dropbox
client_id = XXX
client_secret = XXX
token = XXX
### Double check the config for sensitive info before posting publicly

Hmm. So if you're getting 30 MB/s without a problem on uploads, maybe it is something with my ISP, region, or dropbox account configuration?

I should note that uploads to AWS go much faster from my local machine, and transfers from AWS to Dropbox go much faster. I'll look into account settings in Dropbox. Very odd!

Animosity022 · October 12, 2023, 12:59pm

If all those settings are good, there is nothing in the debug logs that show a slow down so we are left with the unfortunate 'internet' in between.

You could have bad peering for that upload or something else going on. I see no issues with any setting you have.

There isn't a Dropbox account setting that impacts this to my knowledge.

system · October 15, 2023, 1:00pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.