DROPBOX reset upload with large files

saqqara · March 15, 2021, 10:38am

What is the problem you are having with rclone?

When i upload large files, when the file is uploaded, rclone starts from zero to upload again. I Know there is a size limitation. 350 GB Max. When i upload a 100 GB file, it is ok. But when the file is bigger than 150 GB, it never stop, always start to upload from zero.

What is your rclone version (output from `rclone version`)

rclone v1.54.1

os/arch: linux/amd64
go version: go1.15.8

Which OS you are using and how many bits (eg Windows 7, 64 bit)Distributor ID: Debian

Description: Debian GNU/Linux 9.9 (stretch)
Release: 9.9
Codename: stretch

Which cloud storage system are you using? (eg Google Drive)

DROPBOX

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

i have tried several, the log that appears below belongs to this instruction:

rclone sync vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo dropbox:SERVIDORES/CLIENTES17/LACADOS -vv --dump responses --transfers=8 --drive-chunk-size=33554432 --log-file=./rclone.log

The rclone config contents with secrets removed.

[dropbox]
type = dropbox
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}

A log from the command with the `-vv` flag

this is the end of the file upload and reset

2021/03/14 14:31:07 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:31:07 DEBUG : vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo: Uploading chunk 4684/4685
2021/03/14 14:31:07 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:31:07 DEBUG : HTTP REQUEST (req 0xc004fb2a00)
2021/03/14 14:31:07 DEBUG : POST /2/files/upload_session/append_v2 HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235703107584},"close":false}
Accept-Encoding: gzip

2021/03/14 14:31:07 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:31:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:31:09 DEBUG : HTTP RESPONSE (req 0xc004fb2a00)
2021/03/14 14:31:09 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:31:09 GMT
Pragma: no-cache
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 75ea59c40e3441d882d763b2e4b77afe
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex
X-Server-Response-Time: 1503

4
null
0

2021/03/14 14:31:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:31:09 DEBUG : vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo: Uploading chunk 4685/4685
2021/03/14 14:31:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:31:09 DEBUG : HTTP REQUEST (req 0xc000193f00)
2021/03/14 14:31:09 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:31:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:31:14 INFO  : 
Transferred:   	  219.598G / 219.598 GBytes, 100%, 21.664 MBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:    2h53m0.3s
Transferring:
 *   vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo:100% /219.598G, 17.021M/s, 0s

2021/03/14 14:32:14 INFO  : 
Transferred:   	  219.598G / 219.598 GBytes, 100%, 21.540 MBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:    2h54m0.3s
Transferring:
 *   vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo:100% /219.598G, 362.717k/s, 0s

2021/03/14 14:32:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:41 DEBUG : HTTP RESPONSE (req 0xc000193f00)
2021/03/14 14:32:41 DEBUG : HTTP/1.1 429 Too Many Requests
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:41 GMT
Retry-After: 15
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 509f0c92ccb740298ae511833162cf76
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

70
{"error_summary": "too_many_requests/..", "error": {"reason": {".tag": "too_many_requests"}, "retry_after": 15}}
0

2021/03/14 14:32:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:41 DEBUG : pacer: low level retry 1/10 (error too_many_requests/..)
2021/03/14 14:32:41 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021/03/14 14:32:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:41 DEBUG : HTTP REQUEST (req 0xc0006d4900)
2021/03/14 14:32:41 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:42 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:42 DEBUG : HTTP RESPONSE (req 0xc0006d4900)
2021/03/14 14:32:42 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:42 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: f408b218e0294f7188ce2efd68e22ae6
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

aa
{"error_summary": "lookup_failed/incorrect_offset/...", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:42 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:42 DEBUG : pacer: low level retry 2/10 (error lookup_failed/incorrect_offset/...)
2021/03/14 14:32:42 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2021/03/14 14:32:42 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:42 DEBUG : HTTP REQUEST (req 0xc0006d4d00)
2021/03/14 14:32:42 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:42 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:43 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:43 DEBUG : HTTP RESPONSE (req 0xc0006d4d00)
2021/03/14 14:32:43 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:43 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 78c2e704b8c4415784613ea68d3753c0
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

aa
{"error_summary": "lookup_failed/incorrect_offset/...", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:43 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:43 DEBUG : pacer: low level retry 3/10 (error lookup_failed/incorrect_offset/...)
2021/03/14 14:32:43 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2021/03/14 14:32:43 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:43 DEBUG : HTTP REQUEST (req 0xc004fb2e00)
2021/03/14 14:32:43 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:43 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:45 DEBUG : HTTP RESPONSE (req 0xc004fb2e00)
2021/03/14 14:32:45 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:45 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 31e10d39849b43ed97f41483ef4cff02
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

a9
{"error_summary": "lookup_failed/incorrect_offset/..", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:45 DEBUG : pacer: low level retry 4/10 (error lookup_failed/incorrect_offset/..)
2021/03/14 14:32:45 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2021/03/14 14:32:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:45 DEBUG : HTTP REQUEST (req 0xc0004d8600)
2021/03/14 14:32:45 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:46 DEBUG : HTTP RESPONSE (req 0xc0004d8600)
2021/03/14 14:32:46 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:46 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 6b231073b7654a709c563974e8c1db6b
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

a7
{"error_summary": "lookup_failed/incorrect_offset/", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:46 DEBUG : pacer: low level retry 5/10 (error lookup_failed/incorrect_offset/)
2021/03/14 14:32:46 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2021/03/14 14:32:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:46 DEBUG : HTTP REQUEST (req 0xc0004d8c00)
2021/03/14 14:32:46 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:47 DEBUG : HTTP RESPONSE (req 0xc0004d8c00)
2021/03/14 14:32:47 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:47 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 128cda1fad3d47008abceb9cce0fbc10
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

a9
{"error_summary": "lookup_failed/incorrect_offset/..", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:47 DEBUG : pacer: low level retry 6/10 (error lookup_failed/incorrect_offset/..)
2021/03/14 14:32:47 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2021/03/14 14:32:47 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:47 DEBUG : HTTP REQUEST (req 0xc0051f0100)
2021/03/14 14:32:47 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:47 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:48 DEBUG : HTTP RESPONSE (req 0xc0051f0100)
2021/03/14 14:32:48 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:48 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: ef410e9bab444419acc697929d363238
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

a7
{"error_summary": "lookup_failed/incorrect_offset/", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:48 DEBUG : pacer: low level retry 7/10 (error lookup_failed/incorrect_offset/)
2021/03/14 14:32:48 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2021/03/14 14:32:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:48 DEBUG : HTTP REQUEST (req 0xc004fb3700)
2021/03/14 14:32:48 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:49 DEBUG : HTTP RESPONSE (req 0xc004fb3700)
2021/03/14 14:32:49 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:49 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 09fcc156fd054f55a28098e0818b0e5b
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

a8
{"error_summary": "lookup_failed/incorrect_offset/.", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:49 DEBUG : pacer: low level retry 8/10 (error lookup_failed/incorrect_offset/.)
2021/03/14 14:32:49 DEBUG : pacer: Rate limited, increasing sleep to 2s
2021/03/14 14:32:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:49 DEBUG : HTTP REQUEST (req 0xc004fb3b00)
2021/03/14 14:32:49 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:50 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:50 DEBUG : HTTP RESPONSE (req 0xc004fb3b00)
2021/03/14 14:32:50 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:50 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 2e2e610a31bf4c89bc5cdb59b16d58db
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

a7
{"error_summary": "lookup_failed/incorrect_offset/", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:50 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:50 DEBUG : pacer: low level retry 9/10 (error lookup_failed/incorrect_offset/)
2021/03/14 14:32:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:51 DEBUG : HTTP REQUEST (req 0xc0051f0900)
2021/03/14 14:32:51 DEBUG : POST /2/files/upload_session/finish HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9KGdGGHYr65Vg","offset":235753439232},"commit":{"path":"/SERVIDORES/CLIENTES17/LACADOS/vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo","mode":{".tag":"overwrite"},"autorename":false,"client_modified":"2021-03-12T23:39:21Z","mute":false,"strict_conflict":false}}
Accept-Encoding: gzip

2021/03/14 14:32:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:52 DEBUG : HTTP RESPONSE (req 0xc0051f0900)
2021/03/14 14:32:52 DEBUG : HTTP/1.1 409 Conflict
Transfer-Encoding: chunked
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:52 GMT
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 0bbacbbe003b438380c0982ddbcb3873
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex

a9
{"error_summary": "lookup_failed/incorrect_offset/..", "error": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 235791405904}}}
0

2021/03/14 14:32:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:52 DEBUG : pacer: low level retry 10/10 (error lookup_failed/incorrect_offset/..)
2021/03/14 14:32:52 DEBUG : vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo: Received error: upload failed: lookup_failed/incorrect_offset/.. - low level retry 1/10
2021/03/14 14:32:52 DEBUG : vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo: Uploading chunk 1/4685
2021/03/14 14:32:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:53 DEBUG : HTTP REQUEST (req 0xc0051f0100)
2021/03/14 14:32:53 DEBUG : POST /2/files/upload_session/start HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"close":false}
Accept-Encoding: gzip

2021/03/14 14:32:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:55 DEBUG : HTTP RESPONSE (req 0xc0051f0100)
2021/03/14 14:32:55 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:55 GMT
Pragma: no-cache
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 40213450410c43189f728a99706b3e46
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex
X-Server-Response-Time: 1275

28
{"session_id": "AAAAAAAAA9XfeeZBK8UgeQ"}
0

2021/03/14 14:32:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:55 DEBUG : pacer: Reducing sleep to 1.5s
2021/03/14 14:32:55 DEBUG : vzdump-qemu-160-2021_03_13-00_10_01.vma.lzo: Uploading chunk 2/4685
2021/03/14 14:32:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:55 DEBUG : HTTP REQUEST (req 0xc0051f0700)
2021/03/14 14:32:55 DEBUG : POST /2/files/upload_session/append_v2 HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.54.1
Transfer-Encoding: chunked
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"cursor":{"session_id":"AAAAAAAAA9XfeeZBK8UgeQ","offset":50331648},"close":false}
Accept-Encoding: gzip

2021/03/14 14:32:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/14 14:32:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/14 14:32:57 DEBUG : HTTP RESPONSE (req 0xc0051f0700)
2021/03/14 14:32:57 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache
Content-Type: application/json
Date: Sun, 14 Mar 2021 13:32:57 GMT
Pragma: no-cache
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Dropbox-Request-Id: 5daa58b2b06a4ce981a16ef8d0f25329
X-Dropbox-Response-Origin: remote
X-Robots-Tag: noindex, nofollow, noimageindex
X-Robots-Tag: noindex, nofollow, noimageindex
X-Server-Response-Time: 1436

4
null
0

ncw · March 16, 2021, 2:13pm

I think this will be fixed by upcoming improvements to the dropbox backend. In the mean time try setting --tpslimit 10 to see if that helps.

saqqara · March 16, 2021, 10:04pm

it worked! thanks a lot.

saqqara · March 18, 2021, 12:16pm

Hello, it only worked one time. it has not worked again. Any idea???

saqqara · March 19, 2021, 9:41am

If i mount dropbox storage with rclone, and use 'cp' to copy file, it works, in case this can help.

ncw · March 22, 2021, 9:54am

Can you try this beta?

v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync on branch fix-dropbox-batch-sync (uploaded in 15-30 mins)

saqqara · March 22, 2021, 6:42pm

Hi, seems to work. .
I'll keep doing tests with other files. Do you want to launch with some specific parameters?
Do you think i can remove
--dump responses --drive-chunk-size=33554432 ???

thanks!!!

I have launched it with this command:

rclone sync vzdump-qemu-402-2021_03_22-03_34_19.vma.lzo temp: -vv --dump responses --transfers=8 --drive-chunk-size=33554432 --log-file=./rclone.log

and this is the end of the log file

{".tag": "in_progress"}
2021/03/22 19:16:44 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:44 DEBUG : Dropbox root 'PRUEBA/BETA': Sleeping for 1s to wait for batch to complete: "in_progress": try 79/120
2021/03/22 19:16:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 19:16:45 DEBUG : HTTP REQUEST (req 0xc000c3fe00)
2021/03/22 19:16:45 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/22 19:16:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 19:16:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:45 DEBUG : HTTP RESPONSE (req 0xc000c3fe00)
2021/03/22 19:16:45 DEBUG : HTTP/2.0 200 OK
Content-Length: 23
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Mon, 22 Mar 2021 18:16:45 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 098ddc1aa4cd46b1a5ffc1bff9ca2ee9
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 164

{".tag": "in_progress"}
2021/03/22 19:16:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:45 DEBUG : Dropbox root 'PRUEBA/BETA': Sleeping for 1s to wait for batch to complete: "in_progress": try 80/120
2021/03/22 19:16:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 19:16:46 DEBUG : HTTP REQUEST (req 0xc000ca3100)
2021/03/22 19:16:46 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/22 19:16:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 19:16:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:47 DEBUG : HTTP RESPONSE (req 0xc000ca3100)
2021/03/22 19:16:47 DEBUG : HTTP/2.0 200 OK
Content-Length: 23
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Mon, 22 Mar 2021 18:16:46 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 46c79bc0d51a4382840f85be25312d54
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 286

{".tag": "in_progress"}
2021/03/22 19:16:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:47 DEBUG : Dropbox root 'PRUEBA/BETA': Sleeping for 1s to wait for batch to complete: "in_progress": try 81/120
2021/03/22 19:16:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 19:16:48 DEBUG : HTTP REQUEST (req 0xc000ca3600)
2021/03/22 19:16:48 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/22 19:16:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 19:16:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:48 DEBUG : HTTP RESPONSE (req 0xc000ca3600)
2021/03/22 19:16:48 DEBUG : HTTP/2.0 200 OK
Content-Length: 23
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Mon, 22 Mar 2021 18:16:47 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 7ebbbb41df5443f9b8f3deeacdf28057
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 107

{".tag": "in_progress"}
2021/03/22 19:16:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:48 DEBUG : Dropbox root 'PRUEBA/BETA': Sleeping for 1s to wait for batch to complete: "in_progress": try 82/120
2021/03/22 19:16:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 19:16:49 DEBUG : HTTP REQUEST (req 0xc000c72400)
2021/03/22 19:16:49 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/22 19:16:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/22 19:16:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:49 DEBUG : HTTP RESPONSE (req 0xc000c72400)
2021/03/22 19:16:49 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Mon, 22 Mar 2021 18:16:49 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 9dfc6c28a0814b8fa1e5f7e213ddad83
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 212

{".tag": "complete", "entries": [{".tag": "success", "name": "vzdump-qemu-402-2021_03_22-03_34_19.vma.lzo.bin", "path_lower": "/prueba/beta/vzdump-qemu-402-2021_03_22-03_34_19.vma.lzo.bin", "path_display": "/PRUEBA/BETA/vzdump-qemu-402-2021_03_22-03_34_19.vma.lzo.bin", "id": "id:SPPKUakybVwAAAAAAAA5fQ", "client_modified": "2021-03-22T03:20:42Z", "server_modified": "2021-03-22T18:16:47Z", "rev": "015be24119e0bf30000000228969a50", "size": 146932228494, "is_downloadable": true, "content_hash": "3905322ae681729d54453bafb05a9176350c4f6c3bcbbd88f005a601ec748cff"}]}
2021/03/22 19:16:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/22 19:16:49 INFO  : vzdump-qemu-402-2021_03_22-03_34_19.vma.lzo: Copied (new)
2021/03/22 19:16:49 INFO  : 
Transferred:   	  136.841G / 136.841 GBytes, 100%, 13.482 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:   2h53m14.1s

2021/03/22 19:16:49 DEBUG : 7 go routines active

saqqara · March 23, 2021, 11:49am

hello, ii have tried with other file, and it was on error.
here is the end of the log file.

{".tag": "in_progress"}
2021/03/23 11:04:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:19 DEBUG : Dropbox root 'LACADOSALACANT/MAQUINAS/1': Sleeping for 1s to wait for batch to complete: "in_progress": try 38/120
2021/03/23 11:04:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/23 11:04:20 DEBUG : HTTP REQUEST (req 0xc0008f2b00)
2021/03/23 11:04:20 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/23 11:04:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/23 11:04:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:20 DEBUG : HTTP RESPONSE (req 0xc0008f2b00)
2021/03/23 11:04:20 DEBUG : HTTP/2.0 200 OK
Content-Length: 23
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 23 Mar 2021 10:04:20 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 87c033292eb34172906662a45dab1040
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 61

{".tag": "in_progress"}
2021/03/23 11:04:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:20 DEBUG : Dropbox root 'LACADOSALACANT/MAQUINAS/1': Sleeping for 1s to wait for batch to complete: "in_progress": try 39/120
2021/03/23 11:04:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/23 11:04:21 DEBUG : HTTP REQUEST (req 0xc00085d800)
2021/03/23 11:04:21 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/23 11:04:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/23 11:04:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:21 DEBUG : HTTP RESPONSE (req 0xc00085d800)
2021/03/23 11:04:21 DEBUG : HTTP/2.0 200 OK
Content-Length: 23
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 23 Mar 2021 10:04:21 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 40f12a31f55a4040a756012117bcaf72
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 49

{".tag": "in_progress"}
2021/03/23 11:04:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:21 DEBUG : Dropbox root 'LACADOSALACANT/MAQUINAS/1': Sleeping for 1s to wait for batch to complete: "in_progress": try 40/120
2021/03/23 11:04:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/23 11:04:22 DEBUG : HTTP REQUEST (req 0xc0008f3100)
2021/03/23 11:04:22 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/23 11:04:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/23 11:04:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:23 DEBUG : HTTP RESPONSE (req 0xc0008f3100)
2021/03/23 11:04:23 DEBUG : HTTP/2.0 200 OK
Content-Length: 23
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 23 Mar 2021 10:04:22 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 0af7932849e54da192d98b6747535030
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 57

{".tag": "in_progress"}
2021/03/23 11:04:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:23 DEBUG : Dropbox root 'LACADOSALACANT/MAQUINAS/1': Sleeping for 1s to wait for batch to complete: "in_progress": try 41/120
2021/03/23 11:04:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/23 11:04:24 DEBUG : HTTP REQUEST (req 0xc0007d2a00)
2021/03/23 11:04:24 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/23 11:04:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/23 11:04:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:24 DEBUG : HTTP RESPONSE (req 0xc0007d2a00)
2021/03/23 11:04:24 DEBUG : HTTP/2.0 409 Conflict
Content-Length: 76
Accept-Encoding: identity,gzip
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Tue, 23 Mar 2021 10:04:24 GMT
Server: envoy
X-Dropbox-Request-Id: 4a06bdab19ca4b0cb6728e936b7227d9
X-Dropbox-Response-Origin: far_remote

{"error_summary": "internal_error/...", "error": {".tag": "internal_error"}}
2021/03/23 11:04:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/23 11:04:24 ERROR : Dropbox root 'LACADOSALACANT/MAQUINAS/1': sync batch commit: failed to commit batch length 1: wait for batch completion: check failed: internal_error/...
2021/03/23 11:04:50 INFO  : 
Transferred:   	  219.570G / 219.517 GBytes, 100%, 21.292 MBytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:    2h56m0.3s
Transferring:
 *   vzdump-qemu-160-2021_03_22-00_10_02.vma.lzo:100% /219.517G, 122.989k/s, -

2021/03/23 11:05:50 INFO  : 
Transferred:   	  219.570G / 219.517 GBytes, 100%, 21.172 MBytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:    2h57m0.3s
Transferring:
 *   vzdump-qemu-160-2021_03_22-00_10_02.vma.lzo:100% /219.517G, 2.559k/s, -

2021/03/23 11:06:50 INFO  : 
Transferred:   	  219.570G / 219.517 GBytes, 100%, 21.053 MBytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:    2h58m0.3s
Transferring:
 *   vzdump-qemu-160-2021_03_22-00_10_02.vma.lzo:100% /219.517G, 54/s, -

2021/03/23 11:07:50 INFO  : 
Transferred:   	  219.570G / 219.517 GBytes, 100%, 20.935 MBytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:    2h59m0.3s
Transferring:
 *   vzdump-qemu-160-2021_03_22-00_10_02.vma.lzo:100% /219.517G, 1/s, -

2021/03/23 11:08:50 INFO  : 
Transferred:   	  219.570G / 219.517 GBytes, 100%, 20.819 MBytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:     3h0m0.3s

ncw · March 23, 2021, 3:52pm

Hmm, that doesn't look good - it is an internal error from dropbox - they should never be returned.

I don't know what has happened here.

There was discussion recently about the large file uploads with rclone

The limit via the API seems to be 350G so you are below that.

That is a strange number.. By default rclone measures everything in k == 1024 bytes if you don't supply a suffix.

Luckily it doesn't apply to dropbox otherwise I think you'd notice a 32G buffer!

However you could try increasing the --dropbox-chunk-size and that might help

  --dropbox-chunk-size SizeSuffix   Upload chunk size. (< 150M). (default 48M)

Try using --dropbox-chunk-size 148M and see if that helps. (Keeping chunk size a multiple of 4M is a good idea for the dropbox backend).

saqqara · March 24, 2021, 8:29pm

Hello again. I have tried with a 300 Gb file. it didn't work. again internal error
I launched rclone with this parameters:

rclone sync vzdump-qemu-160-2021_03_22-00_10_02.vma.lzo temp: -vv --dump responses --dropbox-chunk-size 148M --log-file=./rclone.log

the end of the log

{".tag": "in_progress"}
2021/03/24 11:52:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/24 11:52:08 DEBUG : Dropbox root 'SERVIDORES/KEOPS/DISCOS': Sleeping for 1s to wait for batch to complete: "in_progress": try 96/120
2021/03/24 11:52:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/24 11:52:09 DEBUG : HTTP REQUEST (req 0xc000910900)
2021/03/24 11:52:09 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/24 11:52:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/24 11:52:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/24 11:52:09 DEBUG : HTTP RESPONSE (req 0xc000910900)
2021/03/24 11:52:09 DEBUG : HTTP/2.0 200 OK
Content-Length: 23
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Wed, 24 Mar 2021 10:52:09 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: a4edb74b7efa49a3b18dcd4bcff7ef05
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 176

{".tag": "in_progress"}
2021/03/24 11:52:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/24 11:52:09 DEBUG : Dropbox root 'SERVIDORES/KEOPS/DISCOS': Sleeping for 1s to wait for batch to complete: "in_progress": try 97/120
2021/03/24 11:52:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/24 11:52:10 DEBUG : HTTP REQUEST (req 0xc000843e00)
2021/03/24 11:52:10 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-beta.5327.83a01e556.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2021/03/24 11:52:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/24 11:52:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/24 11:52:11 DEBUG : HTTP RESPONSE (req 0xc000843e00)
2021/03/24 11:52:11 DEBUG : HTTP/2.0 409 Conflict
Content-Length: 75
Accept-Encoding: identity,gzip
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/json
Date: Wed, 24 Mar 2021 10:52:10 GMT
Server: envoy
X-Dropbox-Request-Id: d57e1064062e48e4b66504fa5af17237
X-Dropbox-Response-Origin: far_remote

{"error_summary": "internal_error/..", "error": {".tag": "internal_error"}}
2021/03/24 11:52:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/24 11:52:11 ERROR : Dropbox root 'SERVIDORES/KEOPS/DISCOS': sync batch commit: failed to commit batch length 1: wait for batch completion: check failed: internal_error/..

At the moment, I am copying the files by mounting the storage. So if it works.

thanks.

ncw · March 25, 2021, 2:27pm

Hmm, internal_error means something went wrong at dropbox.

Have you tried asking on the dropbox forum or asking their support?

saqqara · March 28, 2021, 9:52am

Hello, I have been doing various tests. It seems that the internal error is with the beta version. I have tried mounting the storage and copy directly and it fails too with the beta version.

Now, i am using version v1.54.1, with parameters: --dump responses --dropbox-chunk-size 148M.
If i mount the storage and use 'cp' it works.
If i use

rclone sync file dropbox:  --dump responses --dropbox-chunk-size 148M

the file is copied, but the rclone process never stop.

regards.

whitenexx · March 29, 2021, 12:57pm

I can confirm this issue with large files. This also happens in copy command. For me this happens in the following scenario:

rclone 1.54.1
dropbox_remote <= crypt_remote <= chunker_remote (chunking to 349G, 320G, 150G or 25G, every scenario) <= file of size 429GB

The rclone command runs infinite and transfers a lot of TB (instead of the 429G) regarding to the retries.
What i also see: 5 hours later i can see the "parts/chunks" appearing in dropbox when the rclone command wasn't running anymore. So it seems that the files occur later in dropbox with a delay. Is this possible?

/UPDATE/ It's 10 hours later and more and more parts of my old copy from last night occur in dropbox. There must be a huge delay of many hours until parts appear visible in dropbox. In this concrete scenario i used 25G chunking/parts. The copy started at ~ 03:01:39 at night and now it's 15:00:00 and part 9 is occuring! It wasn't visible 1 hour ago. More and more parts get visible.

/UPDATE 2/ Sorry my bad. The information i've given before is wrong. The parts that occur are parts from another process which writes directly into an rclone mount. So there is no delay and no relation to the copy/upload from the night before. The main bug, that uploading large files via rclone copy doesn't work is still there.

I already talked to the dropbox support. They don't want to help me because i'm not using the official dropbox client.

ncw · March 29, 2021, 6:39pm

What errors are you getting in your log? If you run with -vv what do the retries say?

system · May 29, 2021, 2:39pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.