Default --onedrive-chunk-size does NOT work

hvrietsc · October 25, 2020, 7:27pm

What is the problem you are having with rclone?

rclone to onedrive stops after uploading 10M of a file
when I use --onedrive-chunk-size 5M everything works fine
Using chunk-size larger than 10M result is either error message saying chunk size must be a multiple of 320K or it (again) stops after uploading 10M

What is your rclone version (output from `rclone version`)

rclone v1.53.1

os/arch: linux/amd64
go version: go1.15.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux archie 5.9.1-arch1-1 #1 SMP PREEMPT Sat, 17 Oct 2020 13:30:37 +0000 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

onedrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy /home/user/dirname onedrive:/dirname/

The rclone config contents with secrets removed.


[onedrive]
type = onedrive
token = {"access_token":"XXXXXX"
drive_id = XXXXXXXX
drive_type = personal

A log from the command with the `-vv` flag

2020/10/25 12:18:48 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "-vv" "-P" "/home/hans/bluetooth/test/" "onedrive:hans/backups/bluetooth/test/"]
2020/10/25 12:18:48 DEBUG : Creating backend with remote "/home/hans/bluetooth/test/"
2020/10/25 12:18:48 DEBUG : Using config file from "/home/hans/.config/rclone/rclone.conf"
2020/10/25 12:18:48 DEBUG : Creating backend with remote "onedrive:hans/backups/bluetooth/test/"
2020/10/25 12:18:49 DEBUG : fs cache: renaming cache item "onedrive:hans/backups/bluetooth/test/" to be canonical "onedrive:hans/backups/bluetooth/test"
2020-10-25 12:18:50 DEBUG : One drive root 'hans/backups/bluetooth/test': Waiting for checks to finish
2020-10-25 12:18:50 DEBUG : One drive root 'hans/backups/bluetooth/test': Waiting for transfers to finish
2020-10-25 12:18:50 DEBUG : 20100729-henna-hands.mov: Starting multipart upload
2020-10-25 12:18:50 DEBUG : 20100729-henna-hands2.mov: Starting multipart upload
2020-10-25 12:18:50 DEBUG : 20100729-making-favor-boxes.mov: Starting multipart upload
2020-10-25 12:18:50 DEBUG : 20100729-naryan1.mov: Starting multipart upload
2020-10-25 12:18:50 DEBUG : 20100729-henna-hands.mov: Reopening on read failure after 4096 bytes: retry 1/10: can't read status of source file while transferring: stat /home/hans/bluetooth/test/20100729-henna-hands.mov: interrupted system call
2020-10-25 12:18:50 DEBUG : 20100729-henna-hands.mov: Uploading segment 0/10826454 size 10485760
2020-10-25 12:18:50 DEBUG : 20100729-henna-hands2.mov: Uploading segment 0/44152677 size 10485760
2020-10-25 12:18:50 DEBUG : 20100729-making-favor-boxes.mov: Uploading segment 0/50491135 size 10485760
2020-10-25 12:18:51 DEBUG : 20100729-naryan1.mov: Uploading segment 0/45318629 size 10485760
2020-10-25 12:21:56 DEBUG : Couldn't decode error response: EOF
2020-10-25 12:21:56 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-25 12:21:56 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020-10-25 12:21:59 DEBUG : Couldn't decode error response: EOF
2020-10-25 12:21:59 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-25 12:21:59 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2020-10-25 12:22:00 DEBUG : Couldn't decode error response: EOF
2020-10-25 12:22:00 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-25 12:22:00 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2020-10-25 12:22:03 DEBUG : Couldn't decode error response: EOF
2020-10-25 12:22:03 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-25 12:22:03 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2020-10-25 12:24:57 DEBUG : Couldn't decode error response: EOF
2020-10-25 12:24:57 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-25 12:24:57 DEBUG : pacer: Rate limited, increasing sleep to 320ms


Transferred:           40M / 199.905 MBytes, 20%, 109.195 kBytes/s, ETA 24m59s
Transferred: 2020-10-25 12:25:05 DEBUG : Couldn't decode error response: EOF
2020-10-25 12:25:05 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-25 12:25:05 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2020-10-25 12:25:06 DEBUG : Couldn't decode error response: EOF
2020-10-25 12:25:06 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-25 12:25:06 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2020-10-25 12:25:07 DEBUG : Couldn't decode error response: EOF
2020-10-25 12:25:07 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-25 12:25:07 DEBUG : pacer: Rate limited, increasing sleep to 2s
           0 / 7, 0%
Elapsed time:      6m16.5s
Transferring:
 *                      20100729-henna-hands.mov: 96% /10.325M, 0/s, 380h46m56s
 *                     20100729-henna-hands2.mov: 23% /42.107M, 0/s, 25867h0m38s
 *               20100729-making-favor-boxes.mov: 20% /48.152M, 0/s, 36125h31m58s
 *                          20100729-naryan1.mov: 23% /43.219M, 0/s, 22097h19m12s

asdffdsa · October 25, 2020, 7:47pm

hello and welcome to the forum

i do not see that error in the log you posted?
can you use a command that will generate that error?

hvrietsc · October 25, 2020, 7:51pm

020/10/25 12:50:57 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "-vv" "-P" "--onedrive-chunk-size" "11M" "/home/hans/bluetooth/test/" "onedrive:hans/backups/bluetooth/test/"]
2020/10/25 12:50:57 DEBUG : Creating backend with remote "/home/hans/bluetooth/test/"
2020/10/25 12:50:57 DEBUG : Using config file from "/home/hans/.config/rclone/rclone.conf"
2020/10/25 12:50:57 DEBUG : Creating backend with remote "onedrive:hans/backups/bluetooth/test/"
2020/10/25 12:50:57 Failed to create file system for "onedrive:hans/backups/bluetooth/test/": onedrive: chunk size: 11M is not a multiple of 320k

asdffdsa · October 25, 2020, 7:59pm

as per the docs, the default value is 10M,

if you want a larger value, then it has to be a multiple of 320k or 327,680 bytes
34 X 327,680 = 11,141,120

--onedrive-chunk-size=11141120

hvrietsc · October 25, 2020, 8:11pm

I know. But the real problem is that for the default value of 10M it just stalls after 10M per file transfer. See my original log. Basically it stalls after 10M per file if --onedrive-chunk-size is 10M or larger.

I can get it to work properly if I use 5M as the chunk size. But to me it seems like there is a bug here.

asdffdsa · October 25, 2020, 8:20pm

ok, i see what you mean now

ncw · October 26, 2020, 12:23pm

hvrietsc:

2020-10-25 12:18:50 DEBUG : 20100729-henna-hands.mov: Reopening on read failure after 4096 bytes: retry 1/10: can't read status of source file while transferring: stat /home/hans/bluetooth/test/20100729-henna-hands.mov: interrupted system call

I think this is the problem line...

This is probably this issue

Can you try with the latest beta to see if it is any different.

Also can you try with setting the enviroment variable

export GODEBUG=asyncpreemptoff=1

And see if that fixes it.

What file system are you using? NFS?

hvrietsc · October 26, 2020, 3:50pm

did the export GODEBUG=asyncpreemptoff=1 (with non-beta rclone)
copying from a regular HDD to onedrive
Still stalls after 4x10M of uploads

looks like the DEBUG : pacer lines might be more interesting to look at
to me it looks like "it" wants me to go slower but at 0b/s I am going as slow as I can

Here is the output:
+ OPTIONS=' -P -vv '
+ export GODEBUG=asyncpreemptoff=1
+ GODEBUG=asyncpreemptoff=1
+ rclone version
rclone v1.53.1
- os/arch: linux/amd64
- go version: go1.15.2
+ uname -a
Linux archie 5.9.1-arch1-1 #1 SMP PREEMPT Sat, 17 Oct 2020 13:30:37 +0000 x86_64 GNU/Linux
+ rclone copy -P -vv /home/hans/test/rclone/ onedrive:hans/test/rclone/
2020/10/26 08:41:02 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "-P" "-vv" "/home/hans/test/rclone/" "onedrive:hans/test/rclone/"]
2020/10/26 08:41:02 DEBUG : Creating backend with remote "/home/hans/test/rclone/"
2020/10/26 08:41:02 DEBUG : Using config file from "/home/hans/.config/rclone/rclone.conf"
2020/10/26 08:41:02 DEBUG : Creating backend with remote "onedrive:hans/test/rclone/"
2020/10/26 08:41:03 DEBUG : fs cache: renaming cache item "onedrive:hans/test/rclone/" to be canonical "onedrive:hans/test/rclone"
2020-10-26 08:41:03 DEBUG : 20100729-making-favor-boxes.mov: Starting multipart upload
2020-10-26 08:41:03 DEBUG : 20100729-henna-hands.mov: Starting multipart upload
2020-10-26 08:41:03 DEBUG : 20100729-naryan1.mov: Starting multipart upload
2020-10-26 08:41:03 DEBUG : One drive root 'hans/test/rclone': Waiting for checks to finish
2020-10-26 08:41:03 DEBUG : One drive root 'hans/test/rclone': Waiting for transfers to finish
2020-10-26 08:41:03 DEBUG : 20100729-henna-hands2.mov: Starting multipart upload
2020-10-26 08:41:04 DEBUG : 20100729-making-favor-boxes.mov: Uploading segment 0/50491135 size 10485760
2020-10-26 08:41:04 DEBUG : 20100729-naryan1.mov: Uploading segment 0/45318629 size 10485760
2020-10-26 08:41:04 DEBUG : 20100729-henna-hands2.mov: Uploading segment 0/44152677 size 10485760
2020-10-26 08:41:04 DEBUG : 20100729-henna-hands.mov: Uploading segment 0/10826454 size 10485760
2020-10-26 08:44:02 DEBUG : Couldn't decode error response: EOF
2020-10-26 08:44:02 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 08:44:02 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020-10-26 08:44:03 DEBUG : Couldn't decode error response: EOF
2020-10-26 08:44:03 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 08:44:03 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2020-10-26 08:44:05 DEBUG : Couldn't decode error response: EOF
2020-10-26 08:44:05 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 08:44:05 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2020-10-26 08:44:05 DEBUG : Couldn't decode error response: EOF
2020-10-26 08:44:05 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 08:44:05 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2020-10-26 08:46:53 DEBUG : Couldn't decode error response: EOF
2020-10-26 08:46:53 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-26 08:46:53 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2020-10-26 08:47:02 DEBUG : Couldn't decode error response: EOF
2020-10-26 08:47:02 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-26 08:47:02 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2020-10-26 08:47:03 DEBUG : Couldn't decode error response: EOF
2020-10-26 08:47:03 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-26 08:47:03 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2020-10-26 08:47:05 DEBUG : Couldn't decode error response: EOF
2020-10-26 08:47:05 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-26 08:47:05 DEBUG : pacer: Rate limited, increasing sleep to 2s
Transferred:           40M / 199.905 MBytes, 20%, 113.193 kBytes/s, ETA 24m6s
Transferred:            0 / 7, 0%
Elapsed time:       6m2.8s
Transferring:
 *                      20100729-henna-hands.mov: 96% /10.325M, 0/s, 275h45m34s
 *                     20100729-henna-hands2.mov: 23% /42.107M, 0/s, 22601h42m3s
 *               20100729-making-favor-boxes.mov: 20% /48.152M, 0/s, 26633h20m41s
 *                          20100729-naryan1.mov: 23% /43.219M, 0/s, 22157h45m31s

hvrietsc · October 26, 2020, 4:07pm

Here are the test results from using the v1.53.2 beta (same problem).

+ OPTIONS=' -P -vv '
+ export GODEBUG=asyncpreemptoff=1
+ GODEBUG=asyncpreemptoff=1
+ rclone-beta version
rclone v1.53.2
- os/arch: linux/amd64
- go version: go1.15.3
+ uname -a
Linux archie 5.9.1-arch1-1 #1 SMP PREEMPT Sat, 17 Oct 2020 13:30:37 +0000 x86_64 GNU/Linux
+ rclone-beta copy -P -vv /home/hans/test/rclone/ onedrive:hans/test/rclone/
2020/10/26 08:58:50 DEBUG : rclone: Version "v1.53.2" starting with parameters ["rclone-beta" "copy" "-P" "-vv" "/home/hans/test/rclone/" "onedrive:hans/test/rclone/"]
2020/10/26 08:58:50 DEBUG : Creating backend with remote "/home/hans/test/rclone/"
2020/10/26 08:58:50 DEBUG : Using config file from "/home/hans/.config/rclone/rclone.conf"
2020/10/26 08:58:50 DEBUG : Creating backend with remote "onedrive:hans/test/rclone/"
2020/10/26 08:58:50 DEBUG : fs cache: renaming cache item "onedrive:hans/test/rclone/" to be canonical "onedrive:hans/test/rclone"
2020-10-26 08:58:51 DEBUG : 20100729-henna-hands.mov: Starting multipart upload
2020-10-26 08:58:51 DEBUG : One drive root 'hans/test/rclone': Waiting for checks to finish
2020-10-26 08:58:51 DEBUG : 20100729-henna-hands2.mov: Starting multipart upload
2020-10-26 08:58:51 DEBUG : 20100729-naryan1.mov: Starting multipart upload
2020-10-26 08:58:51 DEBUG : 20100729-making-favor-boxes.mov: Starting multipart upload
2020-10-26 08:58:51 DEBUG : One drive root 'hans/test/rclone': Waiting for transfers to finish
2020-10-26 08:58:51 DEBUG : 20100729-henna-hands.mov: Uploading segment 0/10826454 size 10485760
2020-10-26 08:58:51 DEBUG : 20100729-henna-hands2.mov: Uploading segment 0/44152677 size 10485760
2020-10-26 08:58:51 DEBUG : 20100729-making-favor-boxes.mov: Uploading segment 0/50491135 size 10485760
2020-10-26 08:58:51 DEBUG : 20100729-naryan1.mov: Uploading segment 0/45318629 size 10485760
2020-10-26 09:01:51 DEBUG : Couldn't decode error response: EOF
2020-10-26 09:01:51 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 09:01:51 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020-10-26 09:01:52 DEBUG : Couldn't decode error response: EOF
2020-10-26 09:01:52 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 09:01:52 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2020-10-26 09:01:52 DEBUG : Couldn't decode error response: EOF
2020-10-26 09:01:52 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 09:01:52 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2020-10-26 09:01:53 DEBUG : Couldn't decode error response: EOF
2020-10-26 09:01:53 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 09:01:53 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2020-10-26 09:04:45 DEBUG : One drive root 'hans/test/rclone': Token expired - 4 uploads in progress - refreshing
2020-10-26 09:04:45 DEBUG : onedrive: Loaded invalid token from config file - ignoring
2020-10-26 09:04:46 DEBUG : Couldn't decode error response: EOF
2020-10-26 09:04:46 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-26 09:04:46 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2020-10-26 09:04:48 DEBUG : onedrive: Saved new token in config file
2020-10-26 09:04:50 DEBUG : pacer: Reducing sleep to 240ms
2020-10-26 09:04:50 DEBUG : One drive root 'hans/test/rclone': Token refresh successful
2020-10-26 09:04:51 DEBUG : Couldn't decode error response: EOF
2020-10-26 09:04:51 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-26 09:04:51 DEBUG : pacer: Rate limited, increasing sleep to 480ms
2020-10-26 09:04:51 DEBUG : Couldn't decode error response: EOF
2020-10-26 09:04:51 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-26 09:04:51 DEBUG : pacer: Rate limited, increasing sleep to 960ms
2020-10-26 09:04:51 DEBUG : Couldn't decode error response: EOF
2020-10-26 09:04:51 DEBUG : pacer: low level retry 2/10 (error 504 Gateway Timeout: )
2020-10-26 09:04:51 DEBUG : pacer: Rate limited, increasing sleep to 1.92s
Transferred:           40M / 199.905 MBytes, 20%, 113.349 kBytes/s, ETA 24m4s
Transferred:            0 / 7, 0%
Elapsed time:       6m2.3s
Transferring:
 *                      20100729-henna-hands.mov: 96% /10.325M, 0/s, 231h42m4s
 *                     20100729-henna-hands2.mov: 23% /42.107M, 0/s, 19638h43m45s
 *               20100729-making-favor-boxes.mov: 20% /48.152M, 0/s, 20304h37m7s
 *                          20100729-naryan1.mov: 23% /43.219M, 0/s, 22040h44m56s

ncw · October 26, 2020, 6:53pm

This looks like the problem in that log. Something (onedrive I guess) is timing out. Rclone retries but it isn't getting anywhere.

I tried uploading a file with 10M chunks and it worked just fine for me.

Are you running through a corporate firewall or something like that?

hvrietsc · October 26, 2020, 7:22pm

Nope just me at home with my ATT link. With --onedrive-chunk-size 5M the problems goes away

asdffdsa · October 26, 2020, 7:44pm

works for me too.

rclone: Version "v1.53.1" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "sync" "C:\\data\\u\\dummy\\dum" "onedrive01:onedrivelimit" "-vv" "--progress" "--onedrive-chunk-size=15M"]
40MB.file: Starting multipart upload
40MB.file: Uploading segment 0/41943040 size 15728640
40MB.file: Uploading segment 15728640/41943040 size 15728640
40MB.file: Uploading segment 31457280/41943040 size 10485760
40MB.file: SHA-1 = 6b414307f639a12baf2ba644a15ffbcd88223c96 OK
40MB.file: Copied (new)
2020/10/26 15:42:12 INFO  :
Transferred:           40M / 40 MBytes, 100%, 5.365 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         8.3s

hvrietsc · October 26, 2020, 7:55pm

Did the same command with 15M chunk size and now it stops after 55M.

Notice the EOF file errors!!!! Once it gets those it starts slowing down rclone uploads (basically to zero).

+ OPTIONS=' -P -vv --onedrive-chunk-size 15M '
+ export GODEBUG=asyncpreemptoff=1
+ GODEBUG=asyncpreemptoff=1
+ rclone-beta version
rclone v1.53.2
- os/arch: linux/amd64
- go version: go1.15.3
+ uname -a
Linux archie 5.9.1-arch1-1 #1 SMP PREEMPT Sat, 17 Oct 2020 13:30:37 +0000 x86_64 GNU/Linux
+ rclone-beta copy -P -vv --onedrive-chunk-size 15M /home/hans/test/rclone/ onedrive:hans/test/rclone/
2020/10/26 12:47:11 DEBUG : rclone: Version "v1.53.2" starting with parameters ["rclone-beta" "copy" "-P" "-vv" "--onedrive-chunk-size" "15M" "/home/hans/test/rclone/" "onedrive:hans/test/rclone/"]
2020/10/26 12:47:11 DEBUG : Creating backend with remote "/home/hans/test/rclone/"
2020/10/26 12:47:11 DEBUG : Using config file from "/home/hans/.config/rclone/rclone.conf"
2020/10/26 12:47:11 DEBUG : Creating backend with remote "onedrive:hans/test/rclone/"
2020/10/26 12:47:12 DEBUG : fs cache: renaming cache item "onedrive:hans/test/rclone/" to be canonical "onedrive:hans/test/rclone"
2020-10-26 12:47:12 DEBUG : 20100729-henna-hands.mov: Starting multipart upload
2020-10-26 12:47:12 DEBUG : 20100729-making-favor-boxes.mov: Starting multipart upload
2020-10-26 12:47:12 DEBUG : 20100729-henna-hands2.mov: Starting multipart upload
2020-10-26 12:47:12 DEBUG : One drive root 'hans/test/rclone': Waiting for checks to finish
2020-10-26 12:47:12 DEBUG : One drive root 'hans/test/rclone': Waiting for transfers to finish
2020-10-26 12:47:12 DEBUG : 20100729-naryan1.mov: Starting multipart upload
2020-10-26 12:47:12 DEBUG : 20100729-henna-hands.mov: Uploading segment 0/10826454 size 10826454
2020-10-26 12:47:12 DEBUG : 20100729-naryan1.mov: Uploading segment 0/45318629 size 15728640
2020-10-26 12:47:12 DEBUG : 20100729-henna-hands2.mov: Uploading segment 0/44152677 size 15728640
2020-10-26 12:47:12 DEBUG : 20100729-making-favor-boxes.mov: Uploading segment 0/50491135 size 15728640
2020-10-26 12:48:41 DEBUG : One drive root 'hans/test/rclone': Token expired - 4 uploads in progress - refreshing
2020-10-26 12:48:41 DEBUG : onedrive: Loaded invalid token from config file - ignoring
2020-10-26 12:48:44 DEBUG : onedrive: Saved new token in config file
2020-10-26 12:48:46 DEBUG : One drive root 'hans/test/rclone': Token refresh successful
2020-10-26 12:50:18 DEBUG : Couldn't decode error response: EOF
2020-10-26 12:50:18 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 12:50:18 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020-10-26 12:51:36 DEBUG : Couldn't decode error response: EOF
2020-10-26 12:51:36 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 12:51:36 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2020-10-26 12:51:39 DEBUG : Couldn't decode error response: EOF
2020-10-26 12:51:39 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 12:51:39 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2020-10-26 12:51:39 DEBUG : Couldn't decode error response: EOF2020-10-26 12:51:39 DEBUG : pacer: low level retry 1/10 (error 504 Gateway Timeout: )
2020-10-26 12:51:39 DEBUG : pacer: Rate limited, increasing sleep to 160ms
Transferred:       55.325M / 199.905 MBytes, 28%, 172.271 kBytes/s, ETA 14m19s
Transferred:            0 / 7, 0%
Elapsed time:      5m30.1s
Transferring:
 *                     20100729-henna-hands2.mov: 35% /42.107M, 768/s, 10h16m30s
 *                      20100729-henna-hands.mov:100% /10.325M, 4/s, 0s
 *               20100729-making-favor-boxes.mov: 31% /48.152M, 981/s, 9h50m10s
 *                          20100729-naryan1.mov: 34% /43.219M, 953/s, 8h37m24s

asdffdsa · October 26, 2020, 7:59pm

it seems to be something specific to your setup.
perhaps internet, perhaps os/kernel
perhaps reboot modem, router, computer?
do you have any other computers to test with, or a virtual machine?

can you do a speedtest?

hvrietsc · October 26, 2020, 8:11pm

Just ran this on my MACbook air same problem:

couldn't decode error response: EOF
and then a slew of rate limited increasing sleep times

Google speed test gives:

15.1 Mbps download

1.81 Mbps upload

Which is normal for me

(OH btw: doing the same thing to S3 works just fine)

asdffdsa · October 26, 2020, 8:24pm

perhaps onedrive has issue with the amount of time it takes to upload a chunk.

given the super slow upload speed, no advantage in changing the chunk size from default.

never used such flags, perhaps you can tweak settings like this
https://rclone.org/docs/#contimeout-time

hvrietsc · October 26, 2020, 9:04pm

The upload speed is using my max upload bandwidth, just when it gets to 10M it simply slows down to slower and slower until it hits 0b/s

This is not a timeout connection issue. From the logs it looks like pacer is trying to slow me down but only after 10M was sent (per file).

ncw · October 27, 2020, 11:32am

Can you run your command with -vv --dump responses and post the log. Or if you don't want to post the whole thing, then the HTTP response that causes the EOF error will be the most interesting thing!

hvrietsc · October 27, 2020, 8:25pm

I can't too much output and I am not sure all those http requests do not contain my "keys" to onedrive.

ncw · October 28, 2020, 11:19am

If you want to private message me a link you can do that.

Rclone writes XXX into the auth token line so you should be OK there.