Opendrive Large Files Fail. 30+GB File

oneandonlyjason · May 31, 2021, 10:16am

What is the problem you are having with rclone?

Uploading large Files to Opendrive fail even with High Timeout set

What is your rclone version (output from `rclone version`)

rclone v1.52.2

os/arch: linux/amd64
go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian 10 64 Bit

Which cloud storage system are you using? (eg Google Drive)

Opendrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync /media/Filme/ plexcrypt:Filme/ -P --timeout 300m -vv

The rclone config contents with secrets removed.

[plex]
type = opendrive
username = REMOVED
password = REMOVED

[plexcrypt]
type = crypt
remote = plex:pms
filename_encryption = standard
directory_name_encryption = true
password = REMOVED
password2 = REMOVED

A log from the command with the `-vv` flag

2021-05-31 10:34:48 DEBUG : slj4gma21bfqj8cui7jun5k1sijlu2loau98qvciaj02dj9fr2q0/33t2tkc38i9dmi9rmup6ltlgied29qqp1gg46virdgsjbbbhp2p0: Uploading chunk 3744, size=10485760, remain=2446122
2021-05-31 10:34:51 DEBUG : slj4gma21bfqj8cui7jun5k1sijlu2loau98qvciaj02dj9fr2q0/33t2tkc38i9dmi9rmup6ltlgied29qqp1gg46virdgsjbbbhp2p0: Uploading chunk 3745, size=2446122, remain=0
2021-05-31 10:49:53 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 10:49:53 DEBUG : pacer: low level retry 1/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 10:49:53 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021-05-31 11:04:53 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 11:04:53 DEBUG : pacer: low level retry 2/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 11:04:53 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2021-05-31 11:19:54 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 11:19:54 DEBUG : pacer: low level retry 3/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 11:19:54 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2021-05-31 11:34:54 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 11:34:54 DEBUG : pacer: low level retry 4/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 11:34:54 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2021-05-31 11:49:56 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 11:49:56 DEBUG : pacer: low level retry 5/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 11:49:56 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2021-05-31 12:04:58 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 12:04:58 DEBUG : pacer: low level retry 6/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 12:04:58 DEBUG : pacer: Rate limited, increasing sleep to 640ms

Animosity022 · May 31, 2021, 11:40am

Probably won't make a difference, but that is a pretty old version. Can you validate with the latest and see if the issue still persists?

oneandonlyjason · May 31, 2021, 11:56am

I updated to the newest and will try again. Will take a couple Hours, will report back then

oneandonlyjason · May 31, 2021, 1:55pm

Now i get this Error:

2021-05-31 15:36:40 DEBUG : slj4gma21bfqj8cui7jun5k1sijlu2loau98qvciaj02dj9fr2q0/33t2tkc38i9dmi9rmup6ltlgied29qqp1gg46virdgsjbbbhp2p0: Uploading chunk 1819, size=10485760, remain=20187534122
2021-05-31 15:36:45 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:45 DEBUG : pacer: low level retry 1/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:45 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021-05-31 15:36:47 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:47 DEBUG : pacer: low level retry 2/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:47 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2021-05-31 15:36:48 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:48 DEBUG : pacer: low level retry 3/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:48 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2021-05-31 15:36:50 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:50 DEBUG : pacer: low level retry 4/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:50 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2021-05-31 15:36:51 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:51 DEBUG : pacer: low level retry 5/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:51 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2021-05-31 15:36:52 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:52 DEBUG : pacer: low level retry 6/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:52 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2021-05-31 15:36:53 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:53 DEBUG : pacer: low level retry 7/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:53 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2021-05-31 15:36:55 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:55 DEBUG : pacer: low level retry 8/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:55 DEBUG : pacer: Rate limited, increasing sleep to 2.56s
2021-05-31 15:36:56 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:56 DEBUG : pacer: low level retry 9/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:56 DEBUG : pacer: Rate limited, increasing sleep to 5.12s
2021-05-31 15:36:58 DEBUG : Couldn't decode error response: EOF
2021-05-31 15:36:58 DEBUG : pacer: low level retry 10/10 (error Unknown 500 Internal Server Error (Error 500))
2021-05-31 15:36:58 DEBUG : pacer: Rate limited, increasing sleep to 10.24s

But i dont really know if this is a Error from Opendrive this Time because this Time the Upload Failed before all got transfered and not after the Transfer technically finished

oneandonlyjason · May 31, 2021, 6:27pm

I retried it with the newer Version again and sadly still get the error that i showed in my First Post

2021-05-31 19:36:27 DEBUG : slj4gma21bfqj8cui7jun5k1sijlu2loau98qvciaj02dj9fr2q0/33t2tkc38i9dmi9rmup6ltlgied29qqp1gg46virdgsjbbbhp2p0: Uploading chunk 3745, size=2446122, remain=0
2021-05-31 19:51:30 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 19:51:30 DEBUG : pacer: low level retry 1/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 19:51:30 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021-05-31 20:06:30 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 20:06:30 DEBUG : pacer: low level retry 2/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 20:06:30 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2021-05-31 20:21:30 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2021-05-31 20:21:30 DEBUG : pacer: low level retry 3/10 (error Unknown 504 Gateway Time-out (Error 504))
2021-05-31 20:21:30 DEBUG : pacer: Rate limited, increasing sleep to 80ms

Strange is that i can upload other Files that are smaller. Thats the only File that doesnt upload

asdffdsa · May 31, 2021, 6:48pm

what is the max file size at opendrive for your specfic plan?

oneandonlyjason · May 31, 2021, 7:32pm

There is no Filesize Limit for my Plan

asdffdsa · May 31, 2021, 7:37pm

well, there is always a file size limit.

oneandonlyjason · May 31, 2021, 7:38pm

There sure is, but they dont show it anywhere and in the Control Panel they just say Unlimited Filesize.

asdffdsa · May 31, 2021, 7:44pm

30GB is not that large of a file size but i never heard of unlimited file size before.

i would try to upload that 30+GB file using opendrive software/website and see what happens.

not sure what the exact problem is but in the meantime, i would contact opendrive tech support.

asdffdsa · May 31, 2021, 7:55pm

are you using a vpn or proxy server?

https://forum.rclone.org/t/dropbox-upload-failed-invalid-character/23030

oneandonlyjason · May 31, 2021, 8:55pm

Im not using a VPN or Proxy Server

Will try this

oneandonlyjason · June 2, 2021, 12:33pm

Ok News for this. I could Upload it this time. It needed a couple more tries but it uploaded eventually. I did log the Responses and it seems like there is happening a Gateway Timeout for Big Files often. And Rclone throws this Error because it does not understand the Timeout Error i guess.

Content-Length: 176
Connection: keep-alive
Content-Type: text/html
Date: Wed, 02 Jun 2021 03:31:38 GMT
Server: nginx
Serverid: s7

<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
2021-06-02 05:31:38 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value

I opened a Support Ticket with them and will see what they answer

oneandonlyjason · June 2, 2021, 1:51pm

They replied that i should increase the Timeout value to 1900 seconds but i guess that this is exactly what the Rclone --timeout flag does?

asdffdsa · June 2, 2021, 2:53pm

there are several timeout flags, not sure what to recommend.
@Animosity022 might have an idea.

as for this message, not sure why rclone cannot parse the response?
is there some issue with the server or with rclone?

Couldn't decode error response: invalid character '<' looking for beginning of value

Animosity022 · June 2, 2021, 2:57pm

By default the timeout is 5 minutes:

   --timeout duration                     IO idle timeout (default 5m0s)

You can bump that up to an hour or something and see if that fixes it.

oneandonlyjason · June 3, 2021, 1:38pm

I already did that thats the Reason i get a Gateway Timeout instead of a RClone Timeout.
I contacted the Support of Opendrive and they told me that i should use the Webdav Endpoint instead of the Opendrive API that Rclone uses when i upload big files because the Webdav endpoint has more Servers behind it and can faster response to big files because of this.

system · August 3, 2021, 9:39am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.