OpenDrive inconsistently failing on files >10GB


Occasional user, first time poster here. I’ve been trying to back up some things to OpenDrive through rclone. Mostly it works fine, but it seems to choke on individual files bigger than 10GB or so. There’s no explicit error (that I saw), it just tries to upload them over and over. It gets to the end, looks like it worked, then starts again.

I am aware that there was originally a problem with large files with the WebDav mode on OpenDrive, but this is using the new OpenDrive mode in 1.42, with encryption if it matters.

The chance of failure seems to increase with size. Files just slightly larger than 10GB worked eventually, but if it’s much more than that (one file I’m testing with is about 15GB), I haven’t been able to get it to upload cleanly.

According to OpenDrive, there are no file size limits for their service, so at least theoretically this should work.

I can provide any logs that would be helpful.

If you could run rclone with -vv --dump responses and post some logs around the failure that would be very useful.

It might just be the way opendrive works but maybe rclone can work around it.

I’ve started running it with those parameters. After leaving it to go while I was at work, the log is already quite large (about 8MB), so what’s the best way to get it to you? And is there anything I would need to redact such as passwords?

If you could zip it and email it to me at - put a link to this forum post in please!

As for redaction - rclone will redact the Auth headers. If there is an /oauth transaction at the start you can delete that.

Just want to verify that this went through when I sent it last week, since I never got a confirmation.

Apologies for the delay…

This appears to be the relevant part of the transaction (redacted), which is the second to last chunk of the file being uploaded

2018/08/16 14:30:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/08/16 14:30:08 DEBUG : encryptedXXXX: Uploading chunk 1632, size=10485760, remain=14566075
2018/08/16 14:30:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/08/16 14:30:08 DEBUG : HTTP REQUEST (req 0xc420100200)
2018/08/16 14:30:08 DEBUG : POST /api/v1/upload/upload_file_chunk.json HTTP/1.1
User-Agent: rclone/v1.42
Content-Length: 10487308
Content-Type: multipart/form-data; boundary=e6009437e443da64c4cc7bcad81ceb2826c39d8cf2fe7c575e69854af8ae
Accept-Encoding: gzip

2018/08/16 14:30:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/08/16 14:30:11 INFO  : 
Transferred:   159.605 GBytes (1.464 MBytes/s)
Errors:                 0
Checks:                 1
Transferred:            0
Elapsed time:   31h1m1.1s
 *   ...XXX: 100% /15.957G, 1.071M/s, 0s

2018/08/16 14:30:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/08/16 14:30:17 DEBUG : HTTP RESPONSE (req 0xc420100200)
2018/08/16 14:30:17 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Cache-Control: no-cache, must-revalidate
Connection: keep-alive
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Aug 2018 19:30:17 GMT
Expires: 0
Last-Modified: Thu, 16 Aug 2018 19:30:17 GMT
Server: nginx
Serverid: s7


2018/08/16 14:30:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/08/16 14:30:17 DEBUG : encryptedXXXX: Uploading chunk 1633, size=10485760, remain=4080315
2018/08/16 14:30:17 ERROR : XXX: Failed to copy: failed to create file: EOF

It isn’t clear to me exactly where that EOF comes from as the transaction appears to be complete.

Probably the thing to do next if you could make a new issue on github with the above in (put a link to here) and I’ll see if I can get some help from the original author of the backend as to what is going on.