Rclone Nextcloud/Webdav Failed to sync: net/http: server replied with more than declared Content-Length; truncated

What is the problem you are having with rclone?

I want to backup my photos and videos from Nextcloud (provider is hetzner) through webdav to my local debian server. Everything works perfectly for smaller files, but for larger files I get a strange error. For demonstration purposes I created a folder Test with one file inside):

20210807-191538.mp4
Filesize: 2,43 GB (2.618.761.246 Bytes)

RClone seems to detect a wrong filesize (7.317 GiByte see log). The correct 2618761246 bytes are getting downloaded, but rclone fails after this.

What is your rclone version (output from rclone version)

v1.56.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian 10

Which cloud storage system are you using? (eg Google Drive)

Nextcloud through webdav

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync NextcloudHetzner:Test /mnt/hdd/rclone/Test/current/ --backup-dir /mnt/hdd/rclone/Test/backup/2021-08-22_191352 -P --multi-thread-streams 0 --log-file=mylogfile.txt --log-level DEBUG

The rclone config contents with secrets removed.

[NextcloudHetzner]
type = webdav
url = https://***.your-storageshare.de/remote.php/dav/files/Backup/
vendor = nextcloud
user = ***
pass = ***

A log from the command with the -vv flag

2021/08/22 19:13:52 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "sync" "NextcloudHetzner:Test" "/mnt/hdd/rclone/Test/current/" "--backup-dir" "/mnt/hdd/rclone/Test/backup/2021-08-22_191352" "-P" "--multi-thread-streams" "0" "--log-file=mylogfile.txt" "--log-level" "DEBUG"]
2021/08/22 19:13:52 DEBUG : Creating backend with remote "NextcloudHetzner:Test"
2021/08/22 19:13:52 DEBUG : Using config file from "/home/raph/.config/rclone/rclone.conf"
2021/08/22 19:13:52 DEBUG : found headers:
2021/08/22 19:13:52 DEBUG : Creating backend with remote "/mnt/hdd/rclone/Test/current/"
2021/08/22 19:13:52 DEBUG : Creating backend with remote "/mnt/hdd/rclone/Test/backup/2021-08-22_191352"
2021/08/22 19:13:53 DEBUG : Local file system at /mnt/hdd/rclone/Test/current/: Waiting for checks to finish
2021/08/22 19:13:53 DEBUG : Local file system at /mnt/hdd/rclone/Test/current/: Waiting for transfers to finish
2021/08/22 19:19:35 DEBUG : 20210807-191538.mp4: Reopening on read failure after 2618761246 bytes: retry 1/10: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:19:35 DEBUG : 20210807-191538.mp4: Reopen failed after 2618761246 bytes read: The start offset (2618761246) exceeded the size of the entity (2618761246): Sabre\DAV\Exception\RequestedRangeNotSatisfiable: 416 Requested Range Not Satisfiable
2021/08/22 19:19:35 NOTICE: 20210807-191538.mp4: Removing partially written file on error: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:19:36 ERROR : 20210807-191538.mp4: Failed to copy: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:19:36 ERROR : Local file system at /mnt/hdd/rclone/Test/current/: not deleting files as there were IO errors
2021/08/22 19:19:36 ERROR : Local file system at /mnt/hdd/rclone/Test/current/: not deleting directories as there were IO errors
2021/08/22 19:19:36 ERROR : Attempt 1/3 failed with 1 errors and: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:19:36 DEBUG : Local file system at /mnt/hdd/rclone/Test/current/: Waiting for checks to finish
2021/08/22 19:19:36 DEBUG : Local file system at /mnt/hdd/rclone/Test/current/: Waiting for transfers to finish
2021/08/22 19:25:18 DEBUG : 20210807-191538.mp4: Reopening on read failure after 2618761246 bytes: retry 1/10: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:25:18 DEBUG : 20210807-191538.mp4: Reopen failed after 2618761246 bytes read: The start offset (2618761246) exceeded the size of the entity (2618761246): Sabre\DAV\Exception\RequestedRangeNotSatisfiable: 416 Requested Range Not Satisfiable
2021/08/22 19:25:18 NOTICE: 20210807-191538.mp4: Removing partially written file on error: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:25:19 ERROR : 20210807-191538.mp4: Failed to copy: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:25:19 ERROR : Local file system at /mnt/hdd/rclone/Test/current/: not deleting files as there were IO errors
2021/08/22 19:25:19 ERROR : Local file system at /mnt/hdd/rclone/Test/current/: not deleting directories as there were IO errors
2021/08/22 19:25:19 ERROR : Attempt 2/3 failed with 1 errors and: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:25:19 DEBUG : Creating backend with remote "/mnt/hdd/rclone/Test/backup/2021-08-22_191352"
2021/08/22 19:25:20 DEBUG : Local file system at /mnt/hdd/rclone/Test/current/: Waiting for checks to finish
2021/08/22 19:25:20 DEBUG : Local file system at /mnt/hdd/rclone/Test/current/: Waiting for transfers to finish
2021/08/22 19:31:01 DEBUG : 20210807-191538.mp4: Reopening on read failure after 2618761246 bytes: retry 1/10: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:31:01 DEBUG : 20210807-191538.mp4: Reopen failed after 2618761246 bytes read: The start offset (2618761246) exceeded the size of the entity (2618761246): Sabre\DAV\Exception\RequestedRangeNotSatisfiable: 416 Requested Range Not Satisfiable
2021/08/22 19:31:01 NOTICE: 20210807-191538.mp4: Removing partially written file on error: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:31:02 ERROR : 20210807-191538.mp4: Failed to copy: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:31:02 ERROR : Local file system at /mnt/hdd/rclone/Test/current/: not deleting files as there were IO errors
2021/08/22 19:31:02 ERROR : Local file system at /mnt/hdd/rclone/Test/current/: not deleting directories as there were IO errors
2021/08/22 19:31:02 ERROR : Attempt 3/3 failed with 1 errors and: net/http: server replied with more than declared Content-Length; truncated
2021/08/22 19:31:02 INFO  :
Transferred:        7.317Gi / 7.317 GiByte, 100%, 7.321 MiByte/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:     17m10.6s

2021/08/22 19:31:02 DEBUG : 3 go routines active
2021/08/22 19:31:02 Failed to sync: net/http: server replied with more than declared Content-Length; truncated

Thank you for your help!

hello and welcome to the forum,

what rclone reported was Transferred: 7.317Gi
that was the amount of data rclone transferred from the remote to local.

if rclone fails to transfer a file, rclone will try three times.
note: 2.43GB x 3 is approximately 7.317GB
that can be seen, three times with Removing partially written file on error and then rclone trying again to transfer the file...
while you are figuring out the problem, perhaps use --retries=1

server replied with more than declared Content-Length
as to why this is happening, not sure.

Thank you very much for your fast answer! Indeed --retries=1
brings the total filesize back to 2.43GB

I mounted Nextcloud webdav through davfs and rsync-ed the Test directory:

mount -t davfs -o uid=raph https://***.your-storageshare.de/remote.php/dav/files/Backup/ /mnt/nextcloud

rsync -avHh --progress /mnt/nextcloud/Test /mnt/hdd/rsync
sending incremental file list
Test/
Test/20210807-191538.mp4
          2.62G 100%   67.86MB/s    0:00:36 (xfr#1, to-chk=0/2)

sent 2.62G bytes  received 39 bytes  7.01M bytes/sec
total size is 2.62G  speedup is 1.00

ls -la /mnt/hdd/rsync/Test/
insgesamt 2557400
-rw-r--r-- 1 raph raph 2618761246 Aug 22 17:27 20210807-191538.mp4

So rsync was able to copy the complete file without errors.
The error must be in rclone then. Any idea how I should continue the debugging process?

To troubleshoot an issue like this I would first try to find out whether it is an issue with the rclone client or the Nextcloud server.

To do this I would first try the download with another webdav client, e.g. cyberduck:
https://cyberduck.io/webdav/

I see you tried with davfs, but I would still try a more generic tool similar to rclone (that is cyberduck).

Then I would try to download the same file from another webdav server, e.g. rclone serve webdav ./localfolder/
https://rclone.org/commands/rclone_serve_webdav/

I would take a look at the communication with the --dump flags.

@asdffdsa You can take it from here if you fancy, I am busy with other things.

not a fan of webdav, sometimes supports modtimes, sometimes support hash.
is there another protocol to use?

not familair with rsync, how does it verify a file transfer, does it create a hash of the source file, create a hash of the dest file and compare them?

imho, the only way to know is to

  • take a local file, get a hash such as md5.
  • upload the file to remote
  • download the file to local using another file name and compare the hashes..

for a more detailed log, perhaps add
--dump=headers --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file=rclone.log

sure but i have little experience with webdav and really do not like it....

Me neither, I guess this is an invitation for others to join if they fancy.

What that means is that the server said it was going to send x bytes, but it actually sent more than x bytes.

If you run the command with -vv --dump headers --retries 1 can you post the output?

Sorry for the inconvenience! I also hate Webdav but it is the only way to access Nextcloud on Hetzner without the nextcloud client.
I tried a lot, but gave up in the end. I abandoned Hetzner and moved all my files to my local self-build arm-nas. I just tried Rclone and it works now without error, so I think it a missconfigured webserver at Hetzner is the cause of the problem.
Thanks for all your help and time!

sure, glad to help,

as per the docs, rclone does support hash and modtime on nextcloud.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.