Serve http behavior when the size is unknown?

Hello! I am currently writing a new backend for myself, and in this case, the size of the file that I want to download is unknown. In order to notate this, I've created my implementation of DirEntry.Size() to return -1:

func (o *Object) Size() int64 {
	return -1

For many operations, such as rclone cat, this is not a problem. However, for the serve http command, the Content-Length header is always set to 0, causing the browser to not download the file.

This is a little bit curious: I actually tried removing all instances where we are setting the Content-Length header in the httplib package, and yet the header is still being set!

Thus, my questions are:

  • Is it expected behavior that serve http requires a Content-Length header?
  • Is it a supported behavior for the remote object size to be unknown (if so, is there an example remote which I could look at to see how it is being handled?)
  • What would be the best way to handle this case?

If it helps, I am running:

rclone v1.54.0-DEV
- os/arch: darwin/amd64
- go version: go1.14.6

Thanks so much!

That is correct

Hmm... that looks like a bug... Try this

v1.54.0-beta.4978.045aff9e5.fix-serve-http-unknown-len on branch fix-serve-http-unknown-len (uploaded in 15-30 mins)

Apparently http.ServeContent can't cope with unknown length files.

The updated code works! Interestingly, the Content-Length header is still being set (on a small file, 13 bytes), however I have not investigated the implementation of the ResponseWriter.

Relatedly, I also see a similar error when running with cmount:

When I run this:

go run -tags cmount cmount sk: ./mnt/

then the files show as 0 bytes, and have a length of 0 when I open them. I'm not sure whether this is a requirement for mounting on MacOS, but the code implies that cmount is intended able to accommodate this case (maybe not on my platform.)

Let me know if I can run other tests? I can try your branch with a more comprehensive set of files with larger sizes.

It sets the content length on files it knows the length of.

I merged that fix into master.

File systems really like to know the length of files!

There is a special workaround for linux which makes it work - it should work for macOS according to the commit message. It depends on how you open the file - try using cat > otherfile on it.

If you want to unknown length files with a more established backend, try a google doc on google drive, or the google photos backend.

Awesome, thank you for fixing and merging HTTP! And also for the explanatory material for cmount. (I am reminded of Linus' colorful remarks about O_DIRECT :slight_smile:

1 Like

He he!

Linus doesn't like O_DIRECT does he!

The thing that has always disturbed me about O_DIRECT is that the whole
interface is just stupid, and was probably designed by a deranged monkey
on some serious mind-controlling substances

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.