Hello! I am currently writing a new backend for myself, and in this case, the size of the file that I want to download is unknown. In order to notate this, I've created my implementation of DirEntry.Size() to return -1:
func (o *Object) Size() int64 {
return -1
}
For many operations, such as rclone cat, this is not a problem. However, for the serve http command, the Content-Length header is always set to 0, causing the browser to not download the file.
This is a little bit curious: I actually tried removing all instances where we are setting the Content-Length header in the httplib package, and yet the header is still being set!
Thus, my questions are:
Is it expected behavior that serve http requires a Content-Length header?
Is it a supported behavior for the remote object size to be unknown (if so, is there an example remote which I could look at to see how it is being handled?)
What would be the best way to handle this case?
If it helps, I am running:
rclone v1.54.0-DEV
- os/arch: darwin/amd64
- go version: go1.14.6
The updated code works! Interestingly, the Content-Length header is still being set (on a small file, 13 bytes), however I have not investigated the implementation of the ResponseWriter.
Relatedly, I also see a similar error when running with cmount:
When I run this:
go run -tags cmount github.com/rclone/rclone cmount sk: ./mnt/
then the files show as 0 bytes, and have a length of 0 when I open them. I'm not sure whether this is a requirement for mounting on MacOS, but the code implies that cmount is intended able to accommodate this case (maybe not on my platform.)
Let me know if I can run other tests? I can try your branch with a more comprehensive set of files with larger sizes.
It sets the content length on files it knows the length of.
I merged that fix into master.
File systems really like to know the length of files!
There is a special workaround for linux which makes it work - it should work for macOS according to the commit message. It depends on how you open the file - try using cat > otherfile on it.
If you want to unknown length files with a more established backend, try a google doc on google drive, or the google photos backend.
Awesome, thank you for fixing and merging HTTP! And also for the explanatory material for cmount. (I am reminded of Linus' colorful remarks about O_DIRECT
The thing that has always disturbed me about O_DIRECT is that the whole
interface is just stupid, and was probably designed by a deranged monkey
on some serious mind-controlling substances
