Feature request: HTTP remote should allow for reading hidden files/folders by name directly

Most directory listing modules on webservers (nginx, etc.) will not show hidden items - this basically means items that start with a period on Unix. However, the webservers will still gladly serve those files if they are directly requested.

For example, suppose there is a webserver running Nginx's standard autoindex module at https://www.example.com/files/. If you mount this URL as an http rclone remote, you can browse the files on the remote as expected.

However, assume there is now a hidden directory on the server at https://www.example.com/files/.extras/. If you visit that URL directly, you get a normal autoindex page listing the hidden directory's contents. However, if you try to access this directory on your rclone mount, you'll receive a "No such file or directory" error.

As expected, if you specify that URL directly as the URL for the http remote, it works, treating the root of the mount as that hidden folder within the HTTP server.

The same behavior applies to hidden files. If there is a file at https://www.example.com/files/.hidden.txt, you are unable to access that file on the HTTP remote mount, even if you specify it directly (e.g. M:\.hidden.txt or /mnt/rclone/.hidden.txt does not exist).

I propose an option to the http remote that separates the directory listing logic from the file access logic. Perhaps this is impossible as rclone is architected, or maybe FUSE just doesn't allow this, or maybe it's an OS thing, but ideally it would basically mean that file open requests and directory listing requests are parsed directly and passed to the HTTP server as-is, and the behavior depends on the response from the server - i.e. if a 404 is returned, then "no such file or directory" is produced.

This wouldn't matter for sync operations - it's fair to expect that a sync operation won't catch hidden files. However, if a user is aware of the hidden paths, it should be possible to directly address them in the path.

This also appears to be specific to the mount operation. If I do something like rclone ls --http-url https://www.example.com/files/ :http:/.hidden it works properly.

If it's an OS level issue, a potential alternative could be that the user could provide a listing of known hidden paths that they expect to be able to access. This could be as simple as a text file listing paths relative to the root that should be treated as if they exist. In the above example, the text file could contain:

.hidden.txt
.extras
.hidden

# and maybe others:
path/to/.hidden-folder
secretstuff/security-through-obscurity-is/.bad.txt

and those paths would work properly on the mount even though they're not included in the remote's directory listings. This is very easy to generate on Linux at least with the find command, and I'm sure there's an equivalent on Windows.

The OS won't open a directory it can't see in a directory listing - I think you'd have to get the hidden files showing up in the listing first.

What you are saying is that these directories would appear in the listings.

If rclone had to check for their existence that would slow down the directory listings somewhat.

If you just have a few of these then you can use the combine backend to build a directory listing with all the things in it