Wait, and how does the mega tactic work exactly in my case?
Basically I'm brainstorming for you to see if I can come up with something that will work for you.
Basically it is a end-to-end encrypted cloud service that provides the ability to access it via webdav. Not to the world but to the webdav server running on the PC with the client software installed.
The other option is to use your existing cloud storage and pair it with rclone serve and caddy if you need SSL.
I already have a ready to use WebDAV server. But I just want my files to be stored encrypted on the server. But the most important thing is that I don't want to use cache because I don't have decent hard drives. Idk what we could do with mega.nz. but maybe we can use that as some kind of caching? Idk a lot about it either.
Ok, so everything is on a webdav server but not in an encrypted state.
Is it your intention to make the files available internally only or accessible via the web as well?
The normal process would be to start with a blank server and create two remotes.
- Webdav to access the server.
- Crypt based on #1 that handles the encryption and de-encryption.
Starting with a blank server you would copy everything up to the server using the second remote and that will encrypt everything based upon the parameters you set when you created the remote. For example you can choose not to encrypt the file names or folder names but just put an extension on the end or you can encrypt everything.
If the server is full of unencrypted data then depending on storage space available you would either create a folder to put all that stuff into so you can upload everything encrypt it again or you would move everything off the server cleaning the server out completely and then re-uploading everything to the server using the Crypt remote.
You would not at this point want to access the webdav server directly anymore you would want to go through rclone serving through whatever protocol you choose including webdav. This way rclone will decrypt the files on the Fly.
Here are some additional notes:
Why do you need the intermediate “proxy" server?
It's an existing WebDAV server created using TGFS ( GitHub - TheodoreKrypton/tgfs: Telegram becomes a WebDAV server · GitHub ). My goal is to add real-time encryption in this existing WebDAV server without using any cache (or optionally 10 GB cache, but with hard limit, it isn't allowed to go even a single byte over it). I want to access my files using SFTP, so the encrypted remote needs to be served over SFTP.
What do you mean?
rclone sftp supports StreamUpload, should be able to upload without using local storage.
bonus, rclone can do file transfer verification using checksums.
or if you must upload to webdav, which does not support StreamUpload,
then spend $60.00 for an external usb drive to use for rclone's temporary storage.
or pay for cloud storage, i use storagebox which offers sftp, webdav and much more for $2.60/TB/month
Sounds like the webdav server is internet routable?
Do you want the encrypted SFTP files available over the Internet or locally because those are two animals that are very different?
Step One is you're going to have to encrypt all the data on the webdav server. This takes two remotes.
Then on the pc/server that will “serve” the files over sftp you must mount the webdav server to a folder using the Crypt remote so the files are readable.
Then you run another instance of rclone serving the decrypted webdav files over sftp.
I don't recommend it but I have tested port forwarding to expose the SFTP files to the internet. You can and should give it a password so it will have password protection even if you serve it over the web.
Did I miss anything?
Given you are creating the server using tgfs, why you don't run tgfs in your own machine and connect rclone (with crypted wrapped WebDAV) connected to the tgfs server in localhost?
What I don't understand is why you need the intermediate “server proxy"
fwiw, never run a rclone serve over the open internet.
hide the server behind tailscale, wireguard, or other vpn
rclone serve sftp will auto-create a key file instead of not secure password.
i did a quick test with a rclone copy.
did not see that rclone was buffering locally?
afiak, that is what the OP is already doing.
the issue is rclone mount with webdav, needs the vfs file cache and the OP cannot afford to upgrade the craptop's 215GB internal hard drive or purchase an external usb drive.
Lol, craptop. But you are right. It's not going to be publicly available on the internet, it's going to be behind tailscale.
And I want to achieve streamUpload on my WebDAV server somehow
about the server, is it just webdav or do you also have ssh access?
i would test rclone copy and see what happens
Why do you need tailscale in the equation?
I only have WebDAV access. Nothing more. I need tailscale for accessing my server because I don't want to port forward on my router
If you are running tgfs in your own machine, in 99% of cases you don't need to open any port in your router
@VulcanoSoftware is suggesting the server is using an internal IP address That's not internet routable. Tail scale is a great option in this situation..
It does add a little overhead that makes things a bit slower so after doing that for a while I switched to a home router that provided a built-in VPN that runs faster the tail scale.