Prevent Nginx from eating storage space when reverse-proxy Rclone

Hellow everyone~
Here is the point:
myPC with rclone mount (Linux) >> VPS (Linux) >> GDrive
Because of the bad connection to GDrive in my local network, I need a VPS as a mediator to transfer traffic so that I could upload to and download from GDrive faster.
This is what I've already confirmed. I was restricted by ISP so I could not connect to GDrive in a high speed. VPS in the only way for me.

VPS was connected to GDrive by using rclone serve webdav
Nginx in VPS is using for reverse-proxy the rclone webdav with SSL
myPC was connected to VPS by using rclone mount
So, my local PC mount a WebDAV which was connected to GDrive.
The only usage of VPS is to transfer traffic.

But my VPS only have a small hard disk. Only 3GB free space left. So it could only transfer traffic to GDrive directly but not download things from my PC first and then upload to GDrive.
I've tried to upload a MP4 bigger than 3GB to WebDAV, and I found that VPS storage ran out step by step, and kind of like clean all things in storage and show 3GB free space again when I copying. In my client PC the copy is contiuouns without interrupt but the process start with 0% and continue to copy. (I found that with rclone -v stats 20s copy )

Then I check VPS with top precisely, and I found that during the process of copying, the Nginx is really busy.
So, in my case, I wanna point out that, my VPS webdav is runing like this:

[Unit]
Description=Rclone
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone serve webdav cc:/bus --addr 127.0.0.1:55555 \
 --umask 0000 \
 --buffer-size 32M \
 --dir-cache-time 12h \
 --vfs-read-chunk-size 64M \
 --vfs-read-chunk-size-limit 1G
Restart=on-abort
User=root

[Install]
WantedBy=default.target

And I use Nginx to get an SSL reverse-proxy of my Rclone. So I think, the problem is because there is a Nginx as a mediator, every thing uploaded to VPS will go into Nginx first and saved in VPS storage. When everything finished in Nginx, Nginx will start to transfer it into GDrive through rclone.

For the above reasons, I tried to rclone serve webdav by this service config (abandon nginx):

[Unit]
Description=Rclone
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone serve webdav cc:/bus --addr :55555 \
 --umask 0000 \
 --buffer-size 32M \
 --dir-cache-time 12h \
 --vfs-read-chunk-size 64M \
 --vfs-read-chunk-size-limit 1G
Restart=on-abort
User=root

[Install]
WantedBy=default.target

And mount the webdav in my PC with only http://ip:port/
Finnally, I could sometimes got my highest upload speed from my PC to GDrive through VPS.

But here is also a problem. My VPS seems like a little bit unsafe. Is there any way to keep using SSL but do not let Nginx save files in VPS storage?

Maybe I'm a little bit greedy. :laughing:

Here is my Nginx's config:

server {
    listen  443 ssl;
    ssl on;
    ssl_certificate       /etc/nginx/ssl/certificate.cer;
    ssl_certificate_key   /etc/nginx/ssl/certificate.key;
    ssl_protocols         TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers           HIGH:!aNULL:!MD5;
    server_name           MY.DOMAIN.COM;
    client_max_body_size 0;
    location / {
        proxy_redirect off;
        proxy_pass http://127.0.0.1:55555;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;

        # Show realip in v2ray access.log
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}

Best regards,
Thank you!

You will probably need to disable proxy_request_buffering

1 Like

I've already tried with a 4GB file.
You do a best job! It dose work!!! Coooooooooool!!
Though the transfer speed was a little bit impaired by SSL, VPS's free space will never be a problem.
Thanks a lot!!

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.