How to change the cache configuration of uploading?

cshenxi · March 16, 2020, 3:59pm

Hellow~ Thanks for clicking in.
In my case, I got a VPS as a bridge to connected with GDrive.
There are three points in my configuration.
myPC >> VPS >> GDrive
I got a really low speed in achieving GDrive, so I thought I could take a VPS as a bridge.
I got a high speed of uploading things to my VPS.
And I could upload from VPS to GDrive in high speed as well.

So, in my opinion, if I deploy rclone serve webdav/http in my VPS connected to GDrive, I could get a high speed of reaching GDrive.

But, here is the problem: my VPS storage space is reeeeaaaaally small.(3GB free only) So downloading is not a problem but uploading is.
I mount the webdav(VPS) in my PC and when I try to upload a single file bigger than 3GB, it will meet an error of .. kind of like Input/Output error(I'm not sure)

And if I upload several small piece of files to GDrive. I found that these files were kept in
> /root/.cache/rclone/vfs/CONFIGNAME/.

Dose it means that the files will transfer to VPS local storage first and then transfer to GDrive?
Is it possible to configure that the VPS(webdav) only transfer the traffic but not store files?
In another words, is there any way for me to upload a file bigger than 3GB to GDrive through VPS?

Configuration shown below:

The command you were trying to run

VPS:
root@v154264:~# cat /etc/systemd/system/rclone-webdav.service

[Unit]
Description=Rclone
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone serve webdav cc:/bus --addr 127.0.0.1:19843  --umask 0000  --buffer-size 32M  --dir-cache-time 12h  --vfs-read-chunk-size 64M  --vfs-read-chunk-size-limit 1G  --vfs-cache-mode writes
Restart=on-abort
User=root

[Install]
WantedBy=default.target

root@v154264:~# cat .config/rclone/rclone.conf

[cc]
type = drive
scope = drive
token = {"access_token":"XXXXXXXXXXXXXXXXXXXXXXXXXX"}
root_folder_id = XXXXXXXXXXXXXX

myPC:
cat /root/.config/rclone/rclone.conf

[test]
type = webdav
url = https://MY.DOMAIN.COM/
vendor = other

/usr/bin/rclone mount test:/ /home/cc/Desktop/test \
 --umask 0000 \
 --default-permissions \
 --allow-non-empty \
 --allow-other \
 --buffer-size 32M \
 --dir-cache-time 12h \
 --vfs-read-chunk-size 64M \
 --vfs-read-chunk-size-limit 1G

Please help me... Thanks!

thestigma · March 16, 2020, 11:40pm

So you have rclone running on the VPS and then just network-sharing from that to your PC? Is that correct? If what you are looking for is to optimize speed I highly doubt that this is the optimal way to go about it unless you have other reasons to want to run it on the VPS.

Yes - if --cache-mode-writes is enabled then any files uploaded via a mount will first be written to the disk on the system where rclone runs - then uploaded after it has finished writing. In other words, you must have enough temporary storage to at least fit the largest file you want to upload (but preferably a good bit more as a buffer).

Yes - you can remove--cache-mode writes to disable the cache (or --cache-mode off ). This will not cache anything being uploaded. However - this also means that compatibility suffers heavily and the mount will no longer be able to do everything that the OS expects a disk to be able to do. Additionally it will not be able to re-try failed uploads if they should occur. This will basically be fine for straight file-copies or syncs of files - but it would not at all be suitable for a mount that should be directly accessed by any application, as the majority of them will ask for more advanced actions that a mount can without a write-cache can not do (you can find spesific documentation about the limits on rclone.org , usage-->mount )

Ok, so now that I have explained the problem, let me suggest a practical solution...
I don't think you are creating what you actually want to create.
The only reason rclone on the VPS seems to be so fast to upload is exactly because you are essentially delivering the file to it's harddrive, so it buffering it for you and the speed you see is not the "real" upload. I highly doubt you can make it go faster on the VPS than you can from home - as all the same bandwidth limitations apply.

Instead - what you should probably do is optimize the settings, and then you might find it best to run rclone from home where you can afford to run --cache-mode writes (with all of it's benefits, as it is generally recommended to use).

For me to suggest spesific optimizations for you - please give me an idea og what datta you are trying to transfer that feels slow... Is it mostly large files above 64MB? Or lots of tiny files? What upload bandwidth do you have? Especially for transfer of larger files there is actually quite a lot we can do to make it go faster - and Gdrive can easily ingest close to 1Gbit with the right workload and rclone settings. Tons of tiny files is somehing Gdrive will never be very fast on though, due to other limitations in the Gdrive system (consider archiving together bunches of tiny files when this is practical to do)

Get back to me with answers once you have digested this

cshenxi · March 17, 2020, 1:02am

Hi~ Thanks for your answer. Really appreciate.
Firstly, I'll answer the questions you raised.

Basically, all the data I'm going to store in GDrive is video file. More than 200MB per file and maybe some will more than 10GB. No tiny files(if there are lots of tiny files you will suggest to use --cache-mode writes, right?).

The uploading bandwidth of my home PC is 50mbps/s, and VPS is bigger. The speed of uploading files in VPS to GDrive could reach at least 10MB/s. But in my home PC... 10KB/s.
I just tested the speed. There is a WebDAV server deployed in my local PC, too(Apache2). And I ssh into my VPS, use the command of wget to download mp4 from my home to VPS directly(no SSL, just ip+port). Things like: 371M, 6~8MB/s, 55s
In VPS, I also tried to cd into my GDrive directory(actually, I also use rclone mount to mount my GDrive in VPS). It spent 58s to finish. And I checked the directory of "/root/.cache/rclone/vfs/xxx". There is no cache file in it.
But if I upload files from PC to GDrive through VPS, speed could reach 2~3MB/s. Slow, but much bigger than 10KB/s.

Secondly, some question appeared after reading your answer. It's about --cache-mode.
Basically, it's about the different between download and upload.
My understanding is that:
Save cache file or not, only depends on the configuration in that computer. The server will never be influenced.
A rclone mount to B. B is GDrive or other WebDAV.
If A is using --cache-mode writes, when downloading, part of the file will download into A's cache(local disk) first, right?
But how about uploading? When I upload things from A to B, the copy of the file will be saved in A's cache(local disk) first and then upload to B?(Because the file is already in local disk fully, though not in cache, it will not really copy it to cache. Right?)
All of the things just because of the local(client) configuration of --cache-mode writes , so B is never influenced by A's configuration.
When I delete --cache-mode writes in A, downloading will only use 32MB buffer in memory due to the configuration of --buffer-size 32M, right?

In my case, --cache-mode writes is written in VPS configuration, so when I streaming a video in my PC from GDrive through VPS, things will stored temporarily in my VPS's cache but not my PC, PC will only store 32MB, right?
When I uploading things from my PC, things will stored in VPS as well, just because there is no file in VPS's local disk, so it cannot ignore the parameter of --cache-mode writes, and then cache it in VPS first and then upload to GDrive. Right?

Thanks for reading.. I am a REEEAAALLY newbie of rclone.

thestigma · March 17, 2020, 1:45am

Uploading large files to Gdrive, if you have 50Mbit/sec then you can max that out. That should not be an issue. Not unless you live in some very remote area of the world or something. Including overheads and such you can expect to achieve around 5,3 - 5,4MB/sec.

Try adding this to your config at home (under the drive section), and save of course.
chunk_size = 128M

Then run a test something like this:
rclone copy /parh/to/source yourgdriveremote:/test -P

Try this first (with no mount of webdav) so we can see that the basics work before we troubleshoot further. Very likely you should get full speed on this. We can work from there.

Before you ask, the chunk-size is important to maximize bandwidth utilization. If you know about details of TCP - you know that this protocol needs a few seconds to reach max speed (due to the way it auto-detects max speed). Normally files are uploaded in 8MB chunks, but on faster connections this means that we have barely started uploading a chunk before it is done and we need to start a new one - so the connection is most of the time "ramping up" and not at full speed. 128MB is what I consider ideal for anyone who can afford it, but 64MB is not far behind in my testing.

Warning!
chunk-size does use your RAM. 128MB will use (maximum) 128MB pr connection. So for example it can be 128MBx4=512MB with the default 4 connections.
make sure you have enough RAM, or else rclone will just crash
Remember you can turn down the number of connections too if needed with --transfers

thestigma · March 17, 2020, 2:10am

Hmm, no that doesn't really matter.
It is just that I can not do much to optimize for many tiny files. This will always be slow on Gdrive unfortunately because it can only start uploading 2-3 new files pr second (not the same as having 2-3 concurrent transfers - because you can have many more concurrent than this).
Big files you can increase performance for much more easily.

Correct. write caching is something rclone can do for you - not something you can do on the Gdrive server. The main reason cache is needed is to make mount very compatible and be able to emulate a real harddrive sort of. Cache can also be used to speed up access to frequently accessed files - but this is a topic for another day

the write cache only applies to files that are being written to the remote.
It does not apply when you reading from the remote.
The latter would be a read-cache. rclone doesn't really have this at the moment (although it will come soon according to Nick).

In this case - streaming from the Gdrive will need no storage space anywhere. Only a few MB in memory temporarily. If you download files this is also not cached anywhere. The only space you need is for the file you download of course

Let me reiterate just to be perfectly clear:
--cache-mode writes applies only to uploads

This is the only time the cache is used, but otherwise correct

I'm not sure I understand this part...

If you have --cache-mode writes enabled then to upload a 10GB file you will need at least 10GB temporary storage space on the VPS.
or...
You can disable the write cache and not need any space - but then the mount will have bad comparability with applications. If all you want to do is copy files to it and stream video then you can probably do this but just be aware of the limitations.

But as I said... you are barking up the wrong tree here. Trust me
I can almost guarantee we can get your full upload speed working directly from your own PC if you are willing to work with me and do some adjustments and tests.

I came here as a newbie too. No shame in being a newbie - as long as you are not a noob (ie. someone who doesn't want to learn). I will teach you, and the people here on the forum are all very friendly and helpful

(going to bed for now - will check in again tomorrow).

cshenxi · March 17, 2020, 2:32am

Thanks for your reply.
But things just as you thought. GDrive is not friendly to my network environment. Actually I lived in China, and luckily I could connect to Google without VPN. But the speed seems like restricted by ISP or other things. So using VPS to transfer the traffic is the best choice...

In this case, I'd like to try to delete --cache-mode writes in my VPS, and add --chunk-size in my home PC. Luckily my RAM of home PC is pretty enough.

I've understand the concept of --cache-mode but only one thing I need to confirm.
Imagine that we only have PC_A and GDrive.
GDrive was mounted by rclone in PC_A and PC_A do have the parameter --cache-mode writes.
At this time, I'm going to upload a 10GB MP4 to GDrive.
The MP4 is already in PC_A's local disk.
When I start to uploading, will rclone take another 10GB storage space to do the cache thing?
If no, what is the parameter --cache-mode writes doing now?

And, if I change --cache-mode writes to --cache-mode full, what will happen?

thestigma · March 17, 2020, 3:04am

Ah ok, I understand. I have seen a few cases where the network location is just really bad but...
I still recommend you just try the test directly in the terminal like I said. I just want to make sure that this is actually the problem and that it is not something else in the configuration. Try - it will only take a moment to check.

Actually, I missed that you were running rclone both for mount at home and on the VPS also.
I think in this case, what you can do is set --cache-mode off in the VPS (or just delete the flag), but use --cache mode writes on your home mount. I think this will be fine, because there is no point in caching it twice. Once the file can be transferred as a unit the webdav on the VPS does not need write-cache to transfer it to Gdrive. This should give you full compatibility but not use any space on the VPS (only your home PC).

Yes it does. But with default setting it will automatically remove the file after 1 hour. You can choose what should be the maximum size and how long to keep files cached (see mount documentation).

Right now I do not recommend using this, because it is badly implemented.
It will cache both reads and writes instead of just writes. The problem is that this can right now only do this by downloading the entire file first before accessing it (so files will not open fast and you can not stream). As I said - you probably don't want to use this - yet.

NCW (the main autor of rclone) is working on implementing some big improvements in the caching system however, so soon the "full" mode will have read+write caching without any of the downsides.
This mode will mostly be useful for people who have a lot of harddrive space and want to save bandwith and/or make access to files faster. Any files that have not changed and are already in the cache will be read from there instead (which is of course many times faster) - and the cache will be smart about keeping the files you use the most often and throwing away the cached files not used for a long time. But this is still in development - so keep your eyes open in the next few months if this sounds like something you would want

cshenxi · March 18, 2020, 2:03am

Hello, I tried your advice to connect to GDrive directly with the cache_size = 128M para in rclone config. But it doesn't work. It seems that the problem is because of ISP. I could hardly connect to Google.

And for me, the main part is uploading single files bigger than 3GB(VPS hard disk), so I tried to delete the para of --vfs-cache-mode writes in my VPS's webdav service.
But it doesn't work at all.
Though it never error with Input/Output error, I found that VPS storage ran out and kind of like clean all things in storage and show 3GB free space again when I copying. In my client PC the copy is contiuouns without interrupt but the process start with 0% and continue to copy. (I found that with rclone -v stats 20s copy )

Then I check VPS with top precisely, and I found that during the process of copying, the Nginx is really busy.
So, in my case, I wanna point out that, my VPS webdav is runing like this:

[Unit]
Description=Rclone
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone serve webdav cc:/bus --addr 127.0.0.1:55555 \
 --umask 0000 \
 --buffer-size 32M \
 --dir-cache-time 12h \
 --vfs-read-chunk-size 64M \
 --vfs-read-chunk-size-limit 1G
Restart=on-abort
User=root

[Install]
WantedBy=default.target

And I use Nginx to get an SSL reverse-proxy of my Rclone. So I think, the problem is because there is a Nginx as a mediator, every thing uploaded to VPS will go into Nginx first and saved in VPS storage. When everything finished in Nginx, Nginx will start to transfer it into GDrive through rclone.

For the above reasons, I tried to rclone serve webdav by this service config:

[Unit]
Description=Rclone
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone serve webdav cc:/bus --addr :55555 \
 --umask 0000 \
 --buffer-size 32M \
 --dir-cache-time 12h \
 --vfs-read-chunk-size 64M \
 --vfs-read-chunk-size-limit 1G
Restart=on-abort
User=root

[Install]
WantedBy=default.target

And mount the webdav in my PC with only http://ip:port/
Finnally, I could sometimes got my highest upload speed from my PC to GDrive through VPS.

But here is also a problem. My VPS seems like a little bit unsafe. Is there any way to keep using SSL but do not let Nginx save files in VPS storage?

Maybe I'm a little bit greedy.

Here is my Nginx's config:

server {
    listen  443 ssl;
    ssl on;
    ssl_certificate       /etc/nginx/ssl/certificate.cer;
    ssl_certificate_key   /etc/nginx/ssl/certificate.key;
    ssl_protocols         TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers           HIGH:!aNULL:!MD5;
    server_name           MY.DOMAIN.COM;
    client_max_body_size 0;
    location / {
        proxy_redirect off;
        proxy_pass http://127.0.0.1:55555;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;

        # Show realip in v2ray access.log
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}

Thanks for your reply. Appreciated!

cshenxi · March 18, 2020, 7:53am

I've already opened a new post~ Welcome your discussion

Prevent Nginx from eating storage space when reverse-proxy Rclone

thestigma · March 19, 2020, 12:25am

I am unfortunately not very well versed in using the webdav, and I know next to nothing about NGIX.
I think opening a new thread was the right thing to do here for you - as you need a Linux expert. Thankfully there are many of them here on the forum. I have barely adequate Linux knowledge myself as a daily Window user

But here is an idea though... can we not attack this problem from another angle and just make the whole system simpler by cutting out the need for the webdav?

How about if you just set up a VPN server on your VPS - so your home PC can connect to it.
Then your PC will be able to work with the network just the same as the VPS can.
As a result you should be able to run a very simple direct Gdrive remote with write-cache on your own computer and the network signal will just pass through the VPS (with no middle-storage).

I believe there are many secure and fast VPN server software packages you can install, and typically this is not very difficult either.
I would really recommend giving this a shot instead of trying to fix your current system that is overly-complex (and therefore more easily can have problems).

cshenxi · March 19, 2020, 4:06am

Yes, you're right.
I've tried to deploy an VPN on my VPS and connect to it from my PC. As you expected, the speed was improved when I mount GDrive in my home PC directly.
And luckily, I've also solved the problem of NGINX.

So, now I have 2 plans to do so. Both of them are fine.

Thanks for your help ~ The problems are totally sovled.

thestigma · March 19, 2020, 4:27am

Cool! Very glad I could help you solve this.
I very much feel for you if you are in area where network gets throttled for various reasons.

Combined with using the tweaked chunk-size what speed do you get now? Pretty close to your estimated maximum of about 5,3 - 5,4MB/sec - right?

cshenxi · March 20, 2020, 5:09pm

actually, I didn't found any significant difference between having chunk-size or not. Maybe the improvement is small so that I could not aware it. Or, it's just because my uploading speed only could reach 5-6MB/s, which is not high enough so that the memory chunk is not that useful.

thestigma · March 20, 2020, 5:35pm

This is true - the default 8M chunk size is a larger problem the more bandwidth you are trying to use.
I would still recommend at least 32M in your case though, or 64M just tp optimize throughput.

Note that this only affects uploads - and only when using a Gdrive remote (so if you are running it via the webdav then it is the VPS's Gdrive remote that should have this setting changed - while if you run it via VPN from home then it should be enabled in the Gdrive remote at home).

I think you should be able to see a clear difference even on your connection - but probably not a huge difference.

PS: Due to high loads right now (due to Corona) on Google and many other cloud providers, the transfer rates are very unstable right now. It is not an ideal time to make exact speed-tests because performance seems to fluctuate:

system · May 20, 2020, 1:35pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.