Google f1-micro (1 vCPU, 0.6 GB memory, 30GB HDD) Settings

[Rakeen70210 (Rakeen Huq)]

What is the problem you are having with rclone?

Optimal settings for use on a low resource vps and how to purge the vfs cache. Since I am limited on storage I would like to limit the size of the "vfs" folder so that does not consume all of my storage when transferring large files such as 45GB files.

With the current settings I have I noticed the vfs folder continues to grow in size regardless of what --vfs-cache limits I impose.

The way I am using the mount is by directly downloading large files into it, through the use of Premiumize and aria2 to download files from Premiumize into the mount.

What is your rclone version (output from rclone version)

rclone v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 LTS (GNU/Linux 5.4.0-1011-gcp x86_64)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

sudo rclone mount gcrypt: /mnt/GDrive -vv --daemon --allow-other --cache-db-purge --buffer-size 0 --use-mmap --drive-chunk-size 64M --vfs-cache-mode writes --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit 1G --vfs-cache-max-size 10G

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = REMOVED
client_secret = REMOVED
scope = drive
token = REMOVED
root_folder_id = REMOVED

[gcache]
type = cache
remote = gdrive:/crypt

[gcrypt]
type = crypt
remote = gcache:
filename_encryption = standard
directory_name_encryption = true
password = REMOVED

A log from the command with the -vv flag

2020/05/27 16:31:40 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/05/27 16:31:41 DEBUG : gcache: wrapped gdrive:crypt at root 
2020/05/27 16:31:41 DEBUG : gcache: Purging the DB
2020/05/27 16:31:41 INFO  : gcache: Cache DB path: /root/.cache/rclone/cache-backend/gcache.db
2020/05/27 16:31:41 INFO  : gcache: Cache chunk path: /root/.cache/rclone/cache-backend/gcache
2020/05/27 16:31:41 INFO  : gcache: Chunk Memory: true
2020/05/27 16:31:41 INFO  : gcache: Chunk Size: 5M
2020/05/27 16:31:41 INFO  : gcache: Chunk Total Size: 10G
2020/05/27 16:31:41 INFO  : gcache: Chunk Clean Interval: 1m0s
2020/05/27 16:31:41 INFO  : gcache: Workers: 4
2020/05/27 16:31:41 INFO  : gcache: File Age: 6h0m0s
2020/05/27 16:31:41 DEBUG : Adding path "cache/expire" to remote control registry
2020/05/27 16:31:41 DEBUG : Adding path "cache/stats" to remote control registry
2020/05/27 16:31:41 DEBUG : Adding path "cache/fetch" to remote control registry
2020/05/27 16:31:42 DEBUG : gcache: wrapped gdrive:crypt at root 
2020/05/27 16:31:42 DEBUG : gcache: Purging the DB
2020/05/27 16:31:42 INFO  : gcache: Cache DB path: /root/.cache/rclone/cache-backend/gcache.db
2020/05/27 16:31:42 INFO  : gcache: Cache chunk path: /root/.cache/rclone/cache-backend/gcache
2020/05/27 16:31:42 INFO  : gcache: Chunk Memory: true
2020/05/27 16:31:42 INFO  : gcache: Chunk Size: 5M
2020/05/27 16:31:42 INFO  : gcache: Chunk Total Size: 10G
2020/05/27 16:31:42 INFO  : gcache: Chunk Clean Interval: 1m0s
2020/05/27 16:31:42 INFO  : gcache: Workers: 4
2020/05/27 16:31:42 INFO  : gcache: File Age: 6h0m0s
2020/05/27 16:31:42 DEBUG : Adding path "cache/expire" to remote control registry
2020/05/27 16:31:42 DEBUG : Adding path "cache/stats" to remote control registry
2020/05/27 16:31:42 DEBUG : Adding path "cache/fetch" to remote control registry
2020/05/27 16:31:42 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "mount" "gcrypt:" "/mnt/GDrive" "-vv" "--daemon" "--allow-other" "--cache-db-purge" "--buffer-size" "0" "--use-mmap" "--drive-chunk-size" "64M" "--vfs-cache-mode" "writes" "--vfs-read-chunk-size" "128M" "--vfs-read-chunk-size-limit" "1G" "--vfs-cache-max-size" "10G"]
2020/05/27 16:31:42 DEBUG : Cache remote gcache:: Services stopped
2020/05/27 16:31:42 DEBUG : Cache remote gcache:: Services stopped

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

have you tried to run rclone without gcache between gdrive and gcrypt?

rclone mount gcrypt: /mnt/GDrive -vv --daemon --allow-other --vfs-cache-mode writes

[Rakeen70210 (Rakeen Huq)]

I thought about it, but I read somewhere its useful when using the mount feature.

I can go ahead and try without cache to see what happens, will update here again with results.

[asdffdsa (jojothehumanmonkey)]

in all my years of using rclone mounts, never once did i use that cache.
sometimes i would use --vfs-cache-mode writes but thats it.

how much do you pay per month for the f1 micro?
how much do you pay per month for premiumize?

[calisro (Rob)]

@Rakeen70210

I thought about it, but I read somewhere its useful when using the mount feature.

I can go ahead and try without cache to see what happens, will update here again with results.

There are edge cases but most times no cache is better. Reddit and other places reference to use cache but that info is old.

If you want to reduce memory usage on the micro, i typically lower or set the --buffer-size to zero. I also use --use-mmap for better memory management.

Also you may just want to set a small swap file to give some cushion.

@asdffdsa

how much do you pay per month for the f1 micro?

Free

[asdffdsa (jojothehumanmonkey)]

free 24/7/365 year after year?

[calisro (Rob)]

Yes. Its free. There are other 'charges' if you can avoid them like egress/ingress. If you're just using this thing to transfer from/to google drives and other google services, you'll be in the free tier.

https://cloud.google.com/free/docs/gcp-free-tier

[asdffdsa (jojothehumanmonkey)]

i am curious about how much the OP pays per month for both f1-micro and premiumize, which is not free.

[Rakeen70210 (Rakeen Huq)]

I do not pay for the f1-micro, I am only using it for "ingress" traffic, so only to bring data from Premiumize into GDrive.

For Premiumize I paid $90 for 15 months, which is $6 per month

[Rakeen70210 (Rakeen Huq)]

Update:

So I am transferring a 54GB file now using command
rclone mount gcrypt: /mnt/GDrive -vv --daemon --allow-other --vfs-cache-mode writes --buffer-size 0 --use-mmap

However it looks like its continuously using up my disk space. I have paused the transfer as I only have 4.3GB remaining on the VPS. Should I disable --vfs-cache-mode writes?

Log:

2020/05/27 17:04:20 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "mount" "gcrypt:" "/mnt/GDrive" "-vv" "--daemon" "--allow-other" "--vfs-cache-mode" "writes" "--buffer-size" "0" "--use-mmap"]
2020/05/27 17:04:20 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/05/27 17:04:21 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "mount" "gcrypt:" "/mnt/GDrive" "-vv" "--daemon" "--allow-other" "--vfs-cache-mode" "writes" "--buffer-size" "0" "--use-mmap"]

[calisro (Rob)]

@Rakeen70210

--vfs-cache-mode writes

It'll cache the file first.

https://rclone.org/commands/rclone_mount/#vfs-cache-mode-writes

[asdffdsa (jojothehumanmonkey)]

thanks,

you might want to read this post,

Good Cheap VPS in the US

I'm looking for a good cheap VPS hosting in the US. I saw this question was asked in 2017. Looking for more recent experiences. I'm currently on OpenVZ servers with EthernetServers.com. But three times now over the last year, I have had a problem where rclone gets stuck in some kind of I/O wait and the server won't shut down and they have to bounce the entire node in order to free up my server which, of course, affects all other clients on that same node. The first two times this happened, th…

i get a seedbox for $11 per month.
ubunutu
1000GB storage
rclone with mount support
plex server
vpn
and a whole lot more.

[Rakeen70210 (Rakeen Huq)]

So for my use case would you recommend I set it to off or minimal? The only thing I am using the mount for is transferring files.

[calisro (Rob)]

Can I ask why you're using a mount? You really should be using copy/move/sync. They'll work better for transfers.

[Rakeen70210 (Rakeen Huq)]

Only reason I am using a mount is because I am using Premiumizer's web interface to manage my Premiumize cloud downloads.

Once a file is downloaded to my cloud I can use the interface to download it to a local folder. In this case my "local" folder is the GDrive mount.

I have used copy/move before without any problems, the only downside was that I needed to always open up a ssh shell and run the command manually. The benefit of the Premiumizer web interface is that I can do everything with just a single click. Also, I can share the web link with others and let them download into my GDrive as well.

[calisro (Rob)]

I see. and you don't have the local diskspace to hold them. You can try without vfs-cache. I'm not sure how premiumize will act.

[Rakeen70210 (Rakeen Huq)]

Looks like it works.. but it takes a few minutes for the transfer speed to ramp up, and even then its at most half of what the speed used to be. I will need to test out using vfs-cache minimal to see if there are any improvements.

On a side note, so if using cache is not useful anymore, should I be disabling even when I use a mount with Plex, or is that a situation where it can be used? I have a VPS with a pretty fast connection running my GDrive mount with Plex. It is currently using vfs-cache writes and I am using a cache remote wrapped around a gdrive remote.

[calisro (Rob)]

@Rakeen70210

I will need to test out using vfs-cache minimal to see if there are any improvements.

I dont think there will be. The difference really is that when you're copying to the mount, its immediately transferring and so you'll need to wait for that to complete. It also won't have retries in the same way so if the transfer fails you'll need to reupload.

@Rakeen70210

so if using cache is not useful anymore, should I be disabling even when I use a mount with Plex, or is that a situation where it can be used?

you don't need it for plex. That is a few year old problem. You may not even need vfs-cache-mode writes but that just depends on if/how your'e writing to it.

[Rakeen70210 (Rakeen Huq)]

Ok thank you for that information. I want to thank you both for being so helpful and responding quickly to my issue.

I really do appreciate it!

[calisro (Rob)]

Just a small correction here. You wont need --use-mmap if you set buffers to zero as there will be nothing for it to do. So you can remove that flag.