Some questions about caching and a strange INFO

Schudtweb · July 6, 2019, 7:00am

Hi,

first of all my config:

[GDrive01]
type = drive
client_id =
client_secret =
scope = drive
token =

[GDrive02]
type = drive
client_id =
client_secret =
scope = drive
token =

[GCache01]
type = cache
remote = GDrive01:
chunk_size = 20M
info_age = 3d
chunk_total_size = 500G

[GCache02]
type = cache
remote = GDrive02
chunk_size = 20M
info_age = 3d
chunk_total_size = 500G

[GCrypt01]
type = crypt
remote = GCache01:
password =
password2 =

[GCrypt02]
type = crypt
remote = GCache02:
password =
password2 =

RClone mount

rclone mount --config=/home/rclone/.config/rclone/rclone.conf --allow-other --drive-use-trash --fast-list --dir-cache-time=72h --drive-chunk-size 64M --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit off --vfs-cache-mode writes --log-file "/home/rclone/logXX.log" --log-level INFO --cache-tmp-upload-path "/home/rclone/.tmp-cacheXX" --cache-tmp-wait-time=5m GCrypXX: /mnt/GCryptXX

MergerFS options

defaults,allow_other,use_ino,fsname=mergerFS

Now my questions

What is better? A cache drive or the vfs options? Or can i use both? Or what is all about?
I made a samba share and share /mnt/Pool. I want to copy alot of files to it and want it to transferred in the background. With the cache-tmp-upload-path and the wait-time its exactly what i want isnt it? I tried it with and without them and with them the copy is a lot faster.
Before i had the cache remotes, the mergerfs mount showed me 2PB. Now i see only 1PB. A df -h /mnt/GCryptXX shows me 1PB for 01 and 1,5TB for 02. 1,5TB is the size of the HDD, where my ubuntu is installed... The log dont say something about not working, i just get one message i cant interpret:
INFO : Encrypted drive 'GCrypt02:': poll-interval is not supported by this remote
What does it mean?
GCrypt01 dont show this message and the mount is exactly the same in my eyes? Where is my mistake?
It works correctly when i mount it directly in the shell (1PB). It doesnt work when i use my systemd service (1.5T).

Can someone help me?

Best regards,
Tobias

Animosity022 · July 6, 2019, 9:42am

You haven't told us what you are using it for so it's hard to tell whats better. I use Plex so I don't use cache.

I wouldn't recommend sharing using samba as it's extra layers of complexity and would mount to whatever the device is that I want to copy or use rclone copy over writing to the mount. It's easier and I always try to keep the most simple setup without adding layers on top of things. Samba/Rclone aren't really a great mix.

You'd have to share your full mount log with some debug to understand so use -vv and share the log of the mount.

Schudtweb · July 6, 2019, 10:49am

Here's the debug log mounted with systemd:

2019/07/06 10:39:15 DEBUG : rclone: Version "v1.48.0" starting with parameters ["/usr/bin/rclone" "mount" "--config=/home/rclone/.config/rclone/rclone.conf" "--allow-other" "--drive-use-trash" "--fast-list" "--dir-cache-time=72h" "--drive-chunk-size" "64M" "--vfs-read-chunk-size" "32M" "--vfs-read-chunk-size-limit" "off" "--vfs-cache-mode" "writes" "--log-file" "/home/rclone/log02.log" "--log-level" "DEBUG" "--cache-tmp-upload-path" "/home/rclone/.tmp-cache02" "--cache-tmp-wait-time=5m" "GCrypt02:" "/mnt/GCrypt02"]
2019/07/06 10:39:15 DEBUG : Using config file from "/home/rclone/.config/rclone/rclone.conf"
2019/07/06 10:39:15 DEBUG : GCache02: wrapped local:/GDrive02 at root
2019/07/06 10:39:15 INFO : GCache02: Cache DB path: /tmp/rclone/cache-backend/GCache02.db
2019/07/06 10:39:15 INFO : GCache02: Cache chunk path: /tmp/rclone/cache-backend/GCache02
2019/07/06 10:39:15 INFO : GCache02: Chunk Memory: true
2019/07/06 10:39:15 INFO : GCache02: Chunk Size: 20M
2019/07/06 10:39:15 INFO : GCache02: Chunk Total Size: 500G
2019/07/06 10:39:15 INFO : GCache02: Chunk Clean Interval: 1m0s
2019/07/06 10:39:15 INFO : GCache02: Workers: 4
2019/07/06 10:39:15 INFO : GCache02: File Age: 3d
2019/07/06 10:39:15 INFO : GCache02: Upload Temp Rest Time: 5m0s
2019/07/06 10:39:15 INFO : GCache02: Upload Temp FS: /home/rclone/.tmp-cache02
2019/07/06 10:39:15 DEBUG : Adding path "cache/expire" to remote control registry
2019/07/06 10:39:15 DEBUG : Adding path "cache/stats" to remote control registry
2019/07/06 10:39:15 DEBUG : Adding path "cache/fetch" to remote control registry
2019/07/06 10:39:15 DEBUG : GCache02: wrapped local:/GDrive02 at root
2019/07/06 10:39:15 INFO : GCache02: Cache DB path: /tmp/rclone/cache-backend/GCache02.db
2019/07/06 10:39:15 INFO : GCache02: Cache chunk path: /tmp/rclone/cache-backend/GCache02
2019/07/06 10:39:15 INFO : GCache02: Chunk Memory: true
2019/07/06 10:39:15 INFO : GCache02: Chunk Size: 20M
2019/07/06 10:39:15 INFO : GCache02: Chunk Total Size: 500G
2019/07/06 10:39:15 INFO : GCache02: Chunk Clean Interval: 1m0s
2019/07/06 10:39:15 INFO : GCache02: Workers: 4
2019/07/06 10:39:15 INFO : GCache02: File Age: 3d
2019/07/06 10:39:15 INFO : GCache02: Upload Temp Rest Time: 5m0s
2019/07/06 10:39:15 INFO : GCache02: Upload Temp FS: /home/rclone/.tmp-cache02
2019/07/06 10:39:15 DEBUG : Adding path "cache/expire" to remote control registry
2019/07/06 10:39:15 DEBUG : Adding path "cache/stats" to remote control registry
2019/07/06 10:39:15 DEBUG : Adding path "cache/fetch" to remote control registry
2019/07/06 10:39:15 DEBUG : Encrypted drive 'GCrypt02:': Mounting on "/mnt/GCrypt02"
2019/07/06 10:39:15 INFO : Encrypted drive 'GCrypt02:': poll-interval is not supported by this remote
2019/07/06 10:39:15 DEBUG : vfs cache root is "/tmp/rclone/vfs/GCrypt02"
2019/07/06 10:39:15 DEBUG : Adding path "vfs/forget" to remote control registry
2019/07/06 10:39:15 DEBUG : Adding path "vfs/refresh" to remote control registry
2019/07/06 10:39:15 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2019/07/06 10:39:15 DEBUG : : Root:
2019/07/06 10:39:15 DEBUG : : >Root: node=/, err=
2019/07/06 10:39:24 DEBUG : /: Attr:
2019/07/06 10:39:24 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2019/07/06 10:39:24 DEBUG : : Statfs:
2019/07/06 10:39:24 DEBUG : : >Statfs: stat={Blocks:396070573 Bfree:391644152 Bavail:371507436 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2019/07/06 10:40:15 DEBUG : Cache remote GCache02:: starting cleanup
2019/07/06 10:40:15 DEBUG : Cache remote GCache02:: starting cleanup

Here with manual mount:

2019/07/06 10:41:43 DEBUG : rclone: Version "v1.48.0" finishing with parameters ["/usr/bin/rclone" "mount" "--config=/home/rclone/.config/rclone/rclone.conf" "--allow-other" "--drive-use-trash" "--fast-list" "--dir-cache-time=72h" "--drive-chunk-size" "64M" "--vfs-read-chunk-size" "32M" "--vfs-read-chunk-size-limit" "off" "--vfs-cache-mode" "writes" "--log-file" "/home/rclone/log02.log" "--log-level" "DEBUG" "--cache-tmp-upload-path" "/home/rclone/.tmp-cache02" "--cache-tmp-wait-time=5m" "GCrypt02:" "/mnt/GCrypt02"]
2019/07/06 10:41:44 DEBUG : Cache remote GCache02:: Services stopped
2019/07/06 10:41:45 DEBUG : Cache remote GCache02:: Services stopped
2019/07/06 10:42:05 DEBUG : rclone: Version "v1.48.0" starting with parameters ["rclone" "mount" "--config=/home/rclone/.config/rclone/rclone.conf" "--allow-other" "--drive-use-trash" "--fast-list" "--dir-cache-time=72h" "--drive-chunk-size" "64M" "--vfs-read-chunk-size" "32M" "--vfs-read-chunk-size-limit" "off" "--vfs-cache-mode" "writes" "--log-file" "/home/rclone/log02.log" "--log-level" "DEBUG" "--cache-tmp-upload-path" "/home/rclone/.tmp-cache02" "--cache-tmp-wait-time=5m" "GCrypt02:" "/mnt/GCrypt02"]
2019/07/06 10:42:05 DEBUG : Using config file from "/home/rclone/.config/rclone/rclone.conf"
2019/07/06 10:42:05 DEBUG : GCache02: wrapped local:/home/rclone/GDrive02 at root
2019/07/06 10:42:05 INFO : GCache02: Cache DB path: /home/rclone/.cache/rclone/cache-backend/GCache02.db
2019/07/06 10:42:05 INFO : GCache02: Cache chunk path: /home/rclone/.cache/rclone/cache-backend/GCache02
2019/07/06 10:42:05 INFO : GCache02: Chunk Memory: true
2019/07/06 10:42:05 INFO : GCache02: Chunk Size: 20M
2019/07/06 10:42:05 INFO : GCache02: Chunk Total Size: 500G
2019/07/06 10:42:05 INFO : GCache02: Chunk Clean Interval: 1m0s
2019/07/06 10:42:05 INFO : GCache02: Workers: 4
2019/07/06 10:42:05 INFO : GCache02: File Age: 3d
2019/07/06 10:42:05 INFO : GCache02: Upload Temp Rest Time: 5m0s
2019/07/06 10:42:05 INFO : GCache02: Upload Temp FS: /home/rclone/.tmp-cache02
2019/07/06 10:42:05 DEBUG : Adding path "cache/expire" to remote control registry
2019/07/06 10:42:05 DEBUG : Adding path "cache/stats" to remote control registry
2019/07/06 10:42:05 DEBUG : Adding path "cache/fetch" to remote control registry
2019/07/06 10:42:05 DEBUG : GCache02: wrapped local:/home/rclone/GDrive02 at root
2019/07/06 10:42:05 INFO : GCache02: Cache DB path: /home/rclone/.cache/rclone/cache-backend/GCache02.db
2019/07/06 10:42:05 INFO : GCache02: Cache chunk path: /home/rclone/.cache/rclone/cache-backend/GCache02
2019/07/06 10:42:05 INFO : GCache02: Chunk Memory: true
2019/07/06 10:42:05 INFO : GCache02: Chunk Size: 20M
2019/07/06 10:42:05 INFO : GCache02: Chunk Total Size: 500G
2019/07/06 10:42:05 INFO : GCache02: Chunk Clean Interval: 1m0s
2019/07/06 10:42:05 INFO : GCache02: Workers: 4
2019/07/06 10:42:05 INFO : GCache02: File Age: 3d
2019/07/06 10:42:05 INFO : GCache02: Upload Temp Rest Time: 5m0s
2019/07/06 10:42:05 INFO : GCache02: Upload Temp FS: /home/rclone/.tmp-cache02
2019/07/06 10:42:05 DEBUG : Adding path "cache/expire" to remote control registry
2019/07/06 10:42:05 DEBUG : Adding path "cache/stats" to remote control registry
2019/07/06 10:42:05 DEBUG : Adding path "cache/fetch" to remote control registry
2019/07/06 10:42:05 DEBUG : Encrypted drive 'GCrypt02:': Mounting on "/mnt/GCrypt02"
2019/07/06 10:42:05 INFO : Encrypted drive 'GCrypt02:': poll-interval is not supported by this remote
2019/07/06 10:42:05 DEBUG : vfs cache root is "/home/rclone/.cache/rclone/vfs/GCrypt02"
2019/07/06 10:42:05 DEBUG : Adding path "vfs/forget" to remote control registry
2019/07/06 10:42:05 DEBUG : Adding path "vfs/refresh" to remote control registry
2019/07/06 10:42:05 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2019/07/06 10:42:05 DEBUG : : Root:
2019/07/06 10:42:05 DEBUG : : >Root: node=/, err=

Manual mount gives me 1P, systemd mount only 1.5T

I have a dedicated root server with windows. There runs a plex and an emby server. At the moment i use Stablebit CloudDrive to mount my GDrive and use it. But since a month its nearly unusable. A lot of download errors and forced unmounts. So i want to try rclone now and test, if i'll got the same problems or not.
RClone for windows worked also, but i want to pool two or more drives and Stablebit DrivePool (i use it at the moment and it work very well) can not handle the rclone mounts and i didnt find another software for pooling drives like rclone. So i installed VMWare Player and set up an Ubuntu VM. There i want to mount two rclone drives and pool it with mergerfs.
Without the cache drive it took 15min to copy 250 jpegs to the samba share. With the cache drive it's a lot faster. But only GCrypt01 has the correct size...

Animosity022 · July 6, 2019, 11:05am

Google Drive only allows 2-3 copies per second so writing directly to the drive with small files is always going to be slow.

Are you running Plex/Emby or copying files up (jpegs)? Not sure how the jpegs fit into your setup.

I use Plex/Emby and do not use the cache backend as it slows things down.

The issue is your missing a : on your 02 remote:

needs to be GDrive02:

You should try to reduce parameters unless you have a reason for changing them or adding them as the defaults are pretty spot on.

--fast-list does nothing on a mount so it can be removed.

Schudtweb · July 6, 2019, 11:33am

The f*cking : I would never saw this...
Thanks alot!!!

In the end, i want to have storage, where are all the files for plex/emby. So i want to read from it and write to it.
The jpegs was just testing. But yes, i have also jpegs. All the coverart and thumbnails and so one. I dont let plex/emby scrape, i scrape it myself with Ember Media Manager. So i have full control about it. And so there are many, many jpegs, nfos and so on to move to the rclone storage.

I just know it from Stablebit CloudDrive. There you have a cache drive for each clouddrive. When you move files to the clouddrive, first they are written to the cache and then CloudDrive upload them to google (Upload amount 10MB or longer then 5min in the cache).
Thats what i try with rclone. And i thought the cache drive its doing this. But im not quite sure

Animosity022 · July 6, 2019, 11:39am

I rather keep my metadata local. Having remote covertart/etc just seems like it would slow things down too much.

23G local on SSD makes Plex really work well and for almost 70TB of stuff, I'm fine with that ratio of local storage on SSD.

[felix@gemini Plex Media Server]$ du -ms
23710	.
[felix@gemini Plex Media Server]$ rclone about GD:
Used:    68.739T
Trashed: 23.785G
Other:   129.377M

I go with use the right tool for the right job. Cloud storage doesn't do well with reading many little files as you have the inherent latency of going to the cloud and back.

Schudtweb · July 6, 2019, 6:21pm

Plex and Emby dont read everything new. They cache the artwork as well. My Plex folder is about 30GB. Emby is smaller, but caches too. Only the files are in the cloud, the databases are local.
Since a month ago it was no problem to play files with them. I'll test it, if it works with rclone too or not. But i keep it in mind, what you said

Animosity022 · July 6, 2019, 7:31pm

Every time a client loads up a show or a new player connects, it will check if the metadata for that show exists and displays it. A client may cache part of the data here/there but it isn't a server action, but a client/player action.

So you split out your metadata folder or what folders did you move to the cloud?

[root@gemini Plex Media Server]# ls -al
total 24
drwxr-xr-x 11 felix felix  203 Jul  5 20:23  .
drwxr-xr-x  3 plex  plex    31 Apr 12 11:28  ..
drwxr-xr-x  6 felix felix 4096 Jul  6 15:29  Cache
drwxr-xr-x 11 felix felix 4096 Jul  3 20:23  Codecs
drwxr-xr-x 20 felix felix 4096 Jul  3 20:19 'Crash Reports'
drwxr-xr-x  2 felix felix    6 Feb  8 17:40  Diagnostics
drwxr-xr-x  3 felix felix 4096 Jul  6 15:19  Logs
drwxr-xr-x  3 felix felix   23 Feb  8 17:41  Media
drwxr-xr-x  6 felix felix   65 Feb  8 17:51  Metadata
-rw-r--r--  1 felix felix    4 Jul  3 20:23  plexmediaserver.pid
drwxr-xr-x  6 felix felix   97 Jun 22 00:37  Plug-ins
drwxr-xr-x  7 felix felix   96 Feb  8 17:40 'Plug-in Support'
-rw-------  1 felix felix 2031 Jul  5 20:23  Preferences.xml

system · October 4, 2019, 7:31pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.