Hard disk full with writing

What is the problem you are having with rclone?

Hello when rclone which is connected to my gsuite transcodes a movie in my emby library, the hard disk is full in writing. which causes pauses I think. What improvement could I make to minimize hard drive usage. I have 2 samsung ssd hard drive in raid yet. Thank!

What is your rclone version (output from rclone version)

rclone v1.53.3 - os/arch: windows/amd64 - go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

windows 10

Which cloud storage system are you using? (eg Google Drive)

Workspace

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here
```rclone mount Gsuite: X: --cache-dir=C:\rclone\cache --allow-other --buffer-size 88M --dir-cache-time 4h --cache-info-age 12h --drive-chunk-size 256M --vfs-read-chunk-size 8M --vfs-read-chunk-size-limit 128M --vfs-cache-mode full --vfs-cache-poll-interval 1m --vfs-read-ahead SizeSuffix --vfs-cache-max-size 100G --vfs-cache-max-age=12h --user-agent="ams


#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

[Gsuite]
type = drive
scope = drive
token = 

[Gcrypt]
type = crypt
remote = Gsuitee:/Personnel
filename_encryption = standard
directory_name_encryption = true

[Gsuitee]
type = drive
scope = drive
token = 


Paste config here




#### A log from the command with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

Paste log here

hello,

not sure that the entire command got posted.

i would check each flag

  • --allow-other does nothing on windows.
  • --vfs-read-ahead SizeSuffix is not valid, there is not value
  • --cache-info-age 12h does nothing based on the posted config, as it is not using the depreciated cache remote.

as for a set of optimized set of flags for gdrive streaming, this is from a fellow rcloner
https://github.com/animosity22/homescripts/blob/master/systemd/rclone.service

emby has to download the movie to transcode it and will use the vfs cache for storage.
what if you remove the vfs flags, how does it perform?

Ok suppose I put the settings like this how can that help apart from removing unnecessary things? I wonder if the problem is not caused by the raid which is done every time there is new on the disk

Thanks!

--cache-dir=C:\rclone\cache --buffer-size 88M --dir-cache-time 1000h --drive-chunk-size 256M --vfs-read-chunk-size 8M --vfs-read-chunk-size-limit 128M --vfs-cache-mode full --vfs-cache-poll-interval 15s --vfs-cache-max-size 100G --vfs-cache-max-age=12h --user-agent="ams"

i gave you my suggestions.

  • if you are running out of disk space, do a test without the vfs flags.
    many rcloners do not use the vfs cache.
    if my memory is correct, rcloners like @VBB
    rclone mount Gsuite: X:

  • use the exact set of flags from that link i shared with you and test.
    after testing, if that is a problem, post the latest command and we can tweak it.

1 Like

I feel like I'm in the minority these days, but I've never had a reason to use the cache, old or new. I'm lucky to have consistent gig-speeds up and down, though. My current mount for Plex:

rclone mount --attr-timeout 1000h --dir-cache-time 1000h --poll-interval 0 --rc --read-only -v

It just works :slight_smile:

Ok thanks in the rcone line that I use I use vfs for faster launching of files in emby. I have 2 1TB disks in raid. with 800 GB free. What could I use or remove more exactly? is your data in the cloud?

Thank you!!

Data is on GDrive, server on VPS (east coast). My home connection (west coast) is 500/500. I have multiple streams going at pretty much all times of the day, from both the US and Europe. Start time is maybe two to three seconds via Plex.

For your cache, I'd probably remove everything except --vfs-cache-mode full and --vfs-cache-max-size 100G. Leave the rest at defaults and test from there.

please, oh great outlaw, preach the prime. :pray:

2 Likes

Thank you, I will try its settings :slight_smile: I noticed that with the chromecast and emby the film is completely transcoded which causes slowness

--attr-timeout 1000h --dir-cache-time 1000h --poll-interval 0 --rc --read-only -v --vfs-cache-mode full --vfs-cache-max-size 100G --user-agent="ams"

Thanks to you too for the leads!

because I mount crypt and google drive (2 drive finally) I had to remove --rc --read-only -v because it is only possible with one mount

you can use a custom port for each rclone mount
https://rclone.org/rc/#rc-addr-ip

--read-only - that is not a global flag; each rclone mount can use the flag.

I don't quite understand what rc is for?

Sorry, just got home. As for the preaching... :wink:

--rc is used to remote control the mount for priming, in my case. I use a separate command for that:

rclone rc vfs/refresh recursive=true --timeout 30m

This will "cache" the files and folders of the entire mount, so that Explorer access and Plex scans are much faster. Note that this is different from the rclone cache feature, and it will not take up any space. It simply reads the mount structure into memory. Once you un-mount, it is lost, and you'll have to prime again on subsequent mounts.

The reason I keep my Plex mount read-only is that it doesn't change unless I want it to. I upload, prime, and scan only once a day, so this works out great for me. I also use a separate, writeable mount for organizing (rclone mount -v).

1 Like

hi so if I understand correctly it is as good as the cache in the sense that the access or the launching of the files will be done more quickly? How can I integrate it into my assembly line?

@echo off
C:\rclone\rclone mount Gcrypt: y: --cache-dir=C:\rclone\cache --attr-timeout 1000h --dir-cache-time 1000h --poll-interval 0 --read-only --vfs-cache-mode full --vfs-cache-max-size 100G --user-agent="ams"

C:\rclone>rclone rc vfs/refresh recursive=true --timeout 30m
2021/03/08 12:29:17 Failed to rc: connection failed: Post "http://localhost:5572/vfs/refresh": dial tcp [::1]:5572: connectex:

Thank you!

the problem with that script is the mount command does not exit until the mount is killed.
so that vfs/refresh will never get executed until its too late.


to enable the ability to prime, add --rc to the mount command.


change the script to

start C:\rclone\rclone mount Gcrypt: y: --rc --cache-dir=C:\rclone\cache --attr-timeout 1000h --dir-cache-time 1000h --poll-interval 0 --read-only --vfs-cache-mode full --vfs-cache-max-size 100G --user-agent="ams"
timeout /t 60
C:\rclone\rclone rc vfs/refresh recursive=true --timeout 30m

I don't think it makes sense to use the actual cache along with a read-only mount, so you might want to remove --read-only, or try without cache. See what works better.

The priming probably won't have much, if any, effect on start times with Plex. It helps with scan times. How long does it take for files to launch? There should be no difference between transcoding and direct play/stream when it comes to Rclone.

You might want to describe your setup in more detail, including internet speeds, etc.

Thank you guys! I try this later I have to format my server in raid0 and synchronize everything my libraries. Except the tps limit do you have other things to avoid a ban from google? This is all I found to scan everything at once. It takes longer, but it works.

Thanks!

The last time I did an initial scan was about 3.5 years ago, and if I remember correctly, I did not use any particular flags on my mount then. I don't recall hitting the API limits, but things could be different now.

How big is your library? Back then, my scan took about two weeks, I think. I don't even want to know how long it would take now... :scream:

i know this is off-topic, but i am always curious about the need for raid0?