Volume 100% - What's the problem please?

Help and Support
1

#### What is the problem you are having with rclone?

Hello,

I am having a big problem with my server since installing Rclone and using it via Google Drive.

My Emby server is idling 99% of the time and doesn't want to display anything. The cause, my server volume is at 100% very often ...

I know the cause is from Rclone but I don't know how to fix it.

Rclone and Emby manages a very large library (around 90TB of media).

Could you help me please? Thank you

https://i.imgur.com/CDs9pcT.png

#### What is your rclone version (output from rclone version)

rclone v1.52.0

  • os/arch: linux/amd64
  • go version: go1.14.3

#### Which OS you are using and how many bits (eg Windows 7, 64 bit)

Synology DSM 6

#### Which cloud storage system are you using? (eg Google Drive)

Google Drive

The rclone config contents with secrets removed.

[remote]
type = drive
client_id = xxxxxxxxx-xxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
client_secret = xxxxxxxxxxxxxxxxxxxx
scope = drive
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-05-02T14:02:34.13462409+02:00"}
root_folder_id = xxxxxxxxxxxx

[cache]
type = cache
remote = remote:
chunk_size = 1M
info_age = 1d
chunk_total_size = 5G
db_path = /volume1/administration/rclone/config/cache
chunk_path = /volume1/administration/rclone/config/temp/rclone_chunk
chunk_clean_interval = 1h0m0s
read_retries = 10
workers = 16
chunk_no_memory = false
rps = 10
writes = true
tmp_upload_path = /volume1/administration/rclone/config/temp/rclone_upload
tmp_wait_time = 1m0s
db_wait_time = 1s

[mycrypt]
type = crypt
remote = cache:xxxxxxxxxxxxxxx
filename_encryption = standard
directory_name_encryption = true
password = xxxxxxxxxxxxxxxxxxx
password2 = xxxxxxxxxxxxxxxxxx

2

hi,

  • perhaps update to latest rclone which has a shiny vfs cache.
  • the cache backend is buggy, not maintained and is in the process of being depreciated.
  • are you using mount command, if so, you did not post it.
  • there are many settings in emby that can use a lot of cpu and network.
  • for streaming, i mount read only and do not vfs cache.
3

Hi asdffdsa,

What mount command do you have please?

Je vais mettre à jour Rclone mais je suis à peu près certain que ça ne règlera pas le soucis.

Et j'ai déjà regardé du côté d'Emby, même au repos, ça fait travailler le serveur mais c'est Rclone et Emby réuni.

Here is my mount command:

MountDir="/volume1/administration/rclone_mount"
RcloneConf="/root/.config/rclone/rclone.conf"
ChunkPATH="/volume1/administration/rclone/config/temp/rclone_chunk"

MountDir="/volume1/administration/rclone_mount"
RcloneConf="/root/.config/rclone/rclone.conf"
ChunkPATH="/volume1/administration/rclone/config/temp/rclone_chunk"

if [ ! -d "$MountDir/Temp" ]; then
rclone mount mycrypt: "$MountDir"
--config "$RcloneConf"
--allow-non-empty --allow-other --vfs-cache-mode writes --no-modtime
--log-level INFO --log-file /volume1/administration/rclone/logs/rclone.log
--cache-dir "$ChunkPATH"
--cache-db-path "$ChunkPATH"
--cache-chunk-path "$ChunkPATH" &
else
echo "Pre Mounting"
fi

4

i use the settings from this post, from @VBB,

1 Like
5

Ok Thanx for reponse.

6

So, just to understand the issue here: you're running rclone on a Synology NAS, with the old cache mode, it looks like. And you're saying that the NAS' CPU is constantly maxed at 100%. Are you also running Emby on that NAS? I have lots of experience with DSM, but I've never run anything on the NAS itself. It just doesn't sound like a good idea, especially with cache, but I could be wrong.

Try a simple mount without cache first, to see if that makes any difference.

7

Yes Emby and Rclone are launched on the DS918 + NAS
Emby via Docker with limitation of system resources.

I think there must be something wrong with my Rclone config that the volume is often at 100%.

I'm not talking about the processor, but rather the VOLUME use.

Example right away:

8

did you update your first post, changing the image.
the first time did not have timestamps and the new image does.

it seems that the spike happens every 10 hours.

emby or something is requesting data and rclone is downloading it into that cache.
90TB is a very large amount of data.

where the the emby database and files stored?

so as we suggested

  • update rclone
  • remove the cache remote
  • set a read only mount
  • check settings in emby, it does run on a schedule.
    on my emby, i have removed every schedule.
    i do not generate thumbnails, previews, scans for subtitles and such.
9

OK, I get it now. Check this out:

High volume use seems to be normal, according to several posts I've seen.

10

update rclone

Update (Rclone v1.53.1) performed but no change after reboot

remove the cache remote

Cache deleted but no change

set a read only mount

Already read-only normally

check settings in emby, it does run on a schedule.
on my emby, i have removed every schedule.
i do not generate thumbnails, previews, scans for subtitles and such.

Already done for a few days but that doesn't change anything.

did you update your first post, changing the image.
the first time did not have timestamps and the new image does.

No

OK, I get it now. Check this out:

Before installing and using Rclone, I didn't have this problem. And this concern grew as my media library on Google Drive grew too

11

seems clear that something is running x numbers of hours on a schedule, agreed?

can you post another screenshot the same as that first screenshot in the first post?

where is the emby server located?

how was emby accessing your media files in gdrive before rclone?

can you post the current rclone mount command and config file?

12

can you post another screenshot the same as that first screenshot in the first post?

For 24h

where is the emby server located?

Emby is installed on NAS Synology 918+ with Docker

Rclone is on NAS Synology 918+ (no Docker)

how was emby accessing your media files in gdrive before rclone?

Before Rclone, no Google Drive. Juste 30 To on 40 To disk on NAS Synology 916+ with share NFS.

can you post the current rclone mount command and config file?

Mount command

MountDir="/volume1/administration/rclone_mount"
RcloneConf="/root/.config/rclone/rclone.conf"
ChunkPATH="/volume1/administration/rclone/config/temp/rclone_chunk"

if [ ! -d "$MountDir/Temp" ]; then
rclone mount mycrypt: "$MountDir"
--config "$RcloneConf"
--allow-non-empty --allow-other --no-modtime --attr-timeout 1000h --dir-cache-time 1000h --poll-interval 0 --rc --read-only --exclude .recycle** --exclude .actors**
--log-level NOTICE --log-file /volume1/administration/rclone/logs/rclone.log
--cache-dir "$ChunkPATH"
--cache-db-path "$ChunkPATH"
--cache-chunk-path "$ChunkPATH" &
else
echo "Pre Mounting"
fi

Config

[remote]
type = drive
client_id = xxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
client_secret = xxxxxxxxxxxxxxxxxxxxxx
scope = drive
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxx"}
root_folder_id = xxxxxxxxxxxxxxxxxxxxxx

[cache]
type = cache
remote = remote:
chunk_size = 5M
info_age = 1d
chunk_total_size = 5G
db_path = /volume1/administration/rclone/config/cache
chunk_path = /volume1/administration/rclone/config/temp/rclone_chunk
chunk_clean_interval = 1h0m0s
read_retries = 10
workers = 16
chunk_no_memory = false
rps = 10
writes = true
tmp_upload_path = /volume1/administration/rclone/config/temp/rclone_upload
tmp_wait_time = 1m0s
db_wait_time = 1s

[mycrypt]
type = crypt
remote = cache:xxxxxxxxxxxxxxxxxxxxxxxx
filename_encryption = standard
directory_name_encryption = true
password = xxxxxxxxxxxxxxxxxxxx
password2 = xxxxxxxxxxxxxxxxxxx

13

as has been suggested a few times.

  • remove --allow-non-empty
  • as a test, do not use the buggy, deprecated cache remote.

have mycrypt point to remote, not cache

[mycrypt]
type = crypt
remote = remote:xxxxxxxxxxxxxxx

and use a command like from the command line

rclone mount mycrypt: "$MountDir"
--config "$RcloneConf"
--allow-other --no-modtime --attr-timeout 1000h --dir-cache-time 1000h --poll-interval 0 --rc --read-only --exclude .recycle** --exclude .actors**
--log-level NOTICE --log-file /volume1/administration/rclone/logs/rclone.log
14

I did everything and restart the NAS.
We will see the statistics by tomorrow, but for the moment, it's not good. The volume is 100% too :frowning:

Here is a screenshot over a year and we can see that the problem has been since the installation and use of Rclone (April 2020).

15

What is Volume Utilization actually showing?

16

can you post the updated

  • config file, redacting id and passwords
  • rclone mount command
17

What is Volume Utilization actually showing?

Sorry for my little english but i don't understand.

can you post the updated

Script command mount

MountDir="/volume1/administration/rclone_mount"
RcloneConf="/root/.config/rclone/rclone.conf"
ChunkPATH="/volume1/administration/rclone/config/temp/rclone_chunk"

if [ ! -d "$MountDir/Temp" ]; then
rclone mount mycrypt: "$MountDir"
--config "$RcloneConf"
--allow-other --no-modtime --attr-timeout 1000h --dir-cache-time 1000h --poll-interval 0 --rc --read-only --exclude .recycle** --exclude .actors**
--log-level NOTICE --log-file /volume1/administration/rclone/logs/rclone.log
--cache-dir "$ChunkPATH"
--cache-db-path "$ChunkPATH"
--cache-chunk-path "$ChunkPATH" &
else
echo "Pre Mounting"
fi

Config File

[remote]
type = drive
client_id = xxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
client_secret = xxxxxxxxxxxxxxxxx
scope = drive
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}
root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxxxx

[cache]
type = cache
remote = remote:
chunk_size = 5M
info_age = 1d
chunk_total_size = 5G
db_path = /volume1/administration/rclone/config/cache
chunk_path = /volume1/administration/rclone/config/temp/rclone_chunk
chunk_clean_interval = 1h0m0s
read_retries = 10
workers = 16
chunk_no_memory = false
rps = 10
writes = true
tmp_upload_path = /volume1/administration/rclone/config/temp/rclone_upload
tmp_wait_time = 1m0s
db_wait_time = 1s

[mycrypt]
type = crypt
remote = remote:xxxxxxxxxxxxxxxx
filename_encryption = standard
directory_name_encryption = true
password = xxxxxxxxxxxxxxxxxxxxxxx
password2 = xxxxxxxxxxxxxxxxxxxxxxx

18

What metric is that actually showing? Disk space? Disk IO? I don't get what the issue is from looking at that picture.

19

From Synology:

"A high percentage of volume utilization does not necessarily represent a problem. The value refers to the percentage of access rate within a designated time frame.

For instance, if a volume is accessed for five seconds within 10 seconds, its volume utilization would be 50%. If it is accessed throughout the entire 10 seconds, the percentage would be 100%.

This simply indicates processing data I/O rather than heavy loading on your Synology NAS"

20

Yes, but the concern is there. NAS is slowed down due to Rclone as I add media to Google Drive and Emby supports it. Emby is also much slower than before. In short, everything is related to Rclone and I am therefore trying to find a solution to find a little youth in the NAS.