Music mount: Help me to replicate this Google Drive Ocamlfuse settings in Rclone

soy_titooo · November 16, 2018, 8:04am

I tried multiple settings to mount and stream my music but none of them were good (minimum 5 seconds to start playing the music or skipping to the next songs).

After reading and trying some alternatives I found out that using Google-Drive-Ocalmfusr with the following settings resolved any playback issue:

acknowledge_abuse=false
apps_script_format=json
apps_script_icon=
cache_directory=
client_id=xxxxxxxxxxxxzzzzzzxxxxxxxxxxxxx5.apps.googleusercontent.com
client_secret=xxxxxxxxxxxxxxxxxxxxx
connect_timeout_ms=5000
curl_debug_off=false
data_directory=
delete_forever_in_trash_folder=false
desktop_entry_exec=
docs_file_extension=true
document_format=odt
document_icon=
download_docs=true
drawing_format=png
drawing_icon=
form_format=zip
form_icon=
fusion_table_format=desktop
fusion_table_icon=
keep_duplicates=false
large_file_read_only=false
large_file_threshold_mb=16
log_directory=
lost_and_found=false
low_speed_limit=0
low_speed_time=0
map_format=desktop
map_icon=
max_cache_size_mb=512
max_download_speed=0
max_memory_cache_size=10000000000
max_retries=8
max_upload_chunk_size=1099511627776
max_upload_speed=0
memory_buffer_size=8388608
metadata_cache_time=60
metadata_memory_cache=true
metadata_memory_cache_saving_interval=30
presentation_format=pdf
presentation_icon=
read_ahead_buffers=5
read_only=false
root_folder=
shared_with_me=false
spreadsheet_format=ods
spreadsheet_icon=
sqlite3_busy_timeout=5000
stream_large_files=true
team_drive_id=xxxxxxxxxxxx
umask=0o022
verification_code=

As I said the playback performance is perfect but it seems to be a bit less stable than Rclone. So I wonder if someone could help me replicate those settings in an Rclone remote.

Thanks

soy_titooo · November 18, 2018, 7:00pm

Any idea?
@Animosity022 the values mentioned above one are the default ones + 3 values that you changed in some tests you made months ago

As I said it works perfectly for music (I don’t care about video as I have a different remote for that), but Rclone is more stable than Google drive ocalmfuse so I’d like to have the equivalent settings for rclone

Animosity022 · November 18, 2018, 9:19pm

Not sure replicating it makes that much sense as they are very different in how they operate.

You can start with my settings and see how that works and go from there.

soy_titooo · December 7, 2018, 4:25pm

Well every single rclone settings I tried doesn’t bring good results for music streaming… It takes several seconds to start playing or switching between songs.

With the Google ocalmfuse settings I posted in the OP the streaming is almost perfect, but Google ocalmfuse has some memory leaks from time to time that lead to a a huge ram usage… That’s why I would like to replicate those settings in rclone. Rclone seems a much more stable app

soy_titooo · December 28, 2018, 10:36am

I have been trying to adapt your gmedia-reclone.service from your github to match what I posted in the OP but I still the scanning and playback performance is way worst than using the google-drive-ocamlfuse settings I posted in the OP (which were done folllowing some settings you used in a different thread).

I’m guessing that I just need to adapt the following google-drive-ocamlfuse settings to the rclone settings:
max_memory_cache_size=10000000000 #10GB max
memory_buffer_size=8388608 #8MB
read_ahead_buffers=5
stream_large_files=true

But I tried countless of different setups and I keep failing…

Would memory_buffer_size be the equivalent to rclone’s --vfs-read-chunk-size?
And is max_memory_cache_size the equivalent to rclone’s --buffer-size?

Animosity022 · December 28, 2018, 10:58am

Are you using your own API key?

What is your actual mount command you are using?

Do you have any debug logs of when you have the bad performance?

Are you seeing any bottlenecks on your system?

soy_titooo · December 28, 2018, 12:13pm

Yes, I’m using my own api keys:

This is my rclone script, which works very well for video playback but not so good for music playback…

[Unit]
Description=Juanmartinmusic Daemon
After=multi-user.target

[Service]
Type=simple
User=0
Group=0
Environment=RCLONE_CONFIG=/data/rclone/rclone.conf
ExecStart=/usr/bin/rclone --allow-non-empty --allow-other mount juanmartin:DeezerMusic/Music /mnt/juanmartinmusic \
–uid=1000 --gid=1000 \
–buffer-size 256M \
–dir-cache-time 72h \
–drive-chunk-size 32M \
–log-level DEBUG \
–log-file /home/tito/logs/juanmartinmusic-rclone.log \
–umask 002 \
–vfs-read-chunk-size 128M \
–vfs-read-chunk-size-limit off \
–rc
ExecStop=/bin/fusermount -uz /mnt/juanmartinmusic
TimeoutStopSec=20
KillMode=process
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

And this is the rclone debug log of what happened while I was playing music (I can upload the whole file if necessary, which also includes the scanning process):
https://pastebin.com/hXEhkhR5

Animosity022 · December 28, 2018, 12:15pm

I can’t view the log as it says it is private.

soy_titooo · December 28, 2018, 12:16pm

sorry it’s public now

Animosity022 · December 28, 2018, 12:36pm

Seems to be your player opens and closes the file repeatedly which causes the problem. What are you using to play the media?

2018/12/28 12:01:08 DEBUG : J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:09 DEBUG : &{J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3 (r)}: Flush:
2018/12/28 12:01:09 DEBUG : J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:09 DEBUG : &{J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3 (r)}: Flush:
2018/12/28 12:01:09 DEBUG : J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:19 DEBUG : &{J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3 (r)}: Flush:
2018/12/28 12:01:19 DEBUG : J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:27 DEBUG : &{J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3 (r)}: Flush:
2018/12/28 12:01:27 DEBUG : J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:29 DEBUG : &{J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3 (r)}: Flush:
2018/12/28 12:01:29 DEBUG : J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:30 DEBUG : &{J. Rawls/Great Live Caper (Album)/06 Great Live Caper (Pt. 2) featuring J-Live (Clean).mp3 (r)}: Flush:
2018/12/28 12:01:30 DEBUG : J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:31 DEBUG : &{J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3 (r)}: Flush:
2018/12/28 12:01:31 DEBUG : J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:31 DEBUG : &{J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3 (r)}: Flush:
2018/12/28 12:01:31 DEBUG : J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:48 DEBUG : &{J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3 (r)}: Flush:
2018/12/28 12:01:48 DEBUG : J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:48 DEBUG : &{J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3 (r)}: Flush:
2018/12/28 12:01:48 DEBUG : J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:49 DEBUG : &{J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3 (r)}: Flush:
2018/12/28 12:01:49 DEBUG : J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3: Open: flags=O_RDONLY
2018/12/28 12:01:50 DEBUG : &{J. Rawls/Great Live Caper (Album)/07 Great Live Caper (Pt.2) (Instrumental).mp3 (r)}: Flush:

You can see it repeatedly opens and closes the file which is the root cause of the slowness. Cache backend might provide some benefit but not sure as I don’t use it.

soy_titooo · December 28, 2018, 12:38pm

In this example I was using Plex.

With google-drive-ocamlfuse (settings from OP) everything plays perfectly and the scanning is also at least 5x faster, but as I mentioned it’s less stable, uses more RAM and folders are not accesible by certain programs…

Animosity022 · December 28, 2018, 12:40pm

ocamlfuse uses memory to store chunks like the cache backend so it’s comparing apples to an orange.

You aren’t going to find a vfs setting to fix the problem as the problem is the way Plex is doing the open/closes and that adds latency, which is what you are seeing.

soy_titooo · December 28, 2018, 12:53pm

Ok thank, then I guess I’ll need to stick to google-drive-ocamlfuse mounts for music playback

Animosity022 · December 28, 2018, 3:36pm

Yeah, seems to be a plex thing as I was able to reproduce it.

Animosity022 · December 28, 2018, 3:56pm

So my suggestion would be to try the cache backend as that seems to be much more suited for this as it works quite well in my testing.

I mounted with this:

felix@gemini:~$ rclone mount -vvv --log-file /home/felix/logs/test.log gmedia:Music /Music --allow-other

The only thing I would add/change is up the dir-cache time and leave all the other things at the defaults.

soy_titooo · December 29, 2018, 9:25am

WOW. I just added a cache remote to my gmedia-reclone.service and got Plex scanning from the cache remote directly instead of from gmedia-reclone.service.
The playback now is perfect! The scanning is still quite slow though but maybe that can’t be improved further.

Just to recap:

Section of my rclone.conf related to my music files:

[Unit]
Description=Juanmartin music Daemon
After=multi-user.target

[Service]
Type=simple
User=0
Group=0
Environment=RCLONE_CONFIG=/data/rclone/rclone.conf
ExecStart=/usr/bin/rclone --allow-non-empty --allow-other mount gmedia: /mnt/juanmartinmusic
–log-file /home/tito/logs/juanmartinmusic-rclone.log
-vvv
ExecStop=/bin/fusermount -uz /mnt/juanmartinmusic
TimeoutStopSec=20
KillMode=process
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Remote where I have the music files:

[juanmartin]
type = drive
client_id = myclientid-apps.googleusercontent.com
client_secret = mysecret-clientkey
scope = drive
root_folder_id =
service_account_file =
token = {“access_token”:“my-token”,“token_type”:“Bearer”,“refresh_token”:my-refreshtoken",“expiry”:“some-numbers”}

Remote cache that I created for the above (which has improved a lot the playback performance in Plex):

[gmedia]
type = cache
remote = juanmartin:DeezerMusic/Music
chunk_size = 10M
info_age = 2d
chunk_total_size = 1G

@Animosity022 the playback is perfect now, would you do something else to improve the scan speed?

PS: I noticed that the gmedia remote of the rclone.conf from your github mentions a remote called gcache: But there is not any gcache: remote in your rclone.conf

Animosity022 · December 29, 2018, 1:08pm

I don’t use the cache for any of my Plex stuff so I usually just add it in when I’m testing.

I don’t think there is much to do with the scanning speed as the cache is a little slower than vfs in that regard.

You can probably increase the age and the chunk size to something bigger if you can as well as all that stuff helps.