Recommended Dropbox (Formally Google Drive) and Plex Mount Settings

raphattack · May 21, 2019, 8:17pm

Popping back in for another question that you may be able to help with.

When running rclone move with the same flags that you have set, I get nice periodic updates in upload.log similar to below:

2019/05/21 02:51:12 INFO  :
Transferred:      250.317G / 250.456 GBytes, 100%, 1.394 MBytes/s, ETA 1m41s
Errors:                 0
Checks:               108 / 108, 100%
Transferred:          106 / 107, 99%
Elapsed time:   51h4m1.8s
Transferring:
 * Season 06/S06E03 - The….mkv: 94% /2.385G, 1.394M/s, 1m41s

When switching the command to rclone copy with the same flags, it only logs to upload.log when a file has finished transferring.

Is there a way to get rclone copy to log similarly to rclone move?

Animosity022 · May 21, 2019, 8:25pm

I write my output to a log file and use -v to generate stats updates every minute I believe it is.

/usr/bin/rclone move /data/local/ gcrypt: --log-file /opt/rclone/logs/upload.log -v --drive-chunk-size 32M --exclude-from /opt/rclone/scripts/excludes --delete-empty-src-dirs

You can put --log-file on any rclone command and use a different log file for mount / moves / copies /etc.

Does that answer your question?

raphattack · May 21, 2019, 8:50pm

I'm also using the -v flag. Actually using the same flags you are, the only difference is I'm using copy instead of move.

/usr/bin/rclone copy \
        -v \        
        "/media/sample_show/" "gcrypt:Shows/sample_show" \
        --checkers 3 \
        --log-file /opt/rclone/logs/upload.log \
        --tpslimit 3 \
        --transfers 3 \
        --drive-chunk-size 32M \
        --exclude-from /opt/rclone/scripts/excludes

For copy operations, it is not logging the stats to the specified --log-file every minute. Instead, it is only logging when the file has completed transferring.

Animosity022 · May 21, 2019, 8:57pm

The default is one minute:

   --stats duration                               Interval between printing stats, e.g 500ms, 60s, 5m. (0 to disable) (default 1m0s)

What version of rclone are you using?

raphattack · May 21, 2019, 9:00pm

Even after one minute, it does not log the stats. The stats are actually printed to standard output, but never reaches the log file.

$ rclone version
rclone v1.47.0
- os/arch: linux/amd64
- go version: go1.12.4

Animosity022 · May 21, 2019, 9:07pm

What's your goal? What do you want to happen?

raphattack · May 21, 2019, 9:15pm

My goal is to periodically see stats logged to the specified --log-file. It looks like -v behaves differently for copy and move operations.

Running move will log the stats to the log file. Running copy will print the stats to standard output, but will not log it to the log file.

Since I'm running the job in the background, standard output doesn't help here.

I can do the following to achieve what I want, but I'd like to know if there exists a way to do it setting rclone flags:

nohup /usr/bin/rclone copy \
        -v \
        "/media/sample_show/" "gcrypt:Shows/sample_show/" \
        --checkers 3 \
        --log-file /opt/rclone/logs/upload.log \
        --tpslimit 3 \
        --transfers 3 \
        --drive-chunk-size 32M \
        --exclude-from /opt/rclone/scripts/excludes
>> /opt/rclone/logs/upload.log &

Animosity022 · May 21, 2019, 9:18pm

You are using:

  -P, --progress                                     Show progress during transfer.

So that prints to stdout as you asked it to show progress.

Remove the -P and you'll see it in the logfile.

raphattack · May 21, 2019, 9:20pm

Should have mentioned that I have removed the -P flag actually. Edited my previous post. Still does not log the stats to the log file.

Animosity022 · May 21, 2019, 9:22pm

I did a test with copy and 5 seconds. Looks good here:

Did you try 5 seconds or something to see?

raphattack · May 21, 2019, 9:31pm

Weird, that seems to work and of course now it works even without --stats.

Maybe I haven't had enough coffee today. Thanks!

crm · May 22, 2019, 12:41pm

Hi, I'm using your config at the moment with the addition of a samba server that shares GD. Do you have any idea on how to get decent performance with vlc (or other players if you can recommend one)? Basically SD works fine, FHD chops, 4K is unplayable. Single transfer copy performance from google drive is 20MB/s though.

Animosity022 · May 22, 2019, 12:55pm

I don't use Samba at all and would think that is generally a bad idea as I'd mount on the system itself. Would suggest making a new post as that's not related to my settings.

Goyo · May 24, 2019, 5:03am

Great setup - I am using your setup but I do have an issue about updating Plex. I copy the files into Google Drive directly on another computer and run Plex and rclone on another computer. Plex seems to notice the updated for awhile but after about a day, the updates do not show. I have to restart my Plex/rclone computer and then it will see the new files in rclone and Plex will see it. I am using the following mount
--dir-cache-time 48h --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit 2G --buffer-size 512M --vfs-cache-mode writes --log-level INFO --log-file ~/rclone.log --timeout=1h --vfs-cache-max-age 5m --rc

Any ideas? I have setup my api in Google Drive. Is it something to do with my dir--cache time? Many thanks.

Animosity022 · May 24, 2019, 9:47am

Would suggest you make a new post and share your settings and a debug log.

Goyo · May 24, 2019, 10:48am

OK, I will make a new post.

ybd · May 27, 2019, 4:26pm

Hi @Animosity022,

Hope you don't mind me posting in here.

I had a perfect setup running not too long back, very similar to yours actually, with no issues whatsoever but had a hard drive fail and didn't have script backup. I used it as an opportunity to go to a new dedicated server.

Now I'm trying to set it up again, and it's simply not running as expected which is strange as I really had a good working MergerFS, Rclone Mount and upload script over night running.

My issue seems to be MergerFS. When I point Plex directly at the rclone mount it runs perfect. No issues whatsoever.

When I point it at the MergerFS mount, it constantly buffers, super slow start time, etc. Just very, very inconsistent.

I did a seektest, and it shows it here too.

Just wondering, any ideas? I've exhausted all of the forum, documentation, etc, plenty of different configurations and so on. No idea why.

The box is a i7 6700, 32gb ram, 3x4TB HDD's, along with 1gbps premium bandwidth at OVH. More than adequate.

Here is the seek test, the first is the rclone mount directly (google_media folder), and the second is the mergerfs folder (gmedia).

27.05.2019 18:21:24 SPEEDSEEK TESTS STARTED
 
MERGERFS MOUNT
mergerfs -o defaults,sync_read,allow_other,category.action=all,category.create=ff,func.getattr=newest /home/ybd/local_media:/home/ybd/google_media /home/ybd/gmedia
 
RCLONE MOUNT
rclone mount media-crypt-1: /home/ybd/google_media --allow-other --buffer-size 512M --dir-cache-time 96h --drive-chunk-size 128M --timeout 1h --umask 002 --vfs-read-chunk-size 64M --vfs-read-chunk-size-limit off --rc --log-file /home/ybd/rclone/rclone.log --log-level DEBUG
 
SEEKSPEED /home/ybd/google_media/tmp/100M.file
2019/05/27 18:21:27 File Open took 2.111071062s
2019/05/27 18:21:28 Reading 170511 from 3997010 took 839.481201ms 
2019/05/27 18:21:28 Reading 459963 from 55931421 took 324.922495ms 
2019/05/27 18:21:28 Reading 643462 from 59171537 took 310.332228ms 
2019/05/27 18:21:29 Reading 578220 from 39190558 took 400.063169ms 
2019/05/27 18:21:29 Reading 280484 from 94963616 took 314.329824ms 
2019/05/27 18:21:29 Reading 350895 from 21454084 took 314.442092ms 
2019/05/27 18:21:30 Reading 614385 from 34699574 took 341.032369ms 
2019/05/27 18:21:30 Reading 300314 from 87272715 took 524.611788ms 
2019/05/27 18:21:31 Reading 285845 from 47057768 took 325.862838ms 
2019/05/27 18:21:31 Reading 625634 from 52633990 took 381.341475ms 
2019/05/27 18:21:31 Reading 67802 from 85725473 took 331.375174ms 
2019/05/27 18:21:32 Reading 401042 from 9333311 took 659.777042ms 
2019/05/27 18:21:32 Reading 445739 from 31415178 took 446.965455ms 
2019/05/27 18:21:33 Reading 85240 from 18272872 took 379.967928ms 
2019/05/27 18:21:33 Reading 923639 from 37014388 took 317.015436ms 
2019/05/27 18:21:33 Reading 1011576 from 39081944 took 17.909944ms 
2019/05/27 18:21:33 Reading 686863 from 96602623 took 282.946311ms 
2019/05/27 18:21:34 Reading 756044 from 27648203 took 379.059032ms 
2019/05/27 18:21:34 Reading 82935 from 30736283 took 435.90867ms 
2019/05/27 18:21:35 Reading 543634 from 13714932 took 319.315218ms 
2019/05/27 18:21:35 Reading 180882 from 66189515 took 341.832518ms 
2019/05/27 18:21:35 Reading 402243 from 72016420 took 470.861486ms 
2019/05/27 18:21:36 Reading 670611 from 81694652 took 389.549544ms 
2019/05/27 18:21:36 Reading 735972 from 28990319 took 433.520007ms 
2019/05/27 18:21:37 Reading 1025113 from 84507910 took 439.286557ms 
2019/05/27 18:21:37 That took 9.722352121s for 25 iterations, 388.894084ms per iteration
Finished in 0m:12s
FileSize 100M
SEEKSPEED /home/ybd/gmedia/tmp/100M.file
2019/05/27 18:21:37 File Open took 624.583µs
2019/05/27 18:21:38 Reading 170511 from 3997010 took 951.868616ms 
2019/05/27 18:21:38 Reading 459963 from 55931421 took 389.30637ms 
2019/05/27 18:21:39 Reading 643462 from 59171537 took 1.298746898s 
2019/05/27 18:21:41 Reading 578220 from 39190558 took 2.038656409s 
2019/05/27 18:21:42 Reading 280484 from 94963616 took 300.407337ms 
2019/05/27 18:21:43 Reading 350895 from 21454084 took 1.099147284s 
2019/05/27 18:21:44 Reading 614385 from 34699574 took 1.392256039s 
2019/05/27 18:21:45 Reading 300314 from 87272715 took 625.392895ms 
2019/05/27 18:21:46 Reading 285845 from 47057768 took 1.174074653s 
2019/05/27 18:21:48 Reading 625634 from 52633990 took 1.852562332s 
2019/05/27 18:21:48 Reading 67802 from 85725473 took 276.736631ms 
2019/05/27 18:21:49 Reading 401042 from 9333311 took 1.031874503s 
2019/05/27 18:21:50 Reading 445739 from 31415178 took 393.389714ms 
2019/05/27 18:21:50 Reading 85240 from 18272872 took 322.618529ms 
2019/05/27 18:21:51 Reading 923639 from 37014388 took 1.057317677s 
2019/05/27 18:21:52 Reading 1011576 from 39081944 took 931.86585ms 
2019/05/27 18:21:54 Reading 686863 from 96602623 took 1.948170445s 
2019/05/27 18:21:55 Reading 756044 from 27648203 took 1.572178725s 
2019/05/27 18:21:56 Reading 82935 from 30736283 took 390.078128ms 
2019/05/27 18:21:57 Reading 543634 from 13714932 took 1.032215351s 
2019/05/27 18:21:58 Reading 180882 from 66189515 took 1.052915028s 
2019/05/27 18:21:59 Reading 402243 from 72016420 took 974.537186ms 
2019/05/27 18:22:01 Reading 670611 from 81694652 took 2.099865938s 
2019/05/27 18:22:03 Reading 735972 from 28990319 took 2.31624195s 
2019/05/27 18:22:05 Reading 1025113 from 84507910 took 2.131252908s 
2019/05/27 18:22:05 That took 28.654696926s for 25 iterations, 1.146187877s per iteration
Finished in 0m:28s
FileSize 100M

There are no errors that I can see in the log, so I'm a bit stumped and lost for what it could be. When a -stats 1s is running, there's a max of 1MB/s when streaming through plex.

Would love any help.

Animosity022 · May 27, 2019, 4:37pm

That is odd.

What version of mergerfs? Not seeing any other hiccups on the system in terms of resources?

[felix@gemini ~]$ mergerfs -v
mergerfs version: 4147b9e
FUSE library version: 2.9.7-mergerfs_2.27.0
fusermount version: 2.9.9
using FUSE kernel interface version 7.27

ybd · May 27, 2019, 4:39pm

Thanks for getting back so quick. Yep must admit I feel like banging head against wall !

No other hiccups, it's a fresh box really so I'm a bit stumped.

[ybd@server]:(1.1Gb)~$ mergerfs -v
mergerfs version: 2.27.1
FUSE library version: 2.9.7-mergerfs_2.27.0
fusermount version: 2.9.4
using FUSE kernel interface version 7.27

Animosity022 · May 27, 2019, 4:46pm

Hmm, I don't think it is you.

My times are super awful on my mergerfs mount.

I wonder if a later version did something odd as that definitely seems new to me.