New files in old dir

fbrassin · March 11, 2019, 2:28pm

Hello,
i have this problem.
I have a cache encrypted mount.
I decided to set info_age = 20d because files never change, but with this configuration new episodes (i’m using with tv_show, with this naming /Serie_name/season) copied into existing dir are not seen by rclone.
The only way is to give manually cache/expire.
When my info_age was set to 6h i had all files seen every 6 hrs.

Is there a way to make rclone see added files in existing dir without waiting info_age expiration?

This is my configuration

[gcache]
type = cache
remote = xxxxxxxx:
plex_url = http://localhost:32400
plex_username = XXXXXXXXX@XXXXXX.XX
plex_password = *** ENCRYPTED ***
chunk_size = 10M
info_age = 20d
chunk_total_size = 10G
plex_token = XXXXXXXXXXX

Animosity022 · March 11, 2019, 2:30pm

rclone version shows what?

fbrassin · March 11, 2019, 2:31pm

rclone --version
rclone v1.45

os/arch: linux/amd64
go version: go1.11.2

Animosity022 · March 11, 2019, 2:32pm

Can you update to 1.46 which is the latest stable version.

How are you moving files to the mount?

fbrassin · March 11, 2019, 2:36pm

I’ll update now.

I’m uploading files from a vps with the command “rclone move”
While i have the problem with my local server with gdrive mounted with rclone

Animosity022 · March 11, 2019, 2:41pm

So is the workflow this?

[felix@gemini Test]$ cd
[felix@gemini ~]$ rclone copy /etc/hosts gcrypt:testfile
[felix@gemini ~]$ ls /Test
100M.file  1G.file  deleteme.file  mounted  Movies  Radarr_Movies  testfile  TV  TV_Ended

Are you uploading to a crypt remote? cache remote? Polling is 1 minute so that’s how long it takes to appear.

fbrassin · March 11, 2019, 2:51pm

I’m using this script:

github.com

ajkis/scripts/blob/master/rclone/rclone-upload.sh

#!/bin/bash
# RCLONE UPLOAD CRON TAB SCRIPT 
# chmod a+x /home/plex/scripts/rclone-upload.sh
# Type crontab -e and add line below (without #) and with correct path to the script
# * * * * * /home/plex/scripts/rclone-upload.sh >/dev/null 2>&1
# if you use custom config path add line bellow in line 20 after --log-file=$LOGFILE 
# --config=/path/rclone.conf (config file location)

if pidof -o %PPID -x "$0"; then
   exit 1
fi

LOGFILE="/home/plex/logs/rclone-upload.log"
FROM="/storage/local/"
TO="gdrivecrypt:/"

# CHECK FOR FILES IN FROM FOLDER THAT ARE OLDER THAN 15 MINUTES
if find $FROM* -type f -mmin +15 | read
  then
  start=$(date +'%s')

This file has been truncated. show original

I’m uploading to a crypt remote called “gcrypt:crypt/”
I’ve waited about 2 days after upload but without having new file. Then gave cache/expire and it came.

The some is not happening with movie. Maybe it depends on the depth of dir?
For movies all file are in movie dir, while series files are in Season dir which is in serie dir.

Animosity022 · March 11, 2019, 2:56pm

If you run the mount in debug and upload somehting, you should see this being updated in ~1 minute:

2019/03/11 10:48:33 DEBUG : Google drive root 'media': Checking for changes on remote
2019/03/11 10:48:33 INFO  : 6r50j5eru0lfub6f2n39ifq4b8: received cache expiry notification
2019/03/11 10:48:33 DEBUG : 6r50j5eru0lfub6f2n39ifq4b8: cache: expired
2019/03/11 10:48:33 DEBUG : 6r50j5eru0lfub6f2n39ifq4b8: notify: expired '6r50j5eru0lfub6f2n39ifq4b8'
2019/03/11 10:48:33 INFO  : alnlhogqsopvc4h8nbn466s28k: received cache expiry notification
2019/03/11 10:48:33 DEBUG : alnlhogqsopvc4h8nbn466s28k: cache: expired
2019/03/11 10:48:33 DEBUG : alnlhogqsopvc4h8nbn466s28k: notify: expired 'alnlhogqsopvc4h8nbn466s28k'
2019/03/11 10:48:33 INFO  : rs88l6p51j4kp0g9if3j294ts0: received cache expiry notification
2019/03/11 10:48:33 DEBUG : Cache remote gcache:: got change notification for non cached entry rs88l6p51j4kp0g9if3j294ts0
2019/03/11 10:48:33 DEBUG : : cache: expired
2019/03/11 10:48:33 DEBUG : rs88l6p51j4kp0g9if3j294ts0: notify: expired ''
2019/03/11 10:48:34 INFO  : testfile: received cache expiry notification
2019/03/11 10:48:34 DEBUG : : invalidating directory cache
2019/03/11 10:48:34 DEBUG : 6r50j5eru0lfub6f2n39ifq4b8: cache: expired
2019/03/11 10:48:34 DEBUG : 6r50j5eru0lfub6f2n39ifq4b8: notify: expired '6r50j5eru0lfub6f2n39ifq4b8'
2019/03/11 10:48:34 INFO  : deleteme.file: received cache expiry notification
2019/03/11 10:48:34 DEBUG : alnlhogqsopvc4h8nbn466s28k: cache: expired
2019/03/11 10:48:34 DEBUG : alnlhogqsopvc4h8nbn466s28k: notify: expired 'alnlhogqsopvc4h8nbn466s28k'
2019/03/11 10:48:34 INFO  : hosts: received cache expiry notification
2019/03/11 10:48:34 DEBUG : : cache: expired

based on the default polling.

Do you have trash off? That’s the only bug in polling I’m aware of which seems to be a GD API bug.

use_trash = false

should be:

use_trash = true

which is the default value.

fbrassin · March 11, 2019, 3:00pm

I haven’t got use_trash setted. Do you think i shoud set it to true?

calisro · March 11, 2019, 3:03pm

It is set to TRUE by default if you’ve not turned it off with the parameter. And it only affects deletes. Please post your log. you should see the polling checks and a cache invalidation when you upload to the remote just as @Animosity022 posted above.

fbrassin · March 11, 2019, 3:06pm

Just show you what i have.

This is the dir in my vps:
-rw-r–r-- 1 root root 2,9G feb 22 23:51 Chicago Fire.S07E01.A Closer Eye.mkv
-rw-r–r-- 1 root root 5,0K feb 23 05:47 Chicago Fire.S07E01.A Closer Eye.nfo
-rw-r–r-- 1 root root 43K feb 23 02:08 Chicago Fire.S07E01.A Closer Eye.srt
-rw-r–r-- 1 root root 2,8G mar 2 18:59 Chicago Fire.S07E02.Going to War (1).mkv
-rw-r–r-- 1 root root 3,1K mar 3 06:58 Chicago Fire.S07E02.Going to War (1).nfo
-rw-r–r-- 1 root root 43K mar 2 23:30 Chicago Fire.S07E02.Going to War (1).srt
-rw-r–r-- 1 root root 3,5G mar 9 19:36 Chicago Fire.S07E03.Thirty Percent Sleight of Hand.mkv
-rw-r–r-- 1 root root 5,0K mar 10 06:11 Chicago Fire.S07E03.Thirty Percent Sleight of Hand.nfo
-rw-r–r-- 1 root root 429 feb 23 05:47 season.nfo

And this in my local server:
-rw-r–r-- 1 root root 2,9G feb 22 23:51 ‘Chicago Fire.S07E01.A Closer Eye.mkv’
-rw-r–r-- 1 root root 5,0K feb 23 05:47 ‘Chicago Fire.S07E01.A Closer Eye.nfo’
-rw-r–r-- 1 root root 43K feb 23 02:08 ‘Chicago Fire.S07E01.A Closer Eye.srt’
-rw-r–r-- 1 root root 2,8G mar 2 18:59 ‘Chicago Fire.S07E02.Going to War (1).mkv’
-rw-r–r-- 1 root root 3,1K mar 3 06:58 ‘Chicago Fire.S07E02.Going to War (1).nfo’
-rw-r–r-- 1 root root 43K mar 2 23:30 ‘Chicago Fire.S07E02.Going to War (1).srt’
-rw-r–r-- 1 root root 429 feb 23 05:47 season.nfo

As you can see it is missing episode S07E03 dated march the 9th.

calisro · March 11, 2019, 3:08pm

your log would help to confirm drive is invalidating the cache or not.

Also can you post the rest of the config? Trying to see what order you link the crypt, cache, and drive.

fbrassin · March 11, 2019, 3:10pm

I’ve got the log file, but it is 54Mb.
Which part do you need?

calisro · March 11, 2019, 3:12pm

Just go to the time you performed that move command and search for the “: invalidating directory cache” and the “polling” entries. (btw, we’re looking for the “mount” logs… Not the ‘Move’ logs.

fbrassin · March 11, 2019, 3:15pm

Do you mean this on my vps?

79566:2019/03/10 06:11:18 INFO : crypt/Serie_Tv/Chicago Fire/Season07/Chicago Fire.S07E03.Thirty Percent Sleight of Hand.nfo: Copied (new)
79592:2019/03/10 06:41:29 INFO : crypt/Serie_Tv/Chicago Fire/Season07/Chicago Fire.S07E03.Thirty Percent Sleight of Hand.nfo: received cache expiry notif

Animosity022 · March 11, 2019, 3:19pm

You’d want the debug log from where it is missing , which I believe is your local server in your config.

fbrassin · March 11, 2019, 3:21pm

Sorry, how to make debug log. Is it normal log?
I have this mount:
rclone -v mount --allow-other --vfs-cache-mode writes --rc --log-file /root/logs/rclone.log cachecrypt:/ /media/google_rclone
Is it ok to have debug log or should i change anything?

Animosity022 · March 11, 2019, 3:23pm

You can add -vv or add --log-level DEBUG and restart it.

You’d have to run a test as you’ve missed the event.

fbrassin · March 11, 2019, 3:39pm

I cannot mout with that.
I have this error:
2019/03/11 16:32:01 Can’t set -v and --log-level

This is the script to mount:
#!/bin/sh
nohup rclone -vv mount
–log-level DEBUG
–allow-other
–vfs-cache-mode writes
–rc
–log-file /root/logs/rclone.log
cachecrypt:/ /media/google_rclone &

Animosity022 · March 11, 2019, 3:41pm

You need to pick one of those items and not both as they do the same thing.