Missing files in mount but present on remote

Stokkes · September 28, 2020, 2:23am

What is your rclone version (output from `rclone version`)

rclone v1.53.1
- os/arch: linux/amd64
- go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

❯ cat /etc/lsb-release                                                                
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

Which cloud storage system are you using? (eg Google Drive)

Google Drive (Team Drive) via crypt remote

Rclone mount command

/usr/bin/rclone mount \
  --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36' \                                                                
  --config=/home/user/.config/rclone/rclone.conf \
  --allow-other \
  --allow-non-empty \
  --rc \
  --rc-addr=localhost:5572 \
  --rc-no-auth \
  --cache-dir=/tmp/rclone \
  --vfs-cache-mode full \
  --vfs-cache-max-size 100G \
  --vfs-cache-max-age 336h \
  --poll-interval=15s \
  --dir-cache-time 1000h \
  --umask=002 \
  --log-file /opt/rclone/logs/rclone.log \
  -vv \
  media: /mnt/remote

What problem are you having with rclone

It would appear ever since the 1.52 or 1.53 (difficult to determine the exact release) - I am getting issues where a mounted Team Drive does not refresh its internal vfs cache in certain situations where a file is deleted from the mount and a new file with the same name is re-uploaded to the Team Drive within a short period of time.

Note that I use a mergerfs mount with the rclone mount underneath and any new files saved to /mnt/local. These files are uploaded periodically via an rclone move to the Team Drive remote.

The below use case happens very randomly, but typically always triggered by Sonarr updating an episode to a new version.

NB If I unmount and re-mount, the files appear fine.

Use case / Timeline:

17:23: File Schitt's Creek - S04E11 - The Rollout (WEBDL-1080p).mkv exists on remote and in mount
17:24: New version of file is downloaded by Sonarr
17:24: Old file is deleted and replaced with the same name
17:36: New file is moved from the local disk in /mnt/local to the remote via an rclone move
17:36: rclone vfs logs a changeNotify event
17:36: File disappears from mount

Ouput of ls "/mnt/media/tvshows/Schitt's Creek (2015)/Season 04/" (Notice Episode 11 is missing)

'Schitt'\''s Creek - S04E01 - Dead Guy in Room 4 (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E01 - Dead Guy in Room 4 (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E02 - Pregnancy Test (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E02 - Pregnancy Test (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E03 - Asbestos Fest (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E03 - Asbestos Fest (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E04 - Girls'\'' Night (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E05 - R.I.P. Moira Rose (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E06 - Open Mic (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E07 - The Barbecue (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E07 - The Barbecue (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E08 - The Jazzaguy (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E08 - The Jazzaguy (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E09 - The Olive Branch (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E09 - The Olive Branch (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E10 - Baby Sprinkle (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E10 - Baby Sprinkle (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E12 - Singles Week (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E12 - Singles Week (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E13 - Merry Christmas, Johnny Rose (WEBRip-1080p).mkv'

Output of actual rclone remote: rclone lsf media:"tvshows/Schitt's Creek (2015)/Season 04" (Notice Episode 11 is present)

Schitt's Creek - S04E01 - Dead Guy in Room 4 (WEBDL-1080p).en.srt 
Schitt's Creek - S04E01 - Dead Guy in Room 4 (WEBDL-1080p).mkv
Schitt's Creek - S04E02 - Pregnancy Test (WEBDL-1080p).en.srt 
Schitt's Creek - S04E02 - Pregnancy Test (WEBDL-1080p).mkv
Schitt's Creek - S04E03 - Asbestos Fest (WEBDL-1080p).en.srt 
Schitt's Creek - S04E03 - Asbestos Fest (WEBDL-1080p).mkv
Schitt's Creek - S04E04 - Girls' Night (WEBDL-1080p).mkv 
Schitt's Creek - S04E05 - R.I.P. Moira Rose (WEBDL-1080p).mkv
Schitt's Creek - S04E06 - Open Mic (WEBDL-1080p).mkv 
Schitt's Creek - S04E07 - The Barbecue (WEBDL-1080p).en.srt
Schitt's Creek - S04E07 - The Barbecue (WEBDL-1080p).mkv 
Schitt's Creek - S04E08 - The Jazzaguy (WEBDL-1080p).en.srt
Schitt's Creek - S04E08 - The Jazzaguy (WEBDL-1080p).mkv 
Schitt's Creek - S04E09 - The Olive Branch (WEBDL-1080p).en.srt
Schitt's Creek - S04E09 - The Olive Branch (WEBDL-1080p).mkv 
Schitt's Creek - S04E10 - Baby Sprinkle (WEBDL-1080p).en.srt
Schitt's Creek - S04E10 - Baby Sprinkle (WEBDL-1080p).mkv 
Schitt's Creek - S04E11 - The Rollout (WEBDL-1080p).mkv
Schitt's Creek - S04E12 - Singles Week (WEBDL-1080p).en.srt 
Schitt's Creek - S04E12 - Singles Week (WEBDL-1080p).mkv 
Schitt's Creek - S04E13 - Merry Christmas, Johnny Rose (WEBRip-1080p).mkv

I've included a copy of the debug log that touches this episode: grep 'S04E11 - The Rollout' rclone.log it's available here: https://pastebin.com/m49rGA8n

Let me know if you need more info from the debug log or any other questions.

Thanks!

Animosity022 · September 28, 2020, 2:49am

I think you have the output of mergerfs remote and not your rclone mount listed here. Can we focus on the rclone mount and it's not there? (I assume that's the case as well).

Stokkes · September 28, 2020, 3:06am

Actually I think it was a listing of /mnt/remote, but just in case it wasn't, here it is again:

❯ ls /mnt/remote/tvshows/Schitt\'s\ Creek\ \(2015\)/Season\ 04/                          Sun 27 Sep 23:04:51 
'Schitt'\''s Creek - S04E01 - Dead Guy in Room 4 (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E01 - Dead Guy in Room 4 (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E02 - Pregnancy Test (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E02 - Pregnancy Test (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E03 - Asbestos Fest (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E03 - Asbestos Fest (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E04 - Girls'\'' Night (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E05 - R.I.P. Moira Rose (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E06 - Open Mic (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E07 - The Barbecue (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E07 - The Barbecue (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E08 - The Jazzaguy (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E08 - The Jazzaguy (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E09 - The Olive Branch (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E09 - The Olive Branch (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E10 - Baby Sprinkle (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E10 - Baby Sprinkle (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E12 - Singles Week (WEBDL-1080p).en.srt'
'Schitt'\''s Creek - S04E12 - Singles Week (WEBDL-1080p).mkv'
'Schitt'\''s Creek - S04E13 - Merry Christmas, Johnny Rose (WEBRip-1080p).mkv'

Stokkes · September 28, 2020, 3:07am

Another data point:

Doing a rclone rc vfs/refresh recursive=true and letting it run doesn't make the file appear. I have to unmount and remount.

ncw · September 28, 2020, 3:49pm

I think this may be a similar issue to this one

What do you think?

Animosity022 · September 28, 2020, 3:55pm

Ah, that does look like it.

darthShadow · September 28, 2020, 4:26pm

I don't think so actually. That issue deals with exactly the opposite scenario, i.e. file deleted in remote is still visible in cache.

I think this may be one of the problems of the drive change notify system where it picks up the change but doesn't invalidate anything, possibly due to missing parents or something.

Do you think it may be interrelated?

ncw · September 28, 2020, 5:50pm

It might be... as in I think they might both be to do with "virtual" directory entries - the symptom of vfs/refresh not fixing it makes it quite likely.

I think what has happened is that there is a virtual delete in the directory for the video file from when it got deleted, so when it appears in the remote again, rclone thinks it shouldn't be there and suppresses it.

darthShadow · September 28, 2020, 6:05pm

Makes sense. Thanks for the explanation.

Stokkes · September 28, 2020, 6:07pm

But there's a good 12 minutes in the timeline I pasted where the file gets deleted from the remote, and the new file is updated. I could see rclone ignoring if there's a race condition and it all happens within a few seconds, but 12 minutes?

Should I file a bug report on github?

darthShadow · September 28, 2020, 7:08pm

If the issue is as @ncw suspects and similar to the earlier mentioned issue (https://github.com/rclone/rclone/issues/4446), then the fix for that should also fix your problem. You can subscribe to that issue on Github so you are notified of any updates.

Stokkes · September 28, 2020, 8:48pm

I can definitely subscribe to that issue. Thanks @darthShadow

Do we know the exact version this was introduced? I'd like to revert back to a version that doesn't show this issue if possible.

darthShadow · September 28, 2020, 9:15pm

Anything pre-1.53 should probably be fine but you will lose the new cache improvements.

ncw · September 29, 2020, 2:14pm

I think the replacement file didn't get uploaded through the mount - is that correct?

If it had been then all would have been well.

This is still a bug in rclone that I need to fix but the workaround is to do everything through the mount.

Stokkes · September 29, 2020, 3:22pm

Correct, wasn't done through the mount.

I've reverted to 1.51 for now (wasn't sure about 1.52 as the issue linked above was created in july and referenced 1.52 with having the issue). I'll run 1.51 for a while to see if the issue returns, but I suspect it won't. The new cache benefits are nice, but not required. I'd rather have stability in the environment.

Thanks!

darthShadow · September 29, 2020, 5:01pm

That was a beta on top of the 1.52 release which had the vfs enhancements still in development. You should be fine with using 1.52.

system · November 29, 2020, 1:01pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.