Mount doesn't refresh when new files are added to GDrive

What is the problem you are having with rclone?

The mount doesn't refresh when new files/folders are added. I can see the changes in the gdrive but it doesn't update the Mount until I run systemd restart rclone-vfs.service

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: debian 10.10 (64 bit)
  • os/kernel: 4.19.0-18-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Gdrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

[Unit]
Description=RClone VFS Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
TimeoutStopSec=60
Environment=GOMAXPROCS=2

ExecStart=%h/bin/rclone mount gdrive-dbj: %h/Stuff/Mount \
  --config %h/.config/rclone/rclone.conf \
  --use-mmap \
  --dir-cache-time 1000h \
  --poll-interval=10s \
  --vfs-cache-mode writes \
  --tpslimit 10

StandardOutput=file:%h/scripts/rclone_vfs_mount.log
ExecStop=/bin/fusermount -uz %h/Stuff/Mount
Restart=on-failure

[Install]
WantedBy=default.target

The rclone config contents with secrets removed.

[gdrive-dbj]
type = drive
client_id = client-id
client_secret = clientsecret
scope = drive
token = {","}
team_drive = 0AHiPYfWUeBfiUk9PVA
root_folder_id =

A log from the command with the -vv flag

Paste  log here

Bump.

These are the debug logs, this is during a period of time I know things are uploaded to GDrive but not being refreshed inside the mount.

https://pastebin.com/raw/tXAuAFD2

good questions and not sure the exact answer

--- seems to be working somewhat
"DEBUG : : changeNotify: relativePath="TV", type=0"

---might want to check out this; to pre-load the vfs dir cache.

https://github.com/animosity22/homescripts/blob/master/systemd/rclone-drive.service
ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5572 _async=true

It's a weird one.

I have an update though, it seems like when individual files get uploaded it instantly appears in the mount. Although, when a new folder is uploaded to the drive, nothing at all appears in the mount.

not sure as i do not use remotes that support polling.

i assume that new folder has files in it?

Ahh. Yeah, each new folder has files in them.

i do not use remotes that support polling so i cannot provide an definitive answer.

tho we have some rcloners experienced with polling and analyzing rclone mount debug logs.

to keep the debug log small, i would kill the mount, delete the log file, restart the mount, copy a folder with a few files, make sure the files are fully uploaded, kill the mount, post the full debug log.

It looks like there were 6 changenotify events...

2022/06/25 21:25:41 DEBUG : : changeNotify: relativePath="TV", type=0
2022/06/25 21:25:41 DEBUG : : >changeNotify:
2022/06/25 21:25:41 DEBUG : : changeNotify: relativePath="", type=0
2022/06/25 21:25:41 DEBUG : : >changeNotify:
2022/06/25 21:25:41 DEBUG : : changeNotify: relativePath="Movies", type=0
2022/06/25 21:25:41 DEBUG : : >changeNotify:
2022/06/25 21:25:56 DEBUG : Google drive root '': Checking for changes on remote
2022/06/25 21:25:56 DEBUG : : changeNotify: relativePath="Movies", type=0
2022/06/25 21:25:56 DEBUG : : >changeNotify:
2022/06/25 21:25:56 DEBUG : : changeNotify: relativePath="", type=0
2022/06/25 21:25:56 DEBUG : : >changeNotify:
2022/06/25 21:25:56 DEBUG : : changeNotify: relativePath="TV", type=0
2022/06/25 21:25:56 DEBUG : : >changeNotify:

Those mean that the directories "" (the root), "Movies" and "TV" were changed, so rclone should have ditched the directory cache for them at that moment.

What was the path of the files that got uploaded?

I'm not getting those event's anymore. Even though new files are being uploaded to the mount.

2022/06/27 15:55:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 15:56:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 15:57:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 15:58:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 15:59:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:00:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:01:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:02:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:03:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:04:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:05:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:06:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:07:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:08:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:09:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:10:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 16:11:11 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)

That is during a time when a few individual files got uploaded to the drive and appeared in the mount's already existing folder(mount/Media/TV/TV/showname/Season 03/newfiles), and a brand new folder got uploaded to the drive but didn't appear in the mount(Drive location: Drive/Media/Movies/Movies/The BFG).


That's an overview of my GDrive.

You only see those on DEBUG level.

To help with this I need more specifics

  • how do you know the file didn't show? ls in the terminal? Check more than one method.
  • what is the path of the file you uploaded? Is it directly in the root or in a subdirectory?
  • did you see change notifies for it in the DEBUG log?
  • was there anything special about the file? A shortcut maybe?

I check the mount with ls and with an SFTP programme.

The path is either in /mount/Media/TV/Movies/.. or /mount/Media/Movies/Movies/...

The only time I seem to see change notifies is when I SFTP into the directory to look if the drive has been updated to match the GDrive. I'm not sure that's the correct behaviour or not?

2022/06/27 19:43:54 DEBUG : Google drive root '': Checking for changes on remote
2022/06/27 19:43:54 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 19:43:54 DEBUG : : changeNotify: relativePath="Media/Movies", type=0
2022/06/27 19:43:54 DEBUG : : >changeNotify:
2022/06/27 19:43:54 DEBUG : : changeNotify: relativePath="Media", type=0
2022/06/27 19:43:54 DEBUG : : >changeNotify:
2022/06/27 19:43:54 DEBUG : : changeNotify: relativePath="Media/TV", type=0
2022/06/27 19:43:54 DEBUG : : >changeNotify:
2022/06/27 19:44:54 DEBUG : Google drive root '': Checking for changes on remote
2022/06/27 19:44:54 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 19:45:54 DEBUG : Google drive root '': Checking for changes on remote
2022/06/27 19:45:54 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 19:46:54 DEBUG : Google drive root '': Checking for changes on remote
2022/06/27 19:46:54 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 19:47:54 DEBUG : Google drive root '': Checking for changes on remote
2022/06/27 19:47:54 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 19:48:54 DEBUG : Google drive root '': Checking for changes on remote
2022/06/27 19:48:54 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/06/27 19:49:54 DEBUG : Google drive root '': Checking for changes on remote
2022/06/27 19:49:54 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)

That notifier occurs when I checked the root folders in SFTP.

The files/folders that are getting uploaded are standard MKV files. The drive is setup with Anchors though, so the "Media" is full of shortcuts to multiple different folders(Movies, TV ect.)

I think this is probably the problem. Where do these shortcuts go? Do they go within the same drive?

Google doesn't send change notifies for shortcuts which are outside of the drive, and for ones inside the drive I'm not sure rclone will interpret them properly.

So can you explain more exactly how the setup with shortcuts work, which drives the files are on, etc - thanks.

That's what I assumed would be the issue. It's a separate main drive, that has a bunch of shortcuts to different drives. I unfortunately can't give you a better explanation, cause I didn't set it up.

Unfortunately that doesn't work for change requests at the moment :frowning:

Your best alternative is to use a vfs/refresh over the rc.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.