Rclone mount - google drive - optimal settings for local file search / grep

Help and Support
1

What is the problem you are having with rclone?

I have an rclone mount successfully set up and synchronising between Google Drive and a local mount point on my desktop PC. By and large, everything works well. One folder contains a tree structure of PDF documents including some large books. I need to pdfgrep these files but I am finding this process is very slow within the mount point. I was hoping my cache/vfs settings along with a post process to trigger a recursive vfs/refresh would materialise all my PDFs locally on my PC but I suspect this isn't truly possible with fuse file systems. My settings below are a hodgepodge from various forum posts here and the online documentation. A typical command I am calling is:

user@pc:~/GoogleDrive/Books/Maths/PSPapers$ pdfgrep lognormal *.pdf

What is your rclone version (output from rclone version)

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.6

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I'm using the following systemd user service unit on Ubuntu 20.04.

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/user/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount GDriveRC: /home/user/GoogleDrive \
--dir-cache-time 8760h \
--poll-interval 30s \
--log-file /tmp/rclone-gdrive.log \
--log-level DEBUG \
--rc \
--rc-addr :5572 \
--rc-no-auth \
--attr-timeout 8700h \
--umask 002 \
--cache-dir=/tmp/rclone-grive-cache \
--vfs-cache-mode writes \
--vfs-cache-max-size 250G \
--vfs-cache-max-age 5000h \
--vfs-cache-poll-interval 5m
ExecStop=/bin/fusermount -u -z /home/user/GoogleDrive
ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5572 _async=true
Restart=on-failure

The rclone config contents with secrets removed.

[GDriveRC]
type = drive
scope = drive

A log from the command with the -vv flag

The tail of the log looks a bit like this after calling above command

2021/12/29 20:35:49 DEBUG : Google drive root '': Checking for changes on remote
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/: ReadDirAll: 
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/: >ReadDirAll: item=42, err=<nil>
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: Attr: 
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: >Attr: a=valid=8700h0m0s ino=0 size=63659854 mode=-rw-rw-r--, err=<nil>
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: Open: flags=OpenReadOnly
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: Open: flags=O_RDONLY
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: >Open: fd=Books/Maths/PSPapers/Proc12345.pdf (r), err=<nil>
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: >Open: fh=&{Books/Maths/PSPapers/Proc12345.pdf (r)}, err=<nil>
2021/12/29 20:36:02 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=16384, offset=0
2021/12/29 20:36:02 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 0 length 134217728
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 0 length 4096 chunkOffset 0 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 4096 length 8192 chunkOffset 0 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 12288 length 16384 chunkOffset 0 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 28672 length 32768 chunkOffset 0 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=16384, err=<nil>
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 61440 length 65536 chunkOffset 0 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=63655936
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 126976 length 131072 chunkOffset 0 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 258048 length 262144 chunkOffset 0 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 520192 length 524288 chunkOffset 0 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 16384 to 63655936 (fs.RangeSeeker)
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 1044480 to 63655936 length -1
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 63655936 chunkSize 134217728
2021/12/29 20:36:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 63655936 length 134217728
2021/12/29 20:36:19 DEBUG : Google drive root '': Checking for changes on remote
2021/12/29 20:36:33 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=3918, err=<nil>
2021/12/29 20:36:33 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=59424768
2021/12/29 20:36:33 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 63659854 to 59424768 (fs.RangeSeeker)
2021/12/29 20:36:33 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 63659854 to 59424768 length -1
2021/12/29 20:36:33 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:33 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 59424768 length 134217728
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59428864 length 8192 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59437056 length 16384 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=16384, offset=59428864
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59453440 length 32768 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=16384, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=32768, offset=59445248
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59486208 length 65536 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=32768, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=65536, offset=59478016
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59551744 length 131072 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=65536, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=59543552
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59682816 length 262144 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=59674624
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59944960 length 524288 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=59805696
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=59936768
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 60469248 length 1048576 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=60067840
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=60198912
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=60329984
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=60461056
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 61517824 length 1048576 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=60592128
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=60723200
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=60854272
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=60985344
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=61116416
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=61247488
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=61378560
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=61509632
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 62566400 length 1048576 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=61640704
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=61771776
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=61902848
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=62033920
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=62164992
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=62296064
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=62427136
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=62558208
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 63614976 length 1048576 chunkOffset 59424768 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=62689280
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=62820352
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=62951424
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=63082496
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=63213568
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=63344640
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=131072, offset=63475712
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=131072, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=49152, offset=63606784
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=49152, err=<nil>
2021/12/29 20:36:48 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=59420672
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 63655936 to 59420672 (fs.RangeSeeker)
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 63659854 to 59420672 length -1
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 59420672 chunkSize 134217728
2021/12/29 20:36:48 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 59420672 length 134217728
2021/12/29 20:36:49 DEBUG : Google drive root '': Checking for changes on remote
2021/12/29 20:37:03 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:37:03 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59424768 length 8192 chunkOffset 59420672 chunkSize 134217728
2021/12/29 20:37:03 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59432960 length 16384 chunkOffset 59420672 chunkSize 134217728
2021/12/29 20:37:03 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=59379712
2021/12/29 20:37:03 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59449344 length 32768 chunkOffset 59420672 chunkSize 134217728
2021/12/29 20:37:03 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 59424768 to 59379712 (fs.RangeSeeker)
2021/12/29 20:37:03 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 59482112 to 59379712 length -1
2021/12/29 20:37:03 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 59379712 chunkSize 134217728
2021/12/29 20:37:03 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 59379712 length 134217728
2021/12/29 20:37:19 DEBUG : Google drive root '': Checking for changes on remote
2021/12/29 20:37:21 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:37:21 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59383808 length 8192 chunkOffset 59379712 chunkSize 134217728
2021/12/29 20:37:21 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59392000 length 16384 chunkOffset 59379712 chunkSize 134217728
2021/12/29 20:37:21 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=59416576
2021/12/29 20:37:21 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59408384 length 32768 chunkOffset 59379712 chunkSize 134217728
2021/12/29 20:37:21 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 59383808 to 59416576 (fs.RangeSeeker)
2021/12/29 20:37:21 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 59441152 to 59416576 length -1
2021/12/29 20:37:21 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 59416576 chunkSize 134217728
2021/12/29 20:37:21 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 59416576 length 134217728
2021/12/29 20:37:34 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:37:34 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59420672 length 8192 chunkOffset 59416576 chunkSize 134217728
2021/12/29 20:37:34 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=28672
2021/12/29 20:37:34 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 59428864 length 16384 chunkOffset 59416576 chunkSize 134217728
2021/12/29 20:37:34 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 59420672 to 28672 (fs.RangeSeeker)
2021/12/29 20:37:34 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 59445248 to 28672 length -1
2021/12/29 20:37:34 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 28672 chunkSize 134217728
2021/12/29 20:37:34 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 28672 length 134217728
2021/12/29 20:37:49 DEBUG : Google drive root '': Checking for changes on remote
2021/12/29 20:37:50 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:37:50 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=12288, offset=16384
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 32768 length 8192 chunkOffset 28672 chunkSize 134217728
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 40960 length 16384 chunkOffset 28672 chunkSize 134217728
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 57344 length 32768 chunkOffset 28672 chunkSize 134217728
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 90112 length 65536 chunkOffset 28672 chunkSize 134217728
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 155648 length 131072 chunkOffset 28672 chunkSize 134217728
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 32768 to 16384 (fs.RangeSeeker)
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 286720 to 16384 length -1
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 16384 chunkSize 134217728
2021/12/29 20:37:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 16384 length 134217728
2021/12/29 20:38:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 20480 length 8192 chunkOffset 16384 chunkSize 134217728
2021/12/29 20:38:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 28672 length 16384 chunkOffset 16384 chunkSize 134217728
2021/12/29 20:38:04 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=12288, err=<nil>
2021/12/29 20:38:04 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=16384, offset=32768
2021/12/29 20:38:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 28672 to 32768 (fs.RangeSeeker)
2021/12/29 20:38:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 45056 to 32768 length -1
2021/12/29 20:38:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 32768 chunkSize 134217728
2021/12/29 20:38:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 32768 length 134217728
2021/12/29 20:38:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 36864 length 8192 chunkOffset 32768 chunkSize 134217728
2021/12/29 20:38:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 45056 length 16384 chunkOffset 32768 chunkSize 134217728
2021/12/29 20:38:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 61440 length 32768 chunkOffset 32768 chunkSize 134217728
2021/12/29 20:38:17 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=16384, err=<nil>
2021/12/29 20:38:17 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=57344
2021/12/29 20:38:17 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 94208 length 65536 chunkOffset 32768 chunkSize 134217728
2021/12/29 20:38:17 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=69632
2021/12/29 20:38:17 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:17 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=143360
2021/12/29 20:38:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 73728 to 143360 (fs.RangeSeeker)
2021/12/29 20:38:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 159744 to 143360 length -1
2021/12/29 20:38:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 143360 chunkSize 134217728
2021/12/29 20:38:17 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 143360 length 134217728
2021/12/29 20:38:19 DEBUG : Google drive root '': Checking for changes on remote
2021/12/29 20:38:31 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:31 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 147456 length 8192 chunkOffset 143360 chunkSize 134217728
2021/12/29 20:38:31 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 155648 length 16384 chunkOffset 143360 chunkSize 134217728
2021/12/29 20:38:31 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=73728
2021/12/29 20:38:31 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 147456 to 73728 (fs.RangeSeeker)
2021/12/29 20:38:31 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 172032 to 73728 length -1
2021/12/29 20:38:31 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:31 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 73728 length 134217728
2021/12/29 20:38:49 DEBUG : Google drive root '': Checking for changes on remote
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 77824 length 8192 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 86016 length 16384 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=16384, offset=77824
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 102400 length 32768 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=16384, err=<nil>
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 135168 length 65536 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 200704 length 131072 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 331776 length 262144 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 593920 length 524288 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 1118208 length 1048576 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 2166784 length 1048576 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=192512
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=147456
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 3215360 length 1048576 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=282624
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=200704
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=16384, offset=204800
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=16384, err=<nil>
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=270336
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:38:49 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=196608
2021/12/29 20:38:49 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 4263936 length 1048576 chunkOffset 73728 chunkSize 134217728
2021/12/29 20:38:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 274432 to 196608 (fs.RangeSeeker)
2021/12/29 20:38:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 5312512 to 196608 length -1
2021/12/29 20:38:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:38:50 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 196608 length 134217728
2021/12/29 20:39:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 200704 length 8192 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:04 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:39:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 208896 length 16384 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:04 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 225280 length 32768 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 258048 length 65536 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 323584 length 131072 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 454656 length 262144 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 716800 length 524288 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 1241088 length 1048576 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 2289664 length 1048576 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 3338240 length 1048576 chunkOffset 196608 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=364544
2021/12/29 20:39:05 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:39:05 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=290816
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 368640 to 290816 (fs.RangeSeeker)
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 4386816 to 290816 length -1
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:05 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 290816 length 134217728
2021/12/29 20:39:19 DEBUG : Google drive root '': Checking for changes on remote
2021/12/29 20:39:19 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=4096, err=<nil>
2021/12/29 20:39:19 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 294912 length 8192 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:19 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=16384, offset=294912
2021/12/29 20:39:19 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 303104 length 16384 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:19 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 319488 length 32768 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:19 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: >Read: read=16384, err=<nil>
2021/12/29 20:39:19 DEBUG : &{Books/Maths/PSPapers/Proc12345.pdf (r)}: Read: len=4096, offset=360448
2021/12/29 20:39:19 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 352256 length 65536 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:20 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 417792 length 131072 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:20 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 548864 length 262144 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:20 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 811008 length 524288 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:20 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at 1335296 length 1048576 chunkOffset 290816 chunkSize 134217728
2021/12/29 20:39:20 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ReadFileHandle.seek from 311296 to 360448 (fs.RangeSeeker)
2021/12/29 20:39:20 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.RangeSeek from 2383872 to 360448 length -1
2021/12/29 20:39:20 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.Read at -1 length 4096 chunkOffset 360448 chunkSize 134217728
2021/12/29 20:39:20 DEBUG : Books/Maths/PSPapers/Proc12345.pdf: ChunkedReader.openRange at 360448 length 134217728
2

hello and welcome to the fourm,

that version of rclone is very old and should update to latest stable and test again.
https://rclone.org/install/#script-installation

3

I've updated to latest/greatest and it doesn't appear to change anything. Running pdfgrep still stalls, the new debug log is nearly identical to what I posted earlier.

What is your rclone version (output from rclone version)

rclone v1.57.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-90-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none
4
  • as per the docs
    --- should create your own client id/secret
    --- gdrive can be very slow when iterating over lots of small files.

what exacly is pdfgrep doing, reading the contents of all the .pdf files or what?

and, oh wait, here comes @ole, yeah!

5

Try changing to:

--vfs-cache-mode full

https://rclone.org/commands/rclone_mount/#vfs-cache-mode-full

6

Pretty much. Hence my point about FUSE limitations :slight_smile:

7

not at all sure that is correct.

how many total pdf files, what is the average file size?

8

I've had suspicions this mode might help. I'll give it a go thanks.

9

Perhaps around 1000 PDF files ranging from around 1MB to 50MB in size.

10

and for pdfgrep lognormal *.pdf, how long did it take to complete?

11

Remember to make two searches. The first will fill your cache. You will hopefully see the effect on the second search, if your cache-size is sufficient to hold all your pdf files.

12

Too long to wait for.. killed the process after 1h

13

Noted. Will do. Actually thought my "vfs/refresh" command would do that. Curiously, if I run

$ rclone rc vfs/refresh dir="Books/Maths/PSPapers/" recursive=true --rc-addr 127.0.0.1:5572

I only see the following in the debug log:

2021/12/29 22:40:52 DEBUG : rc: "vfs/refresh": with parameters map[dir:Books/Maths/PSPapers/ recursive:true]
2021/12/29 22:40:52 DEBUG : Books/Maths/PSPapers: Reading directory tree
2021/12/29 22:40:53 DEBUG : Books/Maths/PSPapers: Reading directory tree done in 1.129642169s
2021/12/29 22:40:53 DEBUG : rc: "vfs/refresh": reply map[result:map[Books/Maths/PSPapers/:OK]]: <nil>

which is strange as there are at least 30 PDF documents in this folder and I would expect log to iterate of each file.

14

for 995 pdf of size 1.3MB stored in wasabi, s3 clone.

rclone size wasabi01:zork/pdf
Total objects: 995 (995)
Total size: 1.240 GiB (1331661235 Byte)

rclone mount wasabi01:zork/pdf /home/user01/rclone/mountpoints/zork/ --allow-other --read-only --vfs-cache-mode=full

ran these commands one after another

  • the first two commands are exactly the same.
  • the first command is the slowest, as rclone needs to download the file contents in the rclone local vfs file cache.
  • the second command is much faster, as rclone does not need to re-download the file contents, just read it from local vfs file cache.
  • the third command reads the data direct from the local rclone vfs cache, in effect bypassing rclone overhead.
time pdfgrep lognormal /home/user01/rclone/mountpoints/zork/*.pdf

real    5m13.966s
user    0m48.156s
sys     0m3.677s

time pdfgrep lognormal /home/user01/rclone/mountpoints/zork/*.pdf

real    1m32.329s
user    0m48.168s
sys     0m3.707s

time pdfgrep lognormal /home/user01/rclone/cache/zork/vfs/wasabi01/zork/pdf/*.pdf

real    0m45.597s
user    0m44.112s
sys     0m1.480s
1 Like
15

Very useful metrics. Good to know what's possible with S3. Thanks

16

I resolved the issue with cat as my first command. Since the files in this folder are relatively large on average (ie 20-50MB) it seems pdfgrep gets caught up on each file. So I ran:

find . -type f -exec cat {} \; >/dev/null

and then I could see the folder being completely populated in the vfs cache folder rather quickly. I could then run pdfgrep there (much like @asdffdsa suggested in his metrics comment) or in the mount point. Both coming back with results in reasonable time.

I perhaps went wrong in assuming the rc command vfs/refresh would do this for me.

Thanks for your help @asdffdsa and @Ole!

17

Great, happy you got it working!

Correct! You probably didn't know, that vfs has two caches:

You can see this by browsing (ls) your cache folder.

vfs/refresh only updates the directory cache.

You may want to dive a bit deeper in the distinction and then check/tune your vfs parameters to this new understanding.

18

Yes you are correct. In my post marked as Solution I incorrectly wrote vfs cache whereas I meant that folder specified by cache-dir. Yes I might do a deeper dive to tune my vfs parameters. Perhaps after I've exhausted my new found productivity! Thanks again :slight_smile:

1 Like
19

might take a read of this summary i created

https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34

20

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.