Cache - Persistence?

Help and Support
1

I’m still trying to figure out what’s missing for me to move over from plexdrive. In general, I don’t wan to do anything special in plex/sonarr/radarr or anything. I just want things to work more out of the box.

I use a local mount point to store anything for 5-7 days till I move it to the cloud. I have an automated daily process “rclone move” to move from local to the cloud.

I use mergerfs to combine the local and cloud mount by using plexdrive+rclone crypt.

I can see from many of the posts, rclone cache can poll and detect changes which seems to be closer.

My other main thing from plexdrive is the fact that once it scans, I have a local DB of the contents and it only updates that from changes. I can scan from anything without creating any API hits and it knows the basic information from the files.

My scans now are super fast:
[root@gemini ~]# time ls -alR /TV/ | wc -l
20581

real	0m0.762s
user	0m0.086s
sys	0m0.108s
[root@gemini ~]# time ls -alR /Movies/ | wc -l
8653

real	0m0.774s
user	0m0.019s
sys	0m0.059s

I did some initial testing over the last day and noticed if I restarted the rclone mount, it seemed to have to rescan everything again creating a lot of API hits. I wanted to make sure it wasn’t my mount command or something else:

rclone mount --buffer-size 256M --checkers 8 --cache-writes --allow-non-empty --allow-other --uid 1000 --gid 1000 --vfs-cache-mode off --vfs-cache-max-age 1h0m0s --log-level DEBUG gmedia: /gmedia

2

The cache database is persistent between remounts, got a debug log to paste?

3

My general testing is mount, run a nice ls -alR | wc -l with a time in front of it and see what it does.

It takes ~10-15 minutes each time and doesn’t even seem to ‘cache’ the structure. Not sure if I’m using the right rclone terms.

2018/03/23 10:26:24 DEBUG : Movies/Horrible.Bosses.2.(2014)/: >Lookup: node=Movies/Horrible.Bosses.2.(2014)/Horrible.Bosses.2.1080p.mkv, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/Horrible.Bosses.2.(2014)/Horrible.Bosses.2.1080p.mkv: Attr:
2018/03/23 10:26:24 DEBUG : Movies/Horrible.Bosses.2.(2014)/Horrible.Bosses.2.1080p.mkv: >Attr: a=valid=0s ino=0 size=9386364668 mode=-rw-rw-r--, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/Horrible.Bosses.2.(2014)/Horrible.Bosses.2.1080p.mkv: Attr:
2018/03/23 10:26:24 DEBUG : Movies/Horrible.Bosses.2.(2014)/Horrible.Bosses.2.1080p.mkv: >Attr: a=valid=0s ino=0 size=9386364668 mode=-rw-rw-r--, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/: Lookup: name="Host,.The.(2013)"
2018/03/23 10:26:24 DEBUG : Movies/: >Lookup: node=Movies/Host,.The.(2013)/, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: Attr:
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: >Attr: attr=valid=0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/: Lookup: name="Host,.The.(2013)"
2018/03/23 10:26:24 DEBUG : Movies/: >Lookup: node=Movies/Host,.The.(2013)/, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: Attr:
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: >Attr: attr=valid=0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: Attr:
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: >Attr: attr=valid=0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: Attr:
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: >Attr: attr=valid=0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013)/: ReadDirAll:
2018/03/23 10:26:24 DEBUG : Movies/Host,.The.(2013): Re-reading directory (7m1.912485948s old)
2018/03/23 10:26:24 DEBUG : : Statfs:
2018/03/23 10:26:24 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2018/03/23 10:26:25 DEBUG : : Statfs:
2018/03/23 10:26:25 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2018/03/23 10:26:26 DEBUG : Movies/Host,.The.(2013)/: >ReadDirAll: item=1, err=<nil>
2018/03/23 10:26:26 DEBUG : : Statfs:
2018/03/23 10:26:26 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2018/03/23 10:26:27 DEBUG : : Statfs:
2018/03/23 10:26:27 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2018/03/23 10:26:28 DEBUG : : Statfs:
2018/03/23 10:26:28 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>

That’s a snip of what’s it is doing for each file every time. My expectation is that once it’s there, it’s there and it knows the metadata for the file and doesn’t keep checking.

4

Is this the cache remote that you’re mounting or directly the google drive one? It looks to be the drive one. What’s your config like?

5

So apparently to cache things, you actually have to have your config pointing to the cache dir.

I had a typo in my config and I was pointing from GD->Crypt instead of going from GD->Cache->Crypt…

Which I can see now the cache building properly…

#facepalm

Happy I posted as it’s fixed now, but I feel dumb.

6

i dont understand why off cache has age?

7

I had an old command that I updated based on some of the other posts. My current mount is:

/usr/bin/rclone mount gmedia: /gmedia --allow-other --uid 1000 --gid 1000 --syslog --buffer-size=16M --cache-chunk-size=32M --cache-info-age=168h --cache-tmp-wait-time=10m --cache-workers=5 --attr-timeout=1s -v

It seems like the mount are good and I can get fast directory listings and such. Just having a problem streaming now, if I play a file, I get:

Mar 23 13:59:20 gemini rclone[19185]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 23 13:59:20 gemini rclone[19185]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
Mar 23 13:59:20 gemini rclone[19185]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
Mar 23 13:59:25 gemini rclone[19185]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 23 13:59:25 gemini rclone[19185]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
8

So my current rclone config as well:

[GD]
type = drive
client_id = client_id
client_secret = secret
token = {"access_token":"token","expiry":"2018-03-23T14:34:43.556384008-04:00"}

[gmedia]
type = crypt
remote = gcache:
filename_encryption = standard
password = somepassword
password2 = somepassword
directory_name_encryption = true

[gcache]
type = cache
remote = GD:media
plex_url = http://127.0.0.1:32400
plex_username = User
plex_password = Password
plex_token = Token
9

I haven’t seen that in a while. What’s your rclone version? rclone -V

10

[root@gemini log]# rclone -V
rclone v1.40

  • os/arch: linux/amd64
  • go version: go1.10
11

Ok. Can you post some more logs? There must be something else funky happening. That’s just says there’s a problem with the reading but it doesn’t say what it might be.

None of the files work at all? No data is playing? I’m inclined to think you might be banned.

12

I initially thought that might be it, but when I flip my config back to plexdrive, I can play everything without a problem. I’ve tried a number of files and the same file that wasn’t playing before via the direct rclone cache mount.

Here is a bigger clip in debug:

Mar 23 16:15:57 gemini rclone[5288]: /: Lookup: name="Movies"
Mar 23 16:15:57 gemini rclone[5288]: /: >Lookup: node=Movies/, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: Movies/: Attr:
Mar 23 16:15:57 gemini rclone[5288]: Movies/: >Attr: attr=valid=0s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: /: Lookup: name="Movies"
Mar 23 16:15:57 gemini rclone[5288]: /: >Lookup: node=Movies/, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: Movies/: Attr:
Mar 23 16:15:57 gemini rclone[5288]: Movies/: >Attr: attr=valid=0s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: Movies/: Lookup: name="King.Arthur.Legend.of.the.Sword.(2017)"
Mar 23 16:15:57 gemini rclone[5288]: Movies/: >Lookup: node=Movies/King.Arthur.Legend.of.the.Sword.(2017)/, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/: Attr:
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/: >Attr: attr=valid=0s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/: Lookup: name="King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv"
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/: >Lookup: node=Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: Attr:
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: >Attr: a=valid=0s ino=0 size=9379617618 mode=-rw-r--r--, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: Attr:
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: >Attr: a=valid=0s ino=0 size=9379617618 mode=-rw-r--r--, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: u
nexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 23 16:15:57 gemini rclone[5288]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (
10485760/9381907602) error (unexpected EOF) response
Mar 23 16:15:57 gemini rclone[5288]: &{Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv (r)}: Read: len=131072, offset=458752
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.seek from 393216 to 458752 (io.Seeker)
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read io.Seeker failed: unexpected EOF
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
Mar 23 16:15:57 gemini rclone[5288]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.seek from 393216 to 458752
Mar 23 16:15:57 gemini rclone[5288]: &{Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv (r)}: Read: len=65536, offset=393216
Mar 23 16:15:57 gemini rclone[5288]: /: Lookup: name="Movies"
Mar 23 16:15:57 gemini rclone[5288]: /: >Lookup: node=Movies/, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: Movies/: Attr:
Mar 23 16:15:57 gemini rclone[5288]: Movies/: >Attr: attr=valid=0s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
Mar 23 16:15:57 gemini rclone[5288]: /: Lookup: name="Movies"

I use a mergerfs mount to bring the local and cloud together, but it’s the exact same mount that I use on plexdrive minus changing the /media to /gmedia.

/usr/bin/mergerfs -o defaults,allow_other,use_ino,category.action=all,category.create=ff /data/local/movies:/media/Movies /Movies

And it does seem to be every file I’ve tried via the direct rclone mount. I can’t get any to work.

13

You can see it’s the same file playing right now via plexdrive:

Fri Mar 23 17:43:01 EDT 2018
[root@gemini system]# lsof /GD
COMMAND   PID  USER   FD   TYPE DEVICE   SIZE/OFF                NODE NAME
rclone  10243 felix    5r   REG   0,48 9381907602 7343681852433449881 /GD/media/tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g
[root@gemini system]#
14

Can you try without cache and see if it works? It sounds like you’re also adding a bunch of tools in between the flow which just makes things harder to troubleshoot. Here’s what I suggest:

  1. mount the drive remote from rclone: list a folder, open a small file - check if this works
  2. mount the cache -> drive remote from rclone: list a folder, open a small file - check if this works
  3. mount the crypt -> cache -> drive remote from rclone: list a folder, open a small file - check if this works
  4. add whatever tools are in between and try the same steps as before

Paste the logs where it fails

15

I use 1 tool which is like unionfus but mergerfs instead. Otherwise, it’s a straight mount with rclone.

I removed the cache and tried to mount directly and that works without an issue on the same file I was having problems with.

When it fails, I get the same logs I’ve posted already, which look like:

Mar 24 07:19:09 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 24 07:19:09 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
Mar 24 07:19:09 gemini rclone[7046]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
Mar 24 07:19:11 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 24 07:19:11 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
Mar 24 07:19:11 gemini rclone[7046]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 2/10: unexpected EOF
Mar 24 07:19:14 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 24 07:19:14 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
Mar 24 07:19:14 gemini rclone[7046]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 3/10: unexpected EOF
Mar 24 07:19:16 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 24 07:19:16 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
Mar 24 07:19:16 gemini rclone[7046]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 4/10: unexpected EOF
Mar 24 07:19:18 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 24 07:19:18 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
Mar 24 07:19:18 gemini rclone[7046]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 5/10: unexpected EOF
Mar 24 07:19:21 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 24 07:19:21 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
Mar 24 07:19:21 gemini rclone[7046]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 6/10: unexpected EOF
Mar 24 07:19:22 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: cache reader closed 10485760
Mar 24 07:19:22 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: moving offset set from 0 to 0
Mar 24 07:19:23 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: cache reader closed 32
Mar 24 07:19:23 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: moving offset set from 0 to 10357248
Mar 24 07:19:23 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 24 07:19:23 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response
Mar 24 07:19:23 gemini rclone[7046]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.Read error: low level retry 7/10: unexpected EOF
Mar 24 07:19:23 gemini rclone[7046]: Movies/King.Arthur.Legend.of.the.Sword.(2017)/King.Arthur.Legend.of.the.Sword.(2017).Bluray-1080p.mkv: ReadFileHandle.seek from 10354688 to 10354688
Mar 24 07:19:24 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: cache reader closed 10485760
Mar 24 07:19:24 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: moving offset set from 0 to 0
Mar 24 07:19:25 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: cache reader closed 32
Mar 24 07:19:26 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: moving offset set from 0 to 10357248
Mar 24 07:19:26 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: unexpected conditions during reading. current position: 10485760, current chunk position: 0, current chunk size: 10485760, offset: 10485760, chunk size: 33554432, file size: 9381907602
Mar 24 07:19:26 gemini rclone[7046]: tnvepu36qiohcun8v84ddhsam0/u87r7rtdmifgo9l0dpiooiqbru19nok9c2qrf24qb9mg6snptu36dgglhe04dok69trgm4qofr3eo/kp2rfperqsuqnm1k8j505u5orstu7gr48f0850bnq3kdo8cuie71rhipmc297hq604rhv063b3r2uof6ebj5m112bn9j342mh6r662g: (10485760/9381907602) error (unexpected EOF) response

I can wipe out my cache db and start over and let it redo to see if thats it.

My mounts:

felix     7046     1  2 07:16 ?        00:00:23 /usr/bin/rclone mount gmedia: /gmedia --allow-other --uid 1000 --gid 1000 --syslog --buffer-size=256 --cache-chunk-size=32M --cache-info-age=168h --cache-workers=8 --log-level DEBUG --read-only --acd-templink-threshold 0

felix    27264 26904  0 07:25 pts/4    00:00:00 /usr/bin/rclone mount gnoncache: /gnoncache --allow-other --uid 1000 --gid 1000 --syslog --buffer-size=256 --log-level DEBUG --read-only --acd-templink-threshold 0

My config I added for the non cache mount:

[gnoncache]
type = crypt
remote = GD:media
filename_encryption = standard
password = something
password2 = something
directory_name_encryption = true

I tested a bunch of other files that worked. A number of small TV shows. I picked a couple 4k movies that are 30GB+ and they worked fine. I’m wondering if that one particular movie is messed up in the cache.

16

Ah so it’s not all of them. If the persistent data is from an older version then you have to start fresh (use --cache-db-purge).

Try that and see how it works. Ultimately there’s also the rc command that you can use to purge a single file/directory

17

That did seem better as I can get listings and mediainfo. Anything I should set to help with 403s? I noticed I had some of these now popping up and I wanted to avoid any bans.

Mar 24 07:56:02 gemini rclone[31362]: tnvepu36qiohcun8v84ddhsam0/ihvahhru95h4ganpr6qhsn389pomvornh11jfo3deq2leoldnvs0/7uficeq5c7qgu4geq3nl6rhluob6mkq0nubgv6jnqi153dblj8ig: 0: chunk retry storage: 0
Mar 24 07:56:02 gemini rclone[31362]: worker-9 <7uficeq5c7qgu4geq3nl6rhluob6mkq0nubgv6jnqi153dblj8ig>: object open failed 234881024: bad response: 403: 403 Forbidden
Mar 24 07:56:02 gemini rclone[31362]: worker-7 <7uficeq5c7qgu4geq3nl6rhluob6mkq0nubgv6jnqi153dblj8ig>: object open failed 268435456: bad response: 403: 403 Forbidden
Mar 24 07:56:02 gemini rclone[31362]: worker-10 <7uficeq5c7qgu4geq3nl6rhluob6mkq0nubgv6jnqi153dblj8ig>: object open failed 335544320: bad response: 403: 403 Forbidden
Mar 24 07:56:02 gemini rclone[31362]: pacer: Rate limited, sleeping for 1.800972481s (1 consecutive low level retries)
Mar 24 07:56:02 gemini rclone[31362]: pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
Mar 24 07:56:02 gemini rclone[31362]: pacer: Rate limited, sleeping for 2.083146229s (2 consecutive low level retries)
Mar 24 07:56:02 gemini rclone[31362]: pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
Mar 24 07:56:03 gemini rclone[31362]: pacer: Resetting sleep to minimum 10ms on success
18

Give it some time to expire.

Second, what’s the order of the wrapping? Is it crypt -> cache -> drive? Are you mounting the crypt remote? It’s important

19

I’m GD->Cache->Crypt. Hoping that’s the right way.

20

Specifically, my rclone:

[GD]
type = drive
client_id = myid
client_secret = secret
token = {"access_token":"ytoken","token_type":"Bearer","refresh_token":"1/YcvLP-bQKnDEOeT-xHwyJikJUdN6AchuBAbd8sESO1Q","expiry":"2018-03-24T08:15:10.992435959-04:00"}

[gcache]
type = cache
remote = GD:media
chunk_size = 8M
info_age = 5m
chunk_total_size = 64G

[gmedia]
type = crypt
remote = gcache:
filename_encryption = standard
password = something
password2 = something
directory_name_encryption = true

and my rclone mounts gmedia: