Crypt + Cache + Mount not playing nice with Sonarr

Help and Support
1

What is the problem you are having with rclone?

When Sonarr has downloaded an episode, it will move the episode from a local disk to my mounted gdrive->crypt->cache.

It moves the file successfully, but it looks like it cannot find the file right after moving it, and thus expecting that the move failed. After manually doing a "Update series info and scan disk" on the show, it recognizes that the file is indeed there. However, the episode still sits in the activity queue, now saying it cannot find a file to import.
This also means that Sonarr doesn't send a message to Plex server to update the library.

What is your rclone version (output from rclone version)

rclone v1.48.0-016-ga1840f6f-beta

  • os/arch: windows/amd64
  • go version: go1.12.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

Here's my conf:

[GDrive]
type = drive
client_id = Censored
client_secret = Censored
scope = drive
token = Censored

[GCache]
type = cache
remote = GDrive:Media
plex_url = http://localhost:32400/
plex_username = Censored
plex_password = Censored
chunk_size = 5M
info_age = 1h
chunk_total_size = 10G

[Media_vfs]
type = crypt
remote = GCache:
filename_encryption = standard
directory_name_encryption = true
password = Censored
password2 = Censored

And here's my mount command:
mount --allow-other --buffer-size 1G --dir-cache-time 48h --drive-chunk-size 64M --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit 2G --cache-chunk-path "F:\RClone\cache-chunk-path" --cache-dir "F:\RClone\cache-dir" --cache-tmp-upload-path="I:\RClone\cache-tmp-upload-path" Media_vfs: N: --config "C:\Users\Inrego\.config\rclone\rclone.conf" --vfs-cache-mode writes

2

I wouldn't be surprised if this has to do with the tmp-upload function which adds another layer on the upload route. the VFS is really good about seamlessly swapping between the in-cache file (still uploading) and the remote (upload done). I am not sure if the cache backend does that job quite as transparently.

Try testing without this and see if the problem persists.

Generally I think I would advice you to not use either tmp-upload or the cache-writes functions of the cache backend unless you have a very good reason for it. After much testing I found there to be way too many bugs and problems with it - and the cache backend is not actively being maintained anymore by the original author, making improvements stagnate. It's likely according to NCW that the VFS will be integrating the missing features cache backend provides in the future and pretty much replace it entirely.

Personally I have moved away from using the cache backend at all recently. The only function I can't adequately replicate using only the VFS is the actual read-caching, and hopefully the VFS will be able to fill this job soon and do it far more efficiently.

3

Just to be clear. What you're saying is remove the --cache-tmp-upload-path parameter from my mount?

4

Yes. At least temporarily for testing to see if it is related to your problem.

5

That did the trick! Thanks a bunch!
Tbh I thought that the cache-temp-upload would always be used. But if I didn't specify it, it would just use a default path. I only included the parameter because I thought I was merely changing it to use a different path (on a different drive).

6

I agree it's a little unconventional. You'd expect there to be an "enable" for it, but apparently not.

"Specifying a value will enable this feature. Without it, it is completely disabled and files will be uploaded directly to the cloud provider"

7

Hmm, seems I was a bit quick on the trigger there. Sometimes it works now, but mostly I still have the same issue.

8

Looks like some kind of tmp upload path is used. Because I can see on my network, it's still sending a lot of data for a while after the transfer to the mount is "complete".

9

Hmm.. Sometimes it works it seems. I've noticed a few bottlenecks on my server, so I'm going to switch up which drives are used for what, and try again.

10

Well, all uploaded files go via the VFS write cache (when that is enabled), so this is normal. Once they hit the mount they will be considered to be on the mounted drive as far as the OS is concerned (although this is accomplished via a bit of fakery and the network transfer isn't done).

So are you having the exact same issue still?
Try manually checking that you can upload a large file and access it from the mounted cloud drive immediately after it has finished the local copy process (but still hasn't actually uploaded). This process should be transparent to the OS and have a seamless handover from the local file to the remote file.

This sound to me like it is the stop where something is going wrong - unless of course this is some kind of Radarr-spesific issue. If that is the case I can't do much to assist as I am not familiar with that software.

At this point it might be a good idea to give us a full debug-log of the problem.
Add...
-vv
...to your launch command to enable debug output.
Now, go replicate the problem (and try to keep it short and sweet, only do what illustrates the problem).
For long long (and debug logs tend to go long), use...
--log-file=FILE
...to send the debug to file rather than the screen

Post it for us so we can have a look once you are sure you captured the problem.
You may have to upload long files to pastebin or other such services and provide a link

11

I'm testing now with log enabled. In regards to your questions about Radarr/Sonarr, they simply just move the file to a specific folder on the mounted drive. And looks like they do a check if file exists right after the move - and this check seems to be what is failing in Sonarr and Radarr.

12

Yea, that's how I understood it from your description also. This shouldn't be happening though, as all files that have made it into the cache will appear to be on the mounted drive even if they are still uploading.

13

Sorry in advance for the quite lengthy log file that you warned me about. But it ran pretty solid for a while before I could replicate it again. But alas, it did happen.

The file it happened with, is Marvels.Daredevil.S03E13.2160p.WEBRip.X264-DEFLATE English.mkv

14

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.