Sonarr / Radarr Hangs when using cache temp upload path

andy012345 · April 25, 2018, 5:12pm

It seems ever since the cache temp upload path was implemented, sonarr and radarr randomly hang when any of their respective files they have downloaded are busy uploading.

Does anyone have a workaround to this?

andy012345 · April 25, 2018, 5:16pm

Seems to be caused by a better version of something being found and trying to delete the old file:

18-4-25 10:13:11.4|Info|RecycleBinProvider|Recycling Bin has not been configured, deleting permanently. /tv/The Flash (2014)/Season 4/The Flash (2014) - S04E19 - Fury Rogue WEBDL-720p.mkv

Sonarr has been hung for just over 8 hours now.

andy012345 · April 25, 2018, 5:23pm

This is on rclone v1.40-116-gbe8bd896β btw

Same can be observed in SSH with mv and rm commands, both will hang as the file is in the upload queue.

mv:

2018/04/25 18:21:24 DEBUG : tv/The Flash (2014)/Season 4/: Rename: oldName=“The Flash (2014) - S04E19 - Fury Rogue WEBDL-720p.mkv”, newName=“test.mkv”, newDir=tv/The Flash (2014)/Season 4/
2018/04/25 18:21:24 DEBUG : Cache remote cache:: moving obj ‘tv/The Flash (2014)/Season 4/The Flash (2014) - S04E19 - Fury Rogue WEBDL-720p.mkv’ -> tv/The Flash (2014)/Season 4/test.mkv

rm:
2018/04/25 18:22:50 DEBUG : tv/The Flash (2014)/Season 4/: Attr:
2018/04/25 18:22:50 DEBUG : tv/The Flash (2014)/Season 4/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2018/04/25 18:22:50 DEBUG : tv/The Flash (2014)/Season 4/: Lookup: name=“The Flash (2014) - S04E19 - Fury Rogue WEBDL-720p.mkv”
2018/04/25 18:22:50 DEBUG : tv/The Flash (2014)/Season 4/: >Lookup: node=tv/The Flash (2014)/Season 4/The Flash (2014) - S04E19 - Fury Rogue WEBDL-720p.mkv, err=
2018/04/25 18:22:50 DEBUG : tv/The Flash (2014)/Season 4/The Flash (2014) - S04E19 - Fury Rogue WEBDL-720p.mkv: Attr:
2018/04/25 18:22:50 DEBUG : tv/The Flash (2014)/Season 4/The Flash (2014) - S04E19 - Fury Rogue WEBDL-720p.mkv: >Attr: a=valid=1s ino=0 size=1110379631 mode=-rw-r–r--, err=
2018/04/25 18:22:50 DEBUG : tv/The Flash (2014)/Season 4/: Remove: name=“The Flash (2014) - S04E19 - Fury Rogue WEBDL-720p.mkv”

darthShadow · April 25, 2018, 6:10pm

You could try increasing the --cache-tmp-wait-time to prevent uploading till the processing has finished. I find that 60min is a good value to account for larger files too. With this value, I have never encountered any issues or hang ups.

andy012345 · April 25, 2018, 6:56pm

My wait time is 600m atm, the problems happen when something is uploading but not the file it’s trying to modify

andy012345 · April 25, 2018, 8:11pm

Doesn’t seem to be related to sonarr / radar, seems to affect simple commands too.

Have file #1 uploading
Copy file #2 to drive
Attempt to mv or rm file #2 while #1 is uploading = hang

Animosity022 · April 26, 2018, 1:56am

I’ve been using the tmp_upload with 60 minute wait time for a few weeks now and run through quite some shows. Haven’t seen an issue with it yet. I do plenty of copies and renames via Sonarr/Radarr while many things are going on.

I use plexautoscan to deal with cache expiration as well.

rbhalla · April 27, 2018, 7:07pm

Is it possible that the copy to the tmp directory is IO bound? If the copy is taking up all your disk IOPS/CPU you may see sonarr and radarr start to hang until it’s complete. I’ve observed similar things on setups with hdds and large files.

andy012345 · April 27, 2018, 7:26pm

I don’t think so, it’s a VM on my home server which isn’t too taxed on the disc.

I’ve found a workaround is just to not use the --cache-tmp-upload-path option and have a background script copy things in with mergerfs. This just makes it so plex sees 2 files if it upgrades while there’s a copy in progress, which is the only drawback.

Animosity022 · April 28, 2018, 12:30am

Did you take a peek at plex_autoscan? Also, I have analyze off in both Sonarr/Radarr for me as I didn’t find much value in having that on.

camjac251 · June 1, 2018, 5:58pm

I’ve been noticing this too lately although just for Sonarr, not Radarr.

Sonarr runs on an SSD with my rclone mount and cache db also on the SSD but the cache itself and tmp upload folder are on the HDD. It will work just fine until a background upload starts, after that starts then Sonarr will hang until the upload it complete, which could take a while with the upload speeds I’ve noticed.

andy012345 · June 1, 2018, 6:24pm

I try out new beta builds every few weeks and it’s still the same, move or delete on an uploading file hangs, so sonarr typically hangs when a new show has come out and it’s deleting a lower quality version (removing a 720p copy to upgrade to 1080p for example).

reticentnz · June 2, 2018, 11:00am

I used Sonarr / Radarr but with union fs instead and have had no problems with this setup for about 18 months. Sonarr writes to a local writable folder, rclone picks up the files (with a rclone move crontab). It might not be what you’re after but it’s another way to achieve the same thing if you’re having problems with the cache temp. I was planning on changing over to that at some point but may wait until it’s a bit more mature.

camjac251 · June 6, 2018, 3:19pm

This sadly isn’t only affecting the rclone cache setup. If the vfs writes mode is used, it seems to cause Sonarr/Radarr to hang as well when a file is done being written to the drive and starts to upload to the remote. It will only resolve again after the transfer is complete.

ybd · July 2, 2018, 5:22pm

Having the same issue unfortunately. Did anyone ever manage to resolve?

jrock · July 8, 2018, 6:37am

Another “me too” post - I am having the same issue on OSX High Sierra.

To summarise what some of us seem to be experiencing:

When using --cache-tmp-upload-path and any file is being uploaded from the local queue/cache over the internet to GDrive (ie. it actually is in the process of uploading, not still waiting for --cache-tmp-wait-time period):
– Any attempt to modify any file in the local queue/cache (not necesarilly the one currently being uploaded) leads to a hang.
– The operation hangs regardless of whether we are attempting to modify the file programatically (eg. sonarr) or by using the command line.
– @camjac251 noticed that this also/still occurs when using the VFS cache.
Using a longer --cache-tmp-wait-time period does not resolve this because sonarr adds new/upgraded files/shows to the local cache whenever it finds them and there is always a good chance that a previously found file/show has waited out the --cache-tmp-wait-time period and is now uploading.
Turning off the Analyze function in Sonarr/Radar will probably prevent file-upgrade-induced modifications occurring (and associated hangs). However this is also not a solution as sonarr seems to perform file modifications even during new file/show addition operations.

A workaround is to move to a more complicated setup:

Abandon rclone cache
Upload & local cache: Use unionfs (or mergerfs) with an overnight crontab rclone move
Download: Improve performance using rclone VFS cache

However, the workaround might not be needed (hopefully!) because @Animosity022 does not seem to have this issue (whether using the rclone cache setup or the mergerfs setup). So it might just be environmental.

Perhaps we are experiencing some sort of permissions/locking error. By design, the rclone temp cache locks the file currently beIng uploaded to prevent modification until the upload is completed. Perhaps something environmental is causing rclone to lock all files in the local temp cache in our environments?

@ncw, sorry to bother you, but any ideas for us to research or test here?

camjac251 · July 8, 2018, 8:39am

Thank you for making this post @jrock. I hope that a solution can be found.

I am using Gentoo as the host OS for rclone. Also using a regular user for mounting with rclone (no sudo) with the following mount config.

TZ=Timezone /home/user/bin/rclone mount gmedia: /folder/gmedia \
       --config=/folder/user/.config/rclone/rclone.conf \
       --bind IPv4 \
       --drive-chunk-size=512M \
       --cache-dir /home/user/rclone \
       --dir-cache-time 48h \
       --vfs-cache-max-age 1m \
       --vfs-read-chunk-size 64M \
       --vfs-read-chunk-size-limit off \
       --vfs-cache-mode writes \
       --buffer-size 512M \
       --umask 077 \
       --log-level INFO \
       --log-file=/folder/user/logs/rclone.log

I’m using mono 5.12.0.226.

I believe this is the issue, since it’s what I’ve experienced, When Sonarr is processing the file, if rclone is copying the file to the temp directory it has setup, it will not hang. Only when a file switches from finishing copying to the temp folder to it starting to upload does it hang Sonarr/Radarr. It’s weird those since if Sonarr is hanging from a transfer, Radarr is fine and unaffected and vice versa, so it might be with how Sonarr/Radarr reads the response after a file is starting to upload that causes the hang. I tried copying files myself with rsync --progress folderwith10files rclonemount to see how it worked and it doesn’t continue copying the files to the temp if it begins uploading another. It will instead throttle until the upload is complete. This might be causing mono to hang possibly if it is waiting for the copy to finish before responding again.

Animosity022 · July 8, 2018, 11:10am

Can you create a log file with some -vv that shows what is happening in rclone when a file is being upload and what Sonarr/Radarr is trying to do?

I can’t reproduce this as I’ve tried a few times.

I posted earlier I have analyze off.

For me, Sonarr or Radarr looks “hung” when it copies a file from it’s tmp location to the rclone area. It “hangs” as long as the copy goes which depends on the size of the file. If you are making the tmp area IO bound, it appears hung longer. Once the copy is done, it resumes and I personally haven’t seen an issue after that when an upload is happening.

ncw · July 8, 2018, 11:13am

Can you reproduce this with standard unix tools? Say modifying the file by echo >>file hello on to the end of it?

If so can you write up how to do it in a new issue on github?

What exactly hangs?

Animosity022 · July 10, 2018, 7:53pm

So I did a bit more testing on this item and what I believe happens is that the way Radarr / Sonarr work, they are waiting for the file copy to complete. If it’s local storage, that all happens relatively speaking, fast.

If it’s trying to copy up to the cloud and it creates some a lockup of the application until the file copy is released.

You can see this in your dmesg:

felix@gemini:~$ dmesg | grep mono
[12929.588343] INFO: task mono:3931 blocked for more than 120 seconds.
[12929.588510] mono            D    0  3931      1 0x00000004
[28637.695135] INFO: task mono:3931 blocked for more than 120 seconds.
[28637.695291] mono            D    0  3931      1 0x00000004

At that point, Sonarr/Radarr are locked up. I don’t think there is much that rclone can do to fix it as we are essentially making the OS thing the file system is local and it’s not. It’s a cloud based fs with latency that the OS isn’t expecting.

Easiest way to reproduce this is to use Radarr and grab a large 4k rip and let it copy using anything really. vfs-cache-mode writes or direct write to the mount or even the cache-tmp-upload.

From my understanding of cache-tmp-upload, it copies locally first so when the background upload happens, I wouldn’t think that would lock mono up as it’s all in the background and any IO would happen on the local file.

If someone has some tests with the cache-tmp-upload, check the dmesg for the mono locks and you can confirm it.