I am currently using a Google Drive mount to provide space for Radarr to copy my movies to. I'm not using hardlinks, as I don't want to seed from Drive. After a certain amount of time, in excess of seeding requirements, the torrent client pauses the torrent and, periodically, I manually go an remove/delete it.
Now, I used to run a similar setup (still do) for Sonarry. But in that case files tend to be smaller and I had never noticed/cared for how it actually works behind the scenes. Now what I'm seeing with movies is that when Radarr starts the importing of a movie, rclone first copies it all in its cache and only later starts uploading.
A test with Rclone Browser (which I suppose uses a simple copy command) has the upload starting immediately. Now, I know that there are reasons for wanting to have the file in cache before uploading, in the case of a mount.
But since the whole process is making my system... glitchy? Slightly unresponsive? I was wondering if there was a way to approach the "copying to Drive" from Radarr in a different way.
As far as I can see Radarr does not offer the ability to run imports through a CLI command (you can run custom scripts based on events, but there is no way that I can see to stop it from copying the movie to its destination directory), otherwise I guess it would be pretty simple. It relies on filesystem to copy stuff to the folders you point it to.
I saw that in the past animosity22 was using a mergerfs (on which I'm totally ignorant) solution, but he has moved away from that and now I can't see what he was actually doing with it.
To sum it up: I would like to straight copy stuff to Drive, avoiding the cache, when using Radarr.
While waiting for answers I was in fact looking at the different options for cache mode.
But if I'm not mistaken writing directly through mount, while not writing to cache, offers zero error correction, as opposed to what happens when using rclone copy. Am I mistaken?
With big files I fear that could be a problem. You used to automate in a different way, are you now relying simply on mount with cache mode full?
(it even leads to lots of disk wearing, which I'd rather avoid, if possible)
Yeah, not sure how related to cache or not cache. I don't know what you mean by that.
Yes, I just use full and let it fly.
So my theory on this one - By the time my cache SSD wears out, it'll be well past the time to replace it. I use one SSD for my cache and beat the crap out of it and at my current rate, it may die in 5-6 years and I'm positive it would be replaced by then so that's $100ish for 5-6 years of use is fine to me. I tend to replace my hardware in pieces every 3-4 years based on new technology coming out that reduces power, improves speed, etc.
Just replaced my huge plex box with a tiny System76 system that should easily last me for 4-5 years. Every person tends to have a unique set of requirements and what they want to work. My setup is geared specifically for me / my number of users and my overall goals of simplicity / hands off and the least amount of moving parts as possible.
If disk space was high on my list, I'd be still with mergerfs as I can hard link and I don't 'double' count my storage for a period of time, but I wanted less parts to I removed mergerfs/upload script/etc.
Files start uploading as soon as copy starts, which is my desired outcome.
I now found the actual piece I had in mind regarding errors, etc.
rclone mount vs rclone sync/copy
File systems expect things to be 100% reliable, whereas cloud storage systems are a long way from 100% reliable. The rclone sync/copy commands cope with this with lots of retries. However rclone mount can't use retries in the same way without making local copies of the uploads. Look at the [VFS File Caching] for solutions to make mount more reliable.
With "minimal" I think I'm not doing anything to make the mount "more reliable" where uploads are concerned. Am I wrong?
Hmmm... I'm actually surprised by *arr behavior here.
Even with vfs cache minimal it copies the file somewhere before uploading. Aaaand... it's copying it under /home/ashlar/.rCache/vfs/gdrive{mLCP3}/DestinationFolderOnDrive/
Where .rCache is my cache directory.
I'm not using hard links because, from the description where it says "hardlinks will work only if source and destination are on the same volume".
But yeah, *arr started uploading the file only after it copied it whole under the cache directory. Cache directory that it shouldn't even be aware of.
I had just copied a file manually (drag and drop) and it started uploading straight away. I can't understand what's happening.
Yeah, I'm sorry... I did not think I would end up in this strange situation. I was simply testing after manually copying and seeing intended behavior, so I did not think about setting up debug logging.
I will follow up with all relevant info as soon as possible.
The gist of it, though, is manually copying by drag and drop from local to remote Drive mount starts uploading straight away (after setting vfs-cache to minimal), whereas Radarr appears to still be copying the whole file under the cache directory. It's as if it's "haning on" to the old configuration somehow. But there's just one instance of rclone running...
2023/04/20 21:54:37 ERROR : Temp/SomeVideo.mkv: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
Don't know what Radarr is doing but I've wasted enough time with this and I don't wanna waste any more of yours (or anybody else's). I'll stick to vfs cache full and call it quits.