I mentioned on the Github that I’m having trouble uploading files via rclone mount (1.46, and tested with latest beta today) to an encrypted gdrive on Windows 10. The drive becomes inaccessible for reading until the upload task finishes. When the problem happens, the drive will just show up like this (unreadable), and there’s nothing you can do about it.
My mount command: rclone mount gcrypt: G: --allow-other --bind 192.168.1.123 --dir-cache-time 72h --fast-list --cache-chunk-path=X:\rclone-cache --cache-db-path=X:\rclone-cache --log-level INFO --vfs-cache-mode writes --cache-tmp-upload-path=X:\rclone-upload --cache-dir=X:\rclone-cache -P --drive-chunk-size 32M --buffer-size 256M --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --rc
(I know some of those flags might be redundant when I’m just using normal crypt+drive vfs, but don’t think they’d be the problem)
rclone.conf is default other than client key and use_trash = false.
I have my stuff download into a temporary folder from Sonarr, then be moved into the GDrive mount when it has finished downloading. Then Plex streams from the GDrive mount. But when Sonarr is doing any uploads (particularly if there are several files uploading at once), it almost always locks up the mount until the uploads finish, so you won’t be able to stream anything on Plex until that finishes.
There is nothing useful in the -vv log when this situation happens. It simply keeps printing: 2019/03/22 10:59:16 DEBUG : <encrypted filename> : Sending chunk 1409286144 length 33554432 every 3 seconds. Once the upload finishes, stuff resumes working, but it’s a huge problem for me that it can’t simultaneously read stuff while uploading.
Is there any real workaround to this that I could try? (I would prefer to avoid hacking up something to use rclone copy with Sonarr) I have 1000/50 connection, so it can become quite bogged down when uploads happen – but normally everything else works amazing. Thanks for any ideas!
I didn’t feel it important, because it simply happens when the “Sending chunk” appears in log, but here you go, I can hope it helps. At the beginning of the log, Radarr adds the file into the mount; then all chaos happens when “sending chunk” happens, and only resolves itself once it’s done uploading all the chunks.
Basically, whatever is happening during that upload process is making everything else in rclone freeze up, and I don’t know why.
Could it be using all you bandwidth to upload?
Yep, but that shouldn’t be a problem? If I max out my upload doing anything else, rclone is still mostly responsive, whereas this situation completely hangs the program. (FWIW, if I use --bwlimit, it just makes the problem worse, because it takes longer for chunks to upload :/)
Edit: if I have it set up basically the same on Linux, it seems to work okay with downloading at the same time as uploading. So it seems to be something Windows-specific. Any ideas? Cheers.
if rclone is using up all your upload bandwidth then the downloads will stop working too.
Hm, why should everything stop if there’s a file being uploaded? Is this a technical limitation, or am I misunderstanding something? Here you say it’s multithreaded.
Using --bwlimit just makes it worse, because it makes the files take even longer to upload. I have even tried setting it to values much lower than my pipe: same result, but even longer-lasting lockup.
Like, the problem is software-based in some kind of way; it shouldn’t be related to networking – other than it hurts me harder than someone with symmetric gigabit, simply because it’d take longer for my files to upload. Everything just completely locks up when rclone is uploading a file into mount, simple as that. (For example, if I take my computer over to my friend’s house with 1000/1000, same problem would happen)
Haven’t noticed any corruption yet, but it does seem like the freezing can happen, it’s just a bit harder to happen (you just need to send a lot more data at it than before).
Curious, does this not happen on Linux? Could chalk it up to “FUSE on Windows isn’t perfect”, which is understandable – it’s amazing what the WinFsp dev has managed to do within the constraints of Windows.
It doesn’t happen with the normal mount on linux, but if you use the alternative rclone cmount (which iisn’t normally compiled in) which uses FUSE then it does happen.
I could just do this fix for windows which would limit the blast radius…
