Rclone Mount Locking Up File Explorer - Windows

#1

I have a Windows 7 computer that I am trying to use with Radarr, Sonarr, Jackett, Deluge, and Rclone mount to load files to Gdrive. Once Deluge completely downloads a few files and pushes the completed files to the Rclone mount it locks up the the mount/file explorer. At this point I can no longer navigate to the mount in Windows file explorer, and Radarr (what I’m primary using) also produces a system error that the drive is unavailable. If I disconnect and reconnect the mount it is immediately fine and I start back at square one.

I was using the latest released version of Rclone but saw another post which I cannot seem to locate now that advised using the latest beta - I tried that with the same results.

I am mounting with the following command:
rclone mount --allow-other --vfs-cache-mode writes gcrypt: Q:

Similar issue (not the one referenced above):

#2

Can you run your mount with -vv and grab the log file and share the output?

If you can use the latest stable as well.

#3

I think this is the thread you are looking for

I warmed up the fix in that thread here

https://beta.rclone.org/branch/v1.47.0-035-g8fd25776-vfs-async-release-beta/ (uploaded in 15-30 mins)

Can you see if it works for you?

#4

Having the same issue as OP - copying from local drive to mounted gdrive (Im using FreeFileSync to do so). Just tried this beta version and it seems to be a little better. The transfer still seems to lock up, but eventually it gets itself sorted and moves to the next file. The problem is it seems to do this on every file, sometimes failing the copy and having to retry.

#5

This is the post I was looking for - and I started running the beta listed in that post. It did not fix the issue for me. I just loaded the new beta and will try that.

Unlike ModuRaziel my issue doesn't seem to resolve itself, I have to stop rclone and remount which fixes the issue temporarily.

I will post my -vv log from last night (as soon as I figure out how to host it for linking since it's huge) which involved multiple files being uploaded to Gdrive, as well as opening the mount in Windows file explorer. It looks like I had a few successful uploads before it locked up between 5AM and 9AM this morning.

#6

The most useful thing of all for debugging would be to run the mount with --rc then use curl http://localhost:5572/debug/pprof/goroutine?debug=1 on it when it has locked up as outlined here. That will tell me whether it is a deadlock or not and if it is, where it is.

If it is a deadlock that will be enough info to fix it (:crossed_fingers:).

#7

(Or rather than use curl you can put that URL into your web browser and paste the result if that is easier).

#8

Do you want us to just paste our output from that URL?

#9

Yes please - just paste the output

#10

im getting an error that new users can only post two links

edit: posted to justpaste.it here

#12

See the output at the link below - I don't believe I had any file transfers yet still had the lockup and am unable to access the mount.

https://jpst.it/1Ixb4

#13

I modified your account so you're a normal user now.

#14

Thanks for the logs...

Looking at those I don't see a deadlock which is good news.

However what appears to be happening is that in each case there are files being uploaded.

What that means is that

  • the calling app calls close()
  • this notifies the kernel which notifies rclone
  • rclone calls Flush()
  • rclone does not return until the file is uploaded
  • which means that the application is blocked while the file is uploading

This is what the patch was supposed to solve, but it obviously doesn't work

Perhaps what rclone needs is a more asynchronous upload, so instead of uploading it directly, rclone could stick it into a queue to be uploaded later.

Pros

  • rclone could limit the number of uploads
  • rclone could cancel uploads for files which were modified again
  • rclone could retry uploads
  • rclone could persist the queue so uploads would resume when rclone was restarted

Cons

  • it would make the upload a write-back operation, so the calling application would think the file was uploaded when it wasn't. Provided you only access the file through the mount everything will be consistent as rclone will use its local copy, but if you looked on the remote you'd see the file wasn't there yet.

Thoughts?

#15

I'm not the best person to answer that question but from my point of view that con is acceptable since the issue renders the mount virtually unusable in my case.

It's interesting I have 2 computers using an identical config file and mount commands, the one is just running the Plex server. The other that is having the problem is running Sonarr, Radarr, Jackett.

#16

Im not sure how the con would affect my use-case, as I am just trying to upload files to gdrive as a backup of my local media. I wont even be accessing files on the gdrive unless there is some catastrophic event that completely wipes both my local and local backup copies.

#17

It would be written pretty quickly. And the refresh would happen at the poll interval. So I'd say the con is worth the pros.

#18

anything is better than nothing is my approach

#19

It is more a matter of data safety for me... However having a queue would let rclone resume transfers which would be a big win I think...

I made this into an issue here: https://github.com/ncw/rclone/issues/3186

2 Likes
#20

WinFsp spawns a number of threads to service file system requests. This number if usually the number of cores you have on the machine with a minimum of 2. So if you have 1 or 2 cores you get 2 threads. If you have 8 cores you get 8 threads.

The FUSE layer of WinFsp has also an option ThreadCount which may be useful to increase the number of threads servicing file system requests. I am not certain if rclone exposes this option to the command line, but if it does try -o ThreadCount=16 to get 16 file system threads. This should be enough to service most workloads, but in extreme cases you can also try -o ThreadCount=64.