MacOS: Rclone takes down entire system

Suspected Bug
1

I need help reproducing an issue on MacOS.
The mount command I'm using is simply 'rclone mount remote: ~/rclone --vfs-cache-mode writes'

On occasion, rclone will take down the entire OS. It will pinwheel the Finder, which is usually solved by relaunching Finder. However, attempting to do so results in Finder quitting, but not re-starting.

Other open apps continue to function, but with extremely reduced performance. This includes Terminal, which is the parent process of rclone. Terminating rclone (or Quitting Terminal does not change anything; Finder does not re-launch, and reduced performance continues.) Activity Monitor shows a still-running rclone process, which can be terminated, but again, makes no difference.

The only way to restore the system to a usable state is a reboot.

The only consistent situation when this instability occurs is that I would say rclone is under "heavy" load. I am probably using 90% of my upload and download bandwidth (my EdgeRouter caps throughput at this level).

I don't see any rclone output that can be used for debugging (I rarely run with -vv and -v is not enough to show anything of concern). Nothing appears in system.log either.

My thinking is that the lack of log output means that it is actually a FUSE issue (or possibly even a Go issue) but I've never seen another application take down my entire system like this.

I'm curious if anyone else has ever experienced this.

I can not reliably reproduce it, so I do not have enough information to open a ticket yet. Any input would be welcome.

rclone v1.50.2

  • os/arch: darwin/amd64
  • go version: go1.13.4 and 1.13.7
1 Like
2

It is almost certainly something to do with FUSE. Rclone uses FUSE to provide a service to the kernel of your laptop which runs everything, so if stuff goes wrong with that interface then it can affect more than just rclone.

That said, it is pretty unusual. I'd check you have the latest OSXFUSE and also the latest rclone (1.51 was just released).

What is probably happening is that some system thread is trying to access the mount but is being blocked because there is an upload in progress.

Is rclone using a lot of CPU at this point?

If you wait for the upload to complete then does everything return to normal?

3

More than usual, but not “a lot”.
Low double digit percentage of a single core (4 core mobile i7).

No. Neither waiting nor killing processes works. This includes the download processes and rclone itself.

I’ll dig through open MacFUSE issues and see what I find. Maybe the key is in there.

It’s hard for me to produce useful debug output because although I’ve seen this happen about 4 times, I’ve not been able to deliberately reproduce it.

4

You really need to run it in debug and get a log file so we can look through it.

What backend are you using?

Are you leaving the terminal just open?

What's the size of the remote you are mounting?

Are you trying to index as well?

5

You really need to run it in debug and get a log file so we can look through it.

I’ll just start running in debug all the time and see what I can get. I have some time tonight, I’ll see if I can duplicate it.

Since @ncw thinks it is FUSE related, would debug of FUSE be relevant, or does that put it beyond the scope of this forum? I’ve never seen another FUSE mount behave this way, so in that respect it seems like an rclone issue—but an indirect one.

What backend are you using?

Google Drive & crypt

Are you leaving the terminal just open?

Yes. And I am terminating the rclone process in the terminal as well.
It continues to appear in Activity Manager, and I can then force quit it from Activity Manager (even though I have killed rclone via the Terminal already)

What's the size of the remote you are mounting?

~250GB

Are you trying to index as well?

No. Having the mount open (I.e. displaying its contents in the GUI) is not a requirement for this behavior to occur.

I think—I’m not sure, but I think—this only occurs when I am interacting with the same mount via a cloud server at the same time. In other words, I am uploading to this remote from my local machine, and another process on another machine is making simultaneous uploads to the same remote, using different API credentials. This doesn’t happen any time I use the remote this way, though, because I do this frequently.

I’m getting the impression nobody has heard of this? It’s just me?

6

It is probably worth checking the kernel log to see if macOS has throttled rclone - we've seen that before.

7

Are you referring to the feature where kernel_task will take up "imaginary" CPU in order to reduce usage by other processes? If so, that did not occur. I see no evidence of that in any logs, either.

Any idea how I can get debug output for FUSE?

8

I'm not really sure what macOS does and doesn't do (I don't have a mac), but I'm glad you checked the logs!

You can set --debug-fuse along with -vv to get FUSE debugging.

9

I'm familiar with the precise symptoms you describe (Finder beachballs, is unable to restart and every IO-related activity slows to a crawl) on my Mac which is currently running 10.15, though I've seen it with previous OS versions as well.

My experience with it is not directly connected to rclone however, but to external USB hard drives. I usually have around 10 USB hard drives connected at any given time, and when this situation has occured I've pretty much always able to resolve it by disconnecting the USB hub with the drives from the computer. All the frozen applications instantly come to life again, along with the execution of any IO-dependent actions I've previously attempted to take within them. I've found that the issue may also resolve itself over time even if I leave the cord in, though it's taken between a few minutes up to an hour for this to happen. It also seems to happen to me less often now, only a few times per year rather than per month as before - perhaps due to OS updates or because I'm using a different set of external drives now.

it's possible that the issues we're experiencing have different causes even if the symptoms are the same, but if you're also using external drives you may wish to try disconnecting them when the issue appears to see if it helps.

10

Intersting. Thanks for this.

While I don't have any external USB devices connected, I do have things using the internal bus that I could try disabling to see if this is, in fact, a USB related issue.

My guess is that it is actually just a disk related issue--whether that disk be a USB volume in your case, or a FUSE volume in my case.

Can you give any other information on what you are doing to cause these symptoms?

11

My experience with it is not directly connected to rclone however, but to external USB hard drives. I usually have around 10 USB hard drives connected at any given time, and when this situation has occured I've pretty much always able to resolve it by disconnecting the USB hub with the drives from the computer. All the frozen applications instantly come to life again, along with the execution of any IO-dependent actions I've previously attempted to take within them.

12

I see.

Are you using FileVault and/or APFS?

Reading this again, I'm realizing that what I haven't done is trued to unload the FUSE kext when this occurs. That's the only thing I can think of that would resemble "unplugging" the drive (or in this case the rclone mount)... I may give that a shot next time. If that works, I will submit a bug to Apple, as I believe it will indicate a problem with MacOS.

13

I am using a MacOS server aswell over here, and works great withouth any problems.
using rclone 1.51.0 latest macfuse, and using rclone browser to mount my drives..

So far no problems at all.
perhaps try mounting using rclone browser for example, and see if that helps..

Im using this version of rclone browser btw :

14

I've had almost the same issue couple of weeks ago.

15

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.