OSX Sierra/High Sierra - Can only copy large files to Cache Mount if nested

Suspected Bug
1

I think there is an issue with Rclone when copying non-nested, larger files into cached mounted drives in OSX High Sierra.

A) Setup:

  • Latest OSX High Sierra
  • Latest RClone beta:
  • Mounted G Suite Drive using cache (no encryption):
    –Remote to G-Suite Drive (“Gdrive”)
    –Mounted Cache to a directory on this this remote (“GdriveCache”)

So it is: Gdrive -> GdriveCache:media

  • Mount script:

rclone mount GdriveCache: /Users/xyz/rclone_data/media_mount_point --allow-other --dir-cache-time=160h --cache-chunk-size=10M --cache-info-age=168h --cache-workers=5 --buffer-size=500M --attr-timeout=1s --syslog --rc --cache-tmp-upload-path /Users/xyz/rclone_data/rclone_upload --cache-tmp-wait-time 1s --modify-window 1s

B) Test Results (all using Finder):

Test 1) Copy a small file (eg. 500 bytes) into mounted GdriveCache, copies fine.

Test 2) Copy a larger file into GdriveCache (eg. 300MB):

  • Starts copying to the folder defined in -cache-tmp-upload-path
  • But stops at about 128MB with the Finder “Error Code --36” error dialog
  • Leaves the partially copied 128MB file in the -cache-tmp-upload-path

Test 3) Copy the same larger file into GdriveCache, only this time copy its parent directory.

So we are copying /ParentFolder/larger_file.mkv to GdriveCache (rather than just larger_file.mkv as in Test 2).

This time, the copy works fine.

So in summary, can copy a large file to the mounted cache successfully if the large file is nested. Cannot copy if it is just the file on its own.

Small files with the same permissions copy fine, so it is not a permissions issue.

Finally, even with DEBUG logging, rclone reports no problems. No other error logs from the OS either.

Any help or thoughts greatly appreciated!

2

I see/saw similar problems on OSX Sierra, Copying files into Cache Mount on OS X although in my case the finder reported permission errors rather than error --36

But I never managed to find a solution, and have given up for the time being, as I did not have time to try different versions of osxfuse in case that made any difference. (I am not sure that it would make any difference, as I use osxfuse and sshfs and copying to those mounts works fine.)

Although it is interesting that it works for you when copying the parent folder. So I might give that a go and see what happens…

I concur. I tried putting the file I was having problems with into a folder and it seems to work fine

The only thing I did notice is that the folders remain in the directory I specified with --cache-tmp-upload-path (The files are removed, just not the folders)

3

This sounds similar possibly to the problem reported by @blim5001 in the other thread.

Do you see the same kernel messages?

4

Thank you both VERY MUCH for replying.

  1. Not the main issue, but to confirm: Like@blim5001, when the copy does work with nested larger files:
  • BAD: The folders do still remain in the --cache-tmp-upload-path folder after the child file has been sent to G Suite Drive
  • GOOD: The files within those folders are (correctly) deleted.
  1. I read the issue referenced above and:

2.A) Kernel Messages:
There are “exceeding limit of 150 wakeups per second over 300 seconds” kernel messages at times when rclone is busy but the full kernel message includes this key pair “Action taken: none” which suggests this is not the cause of the break.

2.B) All power saving options in OSX are switched off.

2.C) Like@blim5001, I am using the latest stable OSX Fuse 3.7.1

So, I think we can rule out:

  • Excessive kernel wakeups (inc. any caused by power settings)
  • Permission issues.

A thought:

Given that nested files do copy successfully, and small un-nested files copy successfully, I am leaning towards this being related to some sort of timeout issue. That is, perhaps the root cause is related to the fact that:

  • small files copy in <1 second
  • folders (but not their larger file constituents) also copy/create in <1 second
  • but un-nested individual larger files copy in >1 second.

Does this help? Any more ideas very welcome!

5

A little searching suggests that might be to do with ._ files. Here is one possible solution. That may be a red-herring, I don’t know!

6

Some more testing results:

  1. I have tried with every version of OSXFuse that supports High Sierra (ie. v3.6.2 up) and the results are the same - error code 36 when copying un-nested larger files.

  2. I have compared the "exceeding limit of 150 wakeups per second over 300 seconds” kernel messages and they are unrelated. Entirely different timestamps from earlier operations.

  3. I have tried higher settings for --modify-window, still get the error.

  4. I have tried all of the --ignore-checksum/time/etc and --no-checksum/etc, still get the error.

  5. If I use any mount flags (including, but not limited to, the -o and --fuse-flag flags) that are passed to Fuse, I see this in the logs:

Apr 29 17:27:16 Mac-mini rclone[15221]: -o/–option not supported with this FUSE backend
Apr 29 17:27:16 Mac-mini rclone[15221]: --fuse-flag not supported with this FUSE backend

  1. If I add the --write-back-cache flag I get a different error:
  • Do not get the Finder error code 36
  • Instead get the Finder permission error reported in the thread referenced above: “The operation can’t be completed because you don’t have permission to access some of the items”
  • No error logs created by rclone or OSX when this happens.
  • Small files and nested larger files still copy file with this flag (and they have the same permissions as the un-nested larger files that lead to the errors).

OK, I’m running out of ideas here team? Any thoughts?

7

Thanks, yes I should have mentioned that I ran dot_clean on the source and also set hidden files to visible. No hidden files at play unfortunately.

8

One more fact that might be relevant…or not…

The remote is always mounted with a trailing forward slash in it’s name.

So, when mounting using config of any of these…

  • Media:
  • Media:/
  • Media:sub_dir
  • Media:/sub_dir

… the Mount always appears as “Media/” (quotes added by myself).

The forward slash is a real pain when trying to quote the mount name in Terminal!

9

Is that the volume name?

rclone sets the volume name to be whatever you pass in as mount root, so Media: or Media:/ etc…

Not sure why it should always end up as Media/…

Note that you don’t need any of the / in your examples above - I normally leave them out, so just Media: or Media:sub_dir

10

Yes that is how the Volume name appears.

Yes, I saw your comment that the / is not needed in config, I was just trying different configs to be thorough.

I think I read somewhere that OS X converts forward slashes in filenames to colons. So maybe this is he same thing happening in reverse?

11

Mine does the same:

If I mount like this:
mount gcache: ~/_mounts/gDrive-rclone-cache

it shows in the finder as gcache/

12

I think this explains what is going on with the forward slash:

13

In the olden days pre macOS X, I think the delimiter for directories was : so that kind of makes sense…

It would be easy to make the volume name something different if you think that would help…

I removed the special charaters from the volume name as an experiment here:

https://beta.rclone.org/v1.41-005-g37bb9b95-osx-volname/ (uploaded in 15-30 mins)

can you give that a go? Should the volume name be settable by the user? I’d like it to default to something sensible though.

14

Thx I will test that experimental release.

Yes that would be helpful I think. For defaulting, taking the remote name stripped of special characters is probably sensible.

15

osxfuse does have a volname option:

volname=NAME, where NAME is a string

don’t know if it would be possible to use that.

I tried adding
–fuse-flag volname=gdrive
to my mount command, but it did not have any effect.

16

Yes I also trie using the volname flag but got the “-o/–option not supported with this FUSE backend” errors that I mentioned in an earler post on this thread.

17

Am I installing this correctly? The rclone file in the zip is called rclone.1 instead of rclone and it is much smaller than the rclone files normally released.

If I try to run it I get errors:

/usr/local/bin/rclone: line 1: .“t: command not found
/usr/local/bin/rclone: line 2: .”: command not found
/usr/local/bin/rclone: line 3: .": command not found
/usr/local/bin/rclone: line 4: .TH: command not found
/usr/local/bin/rclone: line 5: .hy: command not found
/usr/local/bin/rclone: line 6: .SH: command not found
/usr/local/bin/rclone: line 7: .PP: command not found
/usr/local/bin/rclone: line 8: syntax error near unexpected token (' /usr/local/bin/rclone: line 8:[IMAGE: Logo (https://rclone.org/img/rclone-120x120.png)] (https://rclone.org/)’

Me doing something dopey?

18

You want the ‘rclone’ file without the .1

19

There is an rclone.1 and an rclone.exe, no osx rclone file?

20

You want to download the Mac OS one.