I don’t have a mac and I’ve never managed to get any of the VMs to work so I rely on helpful mac users to test stuff - thank you 
Your welcome, (fwiw, I have just managed to setup an osx vm using virtualbox, so as not to go off topic, if I can help drop me a PM)
Hi all - apologies for the radio silence - real life took over and I have only just now been able to get back to rclone!
I have tested the binary referenced in the github issue and am very happy to confirm that, like @blim5001 on Sierra (right?), this binary works for me on High Sierra:
-
--volnameallows changing the volume name in the mount command - When mounting without both the
--noappledoubleand--noapplexattrflags there are no error code 36 exceptions reported by Finder any more.
Thx @ncw!
Great - thanks for testing!
I’ve merged this to master now so these changes will be in the latest beta and in 1.42 when it is released.
Ok so I think there is something else environmental going on here.
For both this latest beta and the binary that worked earlier in “Test 1: Using the mount (not cmount) build with NoAppleXattr disabled” above, I am now seeing:
- Small text files copy successfully whether or not they have any xattr’s (expected behaviour)
- Larger files (eg. 300mb) with the same permissions fail to copy with error code 36 again, also whether or not they have any xattr’s.
So this time it seems to have nothing to do with the extended attributes and more to do with the file size.
If it was just the new beta binary that had this issue then I would point the finger there, but given that the previously working test binary now errors out, I am thinking there must be something else environmental contributing to this issue. I will generate logs and keep trying to identify what has changed.
Sorry to be opening this up again, I was sure we had crushed it!
The only thing I can think of that has changed is that there was an OS X High Sierra update recently.
My suspicion is that this is somehow related to the --attr-timeout flag.
What happens is:
- Mounting Gdrive->GdriveCache->GdriveCacheCrypt. Results are the same without encryption.
- Latest beta.
- Small files copied from the OS hard disk appear in the --cache-tmp-upload-path folder and are subsequently sent to Gdrive and then deleted from the --cache-tmp-upload-path folder (expected behaviour)
- Large files start to copy to the --cache-tmp-upload-path folder but at about the 25MB mark Finder reports the error code -36 exception and then the copy is aborted. However the partially copied, truncated 25MB file is left behind in the --cache-tmp-upload-path folder (but removed from the Volume).
- Whether a file has xattr’s make no difference this time (which is expected as the latest beta resolves this).
This appears to be similar to this quote discussing the -attr-timeout flag in the docs:
You may see corruption if the remote file changes length during this window. It will show up as either a truncated file or a file with garbage on the end.
Mount command used is:
rclone mount Media: ~/rclone_data/media_mnt_point --allow-other --dir-cache-time=160h --cache-chunk-size=10M --cache-info-age=168h --cache-workers=5 --buffer-size=500M --attr-timeout=1s --cache-tmp-upload-path ~/rclone_data/rclone_upload --cache-tmp-wait-time 60m --umask 002 --drive-use-trash=false
I have tested --attr-timeout values of 0s and 60s, no difference.
I have -vv --debug-fuse output as well but it is too verbose to post here I think. I couldn’t see any obvious errors being reported by rclone but the full debug logs are quite challenging for me to understand.
I just tried copying files on my main mac running Sierra (10.12.6) and copying files on a HighSierra (10.13.4) VM
Both worked as expected. (The VM is currently still uploading the file)
I see there is a security update, so when the upload has finished, I will make a snapshot of my VM and will test copying files again after the update…
Edit: Seems a bit challenging to get the update to work when running in a VM!
Edit 2: Ok, finally managed to install the update, although not exactly sure how… but software update shows it as installed (but not sure that I am convinced it actually installed). Tried copying a file and again it worked as expected.
Thx @blim5001. Just to clarify:
- What mount flags did you use pls?
- Which rclone version did you use?
- Did you include large files in the copy?
thx.
Mount Command:
rclone mount gcache: ~/_mounts/gDrive-rclone-cache \
--allow-other \
--dir-cache-time=8h \
--cache-chunk-size=5M \
--cache-info-age=10h \
--cache-workers=8 \
--buffer-size 0M \
--attr-timeout=1s \
--volname=myDrive \
--rc \
--cache-writes \
--vfs-cache-mode writes \
--cache-tmp-upload-path ~/_mounts/_rclone_upload \
--cache-tmp-wait-time 5m \
--cache-db-path /Users/blim/.cache/rcloneCache \
--cache-chunk-path=/Users/blim/.cache/rcloneChunks \
-vv
rclone v1.41-014-gb15b4078-fix-2287-osx-mountβ
- os/arch: darwin/amd64
- go version: go1.10.1
Same version on my iMac and my VM
Copied files that were about 500MB
Edit: But I am going to try and build a High Sierra VM with VMWare Fusion to see if I can get the Security Update to install more reliably, as I mentioned before I am not convinced it worked properly in the VirtualBox VM
No worries… Software is a continuous process of improvement!
Did you check the kernel logs for any errors?
Are the files changing during the --attr-timeout windows? If not then it probably isn’t to do with the attr timeout.
Can you see if a plain mount works (without cache & crypt)? Try with and without --vfs-cache-mode writes?
If you want to upload them somewhere I can take a look. Though if you can get just the plain mount to fail, logs from that would be more useful.
Ok, so I built a HighSierra VM with VMWare Fusion, and that seemed much happier applying the security update.
So using the same version of rclone as in the previous post.
The results of that were the same as before.
I copied a 3GB file and It worked without issue.
OK I found a hint in the kernel logs. Appears to be related to OSX’s sandboxing (so I increasingly suspect that the latest osx Security Update did change something on my machine).
When the Finder copy fails on a large file I see the following 4 errors in rapid succession:
Sandbox: Finder(291) System Policy: deny(6) file-read-xattr
Sandbox: fseventsd(54) System Policy: deny(6) file-read-metadata
Sandbox: DesktopServicesH(15533) System Policy: deny(6) file-write-data
Sandbox: DesktopServicesH(15533) System Policy: deny(6) file-write-xattr
All of these errors reference the destination mount point folder which in my case is:
/Users/username/rclone_data/media_mount_point/video_file.avi
The full detail of each exception is too verbose to post here, but I can make it available online if needed.
So this points to permissions and xattr’s playing a role again. But, confusingly:
- Get the error whether the source large file has xattr’s or not
- Small files with the same permissions, from the same source folder, do not present these errors.
Any thoughts?
Hang on, I was filtering the log on “rclone” and missed these important 4 lines that occur just before the errors in my post above:
osxfuse: filehandle_get: failed for video_file.avi (type=0, err=2, caller=<private>)
osxfuse: filehandle_get failed in open (type=0, err=2)
Sandbox: Finder(291) System Policy: deny(6) file-read-xattr
Sandbox: fseventsd(54) System Policy: deny(6) file-read-metadata
Sandbox: DesktopServicesH(15533) System Policy: deny(6) file-write-data
Sandbox: DesktopServicesH(15533) System Policy: deny(6) file-write-xattr
Does that help?
That is certainly suggestive that it is the System Integrity Protection kicking in.
It is possible to disable that temporarily so it might be interesting if you did that then tried rclone. I wouldn’t want to disable it permanently though.
Another thing you could try is seeing whether you get the same problem with other OSXFUSE based filinsystems, eg sshfs or encfs…
It might also be worth looking through or posting on the osxfuse mailing list: https://groups.google.com/forum/#!forum/osxfuse-group - I’m certainly well out of my depth here!
@jrock, which log file are you finding these in, I will try copying some files later when I get home and see if I see these errors as well
Tried some copying a 1 GB file and did not see messages related to the file i was copying.
There were quite a lot of
System policy: deny
errors in the console, but not relating to the file I copied.
What I just tried was using the mount settings that @jrock posted and got the error
So it seem to be something in the settings that is causing the problem, so now to try and work out what…
And i just tried mounting with my settings and copying the same file worked (just to make sure it was not a specific file doing it)
So far I tried removing these 2 from jrock’s config:
–umask 002 --drive-use-trash=false
same result, copy failed, so i guess its not those 2
He thx @blim5001, that gives me a path to follow. I will try eliminating settings one at a time and revert back. Do you mind posting your correctly working settings?