1.63 Beta fails to mount on macOS with on-the-fly crypt remote

Suspected Bug
1

What is the problem you are having with rclone?

Using the latest 1.63 beta, rclone crashes (and crashes FUSE which locks the system) when trying to mount an on-the-fly crypt remote.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0-beta.7072.4f8dab8bc
- os/version: darwin 13.4 (64 bit)
- os/kernel: 22.5.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.5
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

crypt and local

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I worked my way up to try to find what break it. Note that rclone1.63-beta is the latest beta. These passwords are just for testing

The following WORKS: (See config below)

rclone1.63-beta mount -vv mycrypt: ~/Desktop/mnt/

FAILS:

export RCLONE_CRYPT_REMOTE='/Users/**USER**/rclone_scratch/cryptfiles/'
export RCLONE_CRYPT_PASSWORD=rCd8GEoJZ3WQTk6aNRYop2gpwQ
export RCLONE_CRYPT_PASSWORD2=ue8qsRrJrBLWKr5q3CWn6ake9w

rclone1.63-beta mount -vv :crypt: ~/Desktop/mnt/

See the log below

The rclone config contents with secrets removed.

On the fly for the one I care about. But the working one pretty vanilla

[mycrypt]
type = crypt
remote = /Users/**USER**/rclone_scratch/cryptfiles/
password = rCd8GEoJZ3WQTk6aNRYop2gpwQ
password2 = ue8qsRrJrBLWKr5q3CWn6ake9w

A log from the command with the -vv flag

2023/06/16 11:11:48 DEBUG : Setting --config "/Users/**USER**/rclone_scratch/config.cfg" from environment variable RCLONE_CONFIG="/Users/**USER**/rclone_scratch/config.cfg"
2023/06/16 11:11:48 DEBUG : Setting default for crypt-remote="/Users/**USER**/rclone_scratch/cryptfiles/" from environment variable RCLONE_CRYPT_REMOTE
2023/06/16 11:11:48 DEBUG : Setting default for crypt-password="rCd8GEoJZ3WQTk6aNRYop2gpwQ" from environment variable RCLONE_CRYPT_PASSWORD
2023/06/16 11:11:48 DEBUG : Setting default for crypt-password2="ue8qsRrJrBLWKr5q3CWn6ake9w" from environment variable RCLONE_CRYPT_PASSWORD2
2023/06/16 11:11:48 DEBUG : rclone: Version "v1.63.0-beta.7072.4f8dab8bc" starting with parameters ["rclone1.63-beta" "mount" "-vv" ":crypt:" "/Users/**USER**/Desktop/mnt/"]
2023/06/16 11:11:48 DEBUG : Creating backend with remote ":crypt:"
2023/06/16 11:11:48 DEBUG : Using config file from "/Users/**USER**/rclone_scratch/config.cfg"
2023/06/16 11:11:48 DEBUG : Setting crypt_remote="/Users/**USER**/rclone_scratch/cryptfiles/" from environment variable RCLONE_CRYPT_REMOTE
2023/06/16 11:11:48 DEBUG : Setting crypt_password="rCd8GEoJZ3WQTk6aNRYop2gpwQ" from environment variable RCLONE_CRYPT_PASSWORD
2023/06/16 11:11:48 DEBUG : Setting crypt_password2="ue8qsRrJrBLWKr5q3CWn6ake9w" from environment variable RCLONE_CRYPT_PASSWORD2
2023/06/16 11:11:48 DEBUG : :crypt: detected overridden config - adding "{vOUl7}" suffix to name
2023/06/16 11:11:48 DEBUG : Setting crypt_remote="/Users/**USER**/rclone_scratch/cryptfiles/" from environment variable RCLONE_CRYPT_REMOTE
2023/06/16 11:11:48 DEBUG : Setting crypt_password="rCd8GEoJZ3WQTk6aNRYop2gpwQ" from environment variable RCLONE_CRYPT_PASSWORD
2023/06/16 11:11:48 DEBUG : Setting crypt_password2="ue8qsRrJrBLWKr5q3CWn6ake9w" from environment variable RCLONE_CRYPT_PASSWORD2
2023/06/16 11:11:48 DEBUG : Creating backend with remote "/Users/**USER**/rclone_scratch/cryptfiles/"
2023/06/16 11:11:48 DEBUG : fs cache: renaming cache item "/Users/**USER**/rclone_scratch/cryptfiles/" to be canonical "/Users/**USER**/rclone_scratch/cryptfiles"
2023/06/16 11:11:48 DEBUG : fs cache: renaming cache item ":crypt:" to be canonical ":crypt,password='rCd8GEoJZ3WQTk6aNRYop2gpwQ',password2='ue8qsRrJrBLWKr5q3CWn6ake9w',remote='/Users/**USER**/rclone_scratch/cryptfiles/':"
2023/06/16 11:11:48 INFO  : Encrypted drive ':crypt{vOUl7}:': poll-interval is not supported by this remote
2023/06/16 11:11:48 DEBUG : Mounting on "/Users/**USER**/Desktop/mnt/" ("crypt,password='rCd8GEoJZ3WQTk6aNRYop2gpwQ',password2='ue8qsRrJrBLWKr5q3CWn6ake9w',remote=' Users **USER** rclone_scratch cryptfiles '")
2023/06/16 11:11:48 DEBUG : Adding "-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC" for macOS
2023/06/16 11:11:48 DEBUG : Encrypted drive ':crypt{vOUl7}:': Mounting with options: ["-o" "attr_timeout=1" "-o" "fsname=:crypt,password='rCd8GEoJZ3WQTk6aNRYop2gpwQ',password2='ue8qsRrJrBLWKr5q3CWn6ake9w',remote='/Users/**USER**/rclone_scratch/cryptfiles/':" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "atomic_o_trunc" "-o" "daemon_timeout=600" "-o" "volname=crypt,password='rCd8GEoJZ3WQTk6aNRYop2gpwQ',password2='ue8qsRrJrBLWKr5q3CWn6ake9w',remote=' Users **USER** rclone_scratch cryptfiles '" "-o" "noappledouble" "-o" "modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC"]
fuse: unknown option `password='rCd8GEoJZ3WQTk6aNRYop2gpwQ''
2023/06/16 11:11:49 ERROR : Encrypted drive ':crypt{vOUl7}:': Mount failed
2023/06/16 11:11:49 Fatal error: failed to mount FUSE fs: mount stopped before calling Init: mount failed

My Guess

There is something accidentally getting passed to FUSE with rclone's formulation of connection strings. The line:

fuse: unknown option `password='rCd8GEoJZ3WQTk6aNRYop2gpwQ''

is fishy

2

just before I start looking into it - did it work before?

3

It works fine on

rclone v1.62.2
- os/version: darwin 13.4 (64 bit)
- os/kernel: 22.5.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.2
- go/linking: dynamic
- go/tags: cmount
4

I can reproduce your issue as well.

Clearly regression problem - maybe related to this change in used go-fuse:

v1.62.2:
github.com/hanwen/go-fuse/v2 v2.2.0

v1.63.0-beta:
github.com/hanwen/go-fuse/v2 v2.2.1-0.20230410213758-80c1c8221982

Could you create an issue on github?

5

Done

1 Like
6

Could one or the other of you bisect that? If you need instructions on how to use git bisect I can provide them. Thank you

7

yes please - also how to build rclone with mount on macOS

8

Build rclone with mount for macOS with

go build -tags cmount

You'll need the fuse c headers first which I install like this on the ci but you may have them already

brew install --cask macfuse

I think that is everything but if it doesn't build it will be because the c compiler can't find the fuse headers.

Here is a high level overview of the bisect

git checkout master
git bisect start 
git bisect bad
git checkout v1.62.0 # not v1.62.2 as that is on a branch 
git bisect good

# now build rclone and test it
go build -tags cmount
./rclone mount ....

# if it worked
git bisect good

# if it failed
git bisect bad

# git will choose another commit
# go back to the go build step

# eventually git will stop and print the offending commit

# at the end
git bisect reset

You might want to look at a git bisect tutorial if you want to understand more.

All of that from memory so hopefully not too many mistakes!

1 Like
9 
git checkout v1.62.0
error: pathspec 'v1.62.0' did not match any file(s) known to git

I am doing this in my fork in case it makes any diff.

I indeed tried it before with v1.62.2 - so it is progress to know what to do

10

Try something like

git fetch --tags

To get the tags too

11

kk. will carry on tomorrow. always nice to learn something new

1 Like
12

Here you are:

3567a47258c7c29aac3870add592975564122f22 is the first bad commit
commit 3567a47258c7c29aac3870add592975564122f22
Author: Nick Craig-Wood <nick@craig-wood.com>
Date:   Fri Apr 28 11:58:49 2023 +0100

    fs: make ConfigString properly reverse suffixed file systems
    
    Before this change we renamed file systems with overridden config with
    {suffix}.
    
    However this meant that ConfigString produced a value which wouldn't
    re-create the file system.
    
    This uses an internal hash to keep note of what config goes which
    which {suffix} in order to remake the config properly.

 fs/newfs.go      | 35 ++++++++++++++++++++++++++++++-----
 fs/newfs_test.go | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 73 insertions(+), 5 deletions(-)
 create mode 100644 fs/newfs_test.go

and makes perfect sense as this is error from DEBUG rclone mount - looks like all remote data is passed to fuse creating havoc:

2023/06/17 09:15:33 DEBUG : Encrypted drive 'cryptx{bd99F}:': Mounting with options: ["-o" "attr_timeout=1" "-o" "fsname=cryptx,password='Asda89das89d7as98d7as897da',remote='/Users/kptsky/Downloads/data':" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "atomic_o_trunc" "-o" "daemon_timeout=600" "-o" "volname=cryptx,password='Asda89das89d7as98d7as897da',remote=' Users kptsky Downloads data'" "-o" "noappledouble" "-o" "modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC"]

vs working:

2023/06/17 08:59:25 DEBUG : Encrypted drive 'cryptx{bd99F}:': Mounting with options: ["-o" "attr_timeout=1" "-o" "fsname=cryptx{bd99F}:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "atomic_o_trunc" "-o" "daemon_timeout=600" "-o" "volname=cryptx{bd99F}" "-o" "noappledouble" "-o" "modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC"]

I could not reproduce the issue on Linux so it seems to be macfuse specific however on Linux canonical name also looks weird - maybe is just never used:

v1.63.0:
2023/06/17 09:44:02 DEBUG : fs cache: renaming cache item "cryptx:" to be canonical "cryptx,password=''Asda89das89d7as98d7as897da',remote='/home/kptsky/Downloads/data':"

v.1.62.2
2023/06/17 10:17:56 DEBUG : fs cache: renaming cache item "cryptx:" to be canonical "cryptx{GqhOm}:"

1 Like
13

I have posted a potential fix to the issue.

Please can you give it a go and reply on the issue - thanks.

14

This is an unexpected consequence. It is probably OK but does look weird as you said.

15

and reveals password which is not nice thing to do in logfiles

16

Now it properly reads env vars with crypt details - only thing to complain is password in clear text in log

rclone: Version "v1.63.0-DEV" starting with parameters ["./rclone/rclone" "mount" "cryptx:" "/Users/kptsky/Downloads
/mount" "-vv" "--allow-non-empty"]
2023/06/20 17:58:38 DEBUG : Creating backend with remote "cryptx:"
2023/06/20 17:58:38 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2023/06/20 17:58:38 DEBUG : Setting type="crypt" for "cryptx" from environment variable RCLONE_CONFIG_CRYPTX_TYPE
2023/06/20 17:58:38 DEBUG : Setting remote="/Users/kptsky/Downloads/data" for "cryptx" from environment variable RCLONE_CONFIG_CRYPTX_REMOTE
2023/06/20 17:58:38 DEBUG : Setting password="akldsj4fgsSDFAqwa3sadAS" for "cryptx" from environment variable RCLONE_CONFIG_CRYPTX_PASSWORD
2023/06/20 17:58:38 DEBUG : cryptx: detected overridden config - adding "{bd99F}" suffix to name
2023/06/20 17:58:38 DEBUG : Setting remote="/Users/kptsky/Downloads/data" for "cryptx" from environment variable RCLONE_CONFIG_CRYPTX_REMOTE
2023/06/20 17:58:38 DEBUG : Setting password="akldsj4fgsSDFAqwa3sadAS" for "cryptx" from environment variable RCLONE_CONFIG_CRYPTX_PASSWORD
2023/06/20 17:58:38 DEBUG : Creating backend with remote "/Users/kptsky/Downloads/data"
2023/06/20 17:58:38 DEBUG : fs cache: renaming cache item "cryptx:" to be canonical "cryptx,password='akldsj4fgsSDFAqwa3sadAS',remote='/Users/kptsky/Downloads/data':"
2023/06/20 17:58:38 INFO  : Encrypted drive 'cryptx{bd99F}:': poll-interval is not supported by this remote
2023/06/20 17:58:38 DEBUG : Mounting on "/Users/kptsky/Downloads/mount" ("cryptx{bd99F}")
2023/06/20 17:58:38 DEBUG : Adding "-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC" for macOS
2023/06/20 17:58:38 DEBUG : Encrypted drive 'cryptx{bd99F}:': Mounting with options: ["-o" "attr_timeout=1" "-o" "fsname=cryptx{bd99F}:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "atomic_o_trunc" "-o" "daemon_timeout=600" "-o" "volname=cryptx{bd99F}" "-o" "noappledouble" "-o" "modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC"]
17

I haven't been able to test yet since I am on the wrong machine but I think that has always been the case with on-the-fly.

I'll test out the beta when I get home (or sooner if I can reproduce on my work mac (work uses FUSE-T, home uses MacFUSE. I don't know yet).

18

I find FUSE-T much more stable.

MacFUSE became disaster - easily bringing all computer down.

19

Yes but the issue with ModTime makes it a near non-starter on some systems.

20

Yeah I was thinking about it but so far it does not bite me. It only happens when app tries to set access time. E.g. Quick View does not do it - at least not anymore.