Unexpected behavior with Alias remote

rwv37 · December 13, 2020, 12:34am

What is the problem you are having with rclone?

An Alias remote pointing to a subdirectory of the root on an S3 remote (i.e. pointing to an S3 bucket) lists the contents of my local computer's current directory when I do an rclone lsf MyAlias:..

I would expect instead that rclone lsf MyAlias:.. should behave like rclone lsf MyS3:MyBucket/..`, i.e. it should list my buckets that are on the remote MyS3.

What is your rclone version (output from `rclone version`)

1.53.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64 bit

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

 rclone lsf A-S3-GDA-NVA:..

The rclone config contents with secrets removed.

[Dropbox]
type = dropbox
token = ...

[OneDrive]
type = onedrive
token = ...
drive_id = ...
drive_type = personal

[Wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = ...
secret_access_key = ...
endpoint = s3.wasabisys.com

[GSuite]
type = drive
client_id = ...
client_secret = ...
scope = drive
token = ...
root_folder_id = ...

[S3-GDA]
type = s3
provider = AWS
env_auth = false
access_key_id = ...
secret_access_key = ...
region = us-east-1
acl = private
server_side_encryption = AES256
storage_class = DEEP_ARCHIVE
bucket_acl = private

[A-S3-GDA-NVA]
type = alias
remote = S3-GDA:MyRedactedBucketName

A log from the command with the `-vv` flag

PS D:\temp> rclone lsf A-S3-GDA-NVA:.. -vv
2020/12/12 19:28:00 DEBUG : rclone: Version "v1.53.3" starting with parameters ["C:\\rclone\\rclone.exe" "lsf" "A-S3-GDA-NVA:.." "-vv"]
2020/12/12 19:28:00 DEBUG : Using config file from "C:\\Users\\bob\\.config\\rclone\\rclone.conf"
2020/12/12 19:28:00 DEBUG : Creating backend with remote "A-S3-GDA-NVA:.."
2020/12/12 19:28:00 DEBUG : Creating backend with remote "."
2020/12/12 19:28:00 DEBUG : fs cache: renaming cache item "." to be canonical "//?/D:/temp"
2020/12/12 19:28:00 DEBUG : fs cache: renaming cache item "A-S3-GDA-NVA:.." to be canonical "//?/D:/temp"
ThisIsAFileOnMyLocalComputerInTheTempDirectoryOfMyDDrive.txt
ThisIsOneToo.txt
2020/12/12 19:28:00 DEBUG : 2 go routines active
PS D:\temp>

Animosity022 · December 13, 2020, 2:24am

I believe an alias is making a new / so you can’t go back a path as that’s not really the use for an alias’s.

I thought it was to make a short cut to a root path further down.

rwv37 · December 13, 2020, 2:27am

That would be fine, but if so, it should give an error, not act upon an entirely unrelated filesystem.

Animosity022 · December 13, 2020, 2:42am

On Linux, it works as expected. Don’t have any Windows machines to test on.

rwv37 · December 13, 2020, 2:47am

I did it on a Windows box. I don't have a Linux box to try it on, but I just now tried it on a FreeBSD box, and got the same behavior as I did on Windows.

Animosity022 · December 13, 2020, 2:59am

Do you get the same thing on google drive as you are seeing in the bucket remotes?

rwv37 · December 13, 2020, 3:02am

Just tried GSuite now (on Windows). Same behavior. I had also previously tried it with Wasabi (again, on Windows), and got the same behavior there too

Animosity022 · December 13, 2020, 3:08am

Ah, I think I had a typo as I can reproduce it now.

[test]
type = alias
remote = GD:test

felix@gemini:/opt/rclone$ rclone ls test:.. -vv
2020/12/12 22:06:26 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "ls" "test:.." "-vv"]
2020/12/12 22:06:26 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/12/12 22:06:26 DEBUG : Creating backend with remote "test:.."
2020/12/12 22:06:26 DEBUG : Creating backend with remote "."
2020/12/12 22:06:26 DEBUG : fs cache: renaming cache item "." to be canonical "/opt/rclone"
2020/12/12 22:06:26 DEBUG : fs cache: renaming cache item "test:.." to be canonical "/opt/rclone"
     1727 rclone.conf
       60 scripts/excludes
      121 scripts/rclone_cleanup
     1106 scripts/upload_cloud
 10690616 logs/debug.log
1278011835 logs/old-rclone.log
     2558 logs/rclone.log
 29287016 logs/restic.log
 35195245 logs/upload.log

I think it's just giving you your current working directory that the command is executed on, which does seem strange.

I'd file a bug report and link to this post.

rwv37 · December 13, 2020, 3:23am

OK, thanks. I reported it here:

ncw · December 13, 2020, 9:19am

Thanks - that is quite weird behaviour! I'll take a look at the bug report shortly

ncw · December 13, 2020, 10:24am

I had a quick go at fixing this...

if myalias points to remote:dir

I guess we need to decide, do we want

myalias:.. == remote:
myalias:.. == remote:dir

I could make an argument for either... I did the most straight forward fix, but that turns out to be option 1 and I'm not sure that is what users expect of an alias.

Do we want the alias backend to cap the remote at the root passed in?

This same mechanism is used for crypt/cache/etc wrapping - I think we probably don't want crypt:.. to be able to look at the directory above.

@Animosity022 - do you have a view here?

rwv37 · December 13, 2020, 10:42am

I know you didn't ask me (and I don't blame you ), but for what it's worth, I would expect it to behave the same way I would expect something similar on a local filesystem to behave. For example, on FreeBSD (and I think on Linux too, but I don't have a Linux box here to verify it on), if you do an ln -s /etc etclink, then ls etclink, it will show you the contents of /etc. If you then do a ls etclink/.., it will show you the contents of / -- NOT the contents of /etc.

I don't really care one way or the other, but I'd be surprised to discover it didn't work like that.

ncw · December 13, 2020, 10:47am

I do appreciate your opinion!

Making things consistent is important and that analogy with symlinks is a good one.

I guess this could be a config option for alias also which stops viewing above the root.

I'm more concerned with the crypt wrapping though, as if I implement option 1) above the crypt:.. will start decrypting the directory above which maybe is OK or maybe not!

Animosity022 · December 13, 2020, 12:32pm

This is a tough one as I think there really isn't a use for doing ".." on a root of a directory as it just seems odd.

If ls /.. on the root, it stops and gives you the the current directory.
If you ls /test/.. with /test linking to something like /home/felix/test, it does follow it and you one back.

For my brain, I think of the remote as / in the equation and any alias I create as the same so I would think it should not show me anything before as that's why I created the alias.

So I would side on stopping viewing above the root.

ncw · December 15, 2020, 12:57pm

I've decided that this should cap out at the root so if you set an
alias of s3:bucket then using alias:.. would not allow looking
above the root of the alias. Listing alias:.. now lists s3:bucket
as you might expect if you did cd / then ls ...

I think allowing alias:.. to look above the intended root is a
potential security problem waiting to happen. Witness lots of web
server data exfiltration due to use of ...

Please give this a go!

v1.54.0-beta.4970.7493dd856.fix-4862-fs-parse on branch fix-4862-fs-parse (uploaded in 15-30 mins)

system · February 14, 2021, 8:57am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.