Rclone unable to handle no-name "folder" on S3

[bbushvt (Bryan Bush)]

What is the problem you are having with rclone?

Unable to do anything on s3 bucket that contains a "folder" with no name
(ie. aws s3 ls s3://bcbush.com/test//ffmpeg.txt)

What is your rclone version (output from rclone version)

bbush@rlcone:~$ rclone --version
rclone v1.51.0

• os/arch: linux/amd64
• go version: go1.13.7
  bbush@rlcone:~$

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18.04 64bit

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsf aws_test:bcbush.com/test

The rclone config contents with secrets removed.

[aws_test]
type = s3
provider = AWS
env_auth = false
access_key_id = <KEY>
secret_access_key = <SECRET>
region = us-east-1
acl = private

A log from the command with the -vv flag

2020/06/07 04:00:34 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "-vv" "lsf" "aws_test:bcbush.com/test"]
2020/06/07 04:00:34 DEBUG : Using config file from "/home/bbush/.config/rclone/rclone.conf"
2020/06/07 04:00:34 ERROR : : Entry doesn't belong in directory "" (same as directory) - ignoring
2020/06/07 04:00:34 DEBUG : 5 go routines active
2020/06/07 04:00:34 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "-vv" "lsf" "aws_test:bcbush.com/test"]

[ncw (Nick Craig-Wood)]

There is an issue about this I'm sure.

Rclone now has the internal features to deal with this, it would be a question of making some encoding rules to help.

The listing routines would need to be fixed too.

So it could be fixed...

How much of a problem is it for you? Would you like to help fix?

[bbushvt (Bryan Bush)]

There is a lot of data I'm trying to access that has been put under "folders" with no name. So its become quite a big problem. I'm willing to help out, what can I do?

[ncw (Nick Craig-Wood)]

The s3 backend does implement filename encodings... This means that if we returned a name with a / in it then rclone would translate that into a unicode equivalent and carry on.

So what we want is rclone to treat the file name file//name as one directory called file and one file called /name. Rclone will then translate this name to [unicode equivalent /]name and away we go.

The listing routine is here: https://github.com/rclone/rclone/blob/c4110780bf64a53ae01d8b8d5e752aa11f80de5a/backend/s3/s3.go#L1384

That was my basic idea for fixing this. I can explain more later if you want, but I've got to dash off now!