Rclone listing fails on folder where a file with the same name exists

Suspected Bug
1

What is the problem you are having with rclone?

Listing commands are ambiguous, it is not always possible to know if you are listing the item or its content. When a file and a folder with the same name (eponym) coexists, it becomes impossible to differentiate if the eponym file is aside or within the folder unless one recursively list the parent folder (which can be very costly). Worse, the content of the directory cannot be listed at all, unless one recursively list the parent folder.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

  • os/version: darwin 14.6.1 (64 bit)
  • os/kernel: 23.6.0 (arm64)
  • os/type: darwin
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.23.1
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Azure (also tested with S3)

But the same problem should exist in any filesystem where folder do not really exists (e.g. only path exists, so an empty folder simply does not exist).

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The context is key here, otherwise commands are meaningless: in one cloud system, let us create this arborescence:

.
└── test/
    ├── README.md
    └── README.md/
        └── README.md
        └── some_other_file.txt

For clarity let's call the first README.md the 'aside' README and the one inside the folder the 'inside' README.
NB this hierarchy cannot be created with rclone itself, but will be easily created with native tools.

rclone ls azure:rnd/test/README.md/

The rclone config contents with secrets removed.

[azure]
type = azureblob
account = XXX
key = XXXXXXXXX

A log from the command with the -vv flag

The output is identical whatever the content of README.md folder, or if either README.md file is deleted (and there are no other files in the README.md folder).

Displaying a full (absolute) path with an option would solve the issue, yet --absolute just add a leading slash (which is clearly stated in the doc) whatever the case and is useless in this context (is it ever useful?).

2024/10/16 14:30:46 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "ls" "-vv" "azure:rnd/test/README.md/"]
2024/10/16 14:30:46 DEBUG : Creating backend with remote "azure:rnd/test/README.md/"
2024/10/16 14:30:46 DEBUG : Using config file from "/Users/delahondes/.config/rclone/rclone.conf"
2024/10/16 14:30:46 DEBUG : fs cache: adding new entry for parent of "azure:rnd/test/README.md/", "azure:rnd/test"
2024/10/16 14:30:46 DEBUG : README.md: Excluded
    18503 README.md
2024/10/16 14:30:46 DEBUG : 7 go routines active
2

Also, ideally, when a folder "exists" (e.g. contains some files in this context), the trailing slash should not be automatically removed by rclone.

3

welcome to the forum,

this is interesting but not sure i understand the problem?

you asked rclone to list the contents of folder azure:test/README.md/
and rclone listed the file README.md

what is wrong with that?

4

It is inaccurate. The real content is README.md AND some_other_file.txt.

5

Hi,

Have you tried this:

  • ls to list size and path of objects only
  • lsl to list modification time, size and path of objects only
  • lsd to list directories only
  • lsf to list objects and directories in easy to parse format
  • lsjson to list objects and directories in JSON format
6

In that very specific setup, none of these options will list what is inside the README.md folder (unless you recursively list the parent folder, e.g. rclone ls azure:rnd:test, as stated).

7

ok, i did notice that. i was a bit confused about your first post.

8

Yes sorry this is confusing as I should perhaps have stated two different issues:

First issue: the aside README.md is masking the README.md folder.

Second issue: when listing azure:rnd/test/README.md, the output is identical if the README.md file is inside, aside (or both) (only lsjson --stat will tell the difference).

When scripting, both issues collide and make the whole situation impossible to deal with, unless you recursively list the parent folder, which I would like to avoid as it could be costly.

9

Another detail on the second issue: I thought that two calls to rclone lsjson --stat would tell the difference: test if aside (lsjson --stat azure:rnd/test/README.md) yield a folder or file, same for the inside (lsjson --stat azure:rnd/test/README.md/README.md) and you're done, but imagine some (theoritical) situation like:
azure:rnd/test/README.md/README.md/README.md/.../README.md

There is no way you know when to stop. While this should not generally happens, it still can theoritically and I like when my code anticipates all situations, not half of them.

10

I think you are hitting a duplicate item as on most OSes, you cannot have a file name and directory name identical.

Some remotes, like Google Drive and S3s (apparently), you can have duplicate file/directory names.

11

i think the first thing is to create a script to reproduce and show the issue.
how about

dest=azure:rnd/test

rclone copyto ~/file.ext $dest/README.md/some_other_file.txt
rclone copyto ~/file.ext $dest/README.md/README.md
rclone copyto ~/file.ext $dest/README.md

rclone tree $dest/README.md/
rclone ls   $dest/README.md/

and this is the output


rclone tree azure:rnd/test/README.md/
/
├── README.md
└── some_other_file.txt

0 directories, 2 files

rclone ls azure:rnd/test/README.md/
        1 README.md
        1 some_other_file.txt
12

@Animosity022 Right, as stated, it only happens on cloud where folders do not really exists (I have tested on Azure and S3, not Google). It certainly cannot be reproduced locally.

13

On Google Drive (as I used that a lot), rclone would toss a duplicate error so you'd know something was up. Seems it doesn't on S3 though so that could be something to add.

Not sure what a fix would be as trying to copy/move that to anything non S3 would not work.

14

fwiw, a bit confusing, as you keep changing the paths
azure:rnd/test/
azure:test/

what is the real names? what is the name of the bucket?

15

@asdffdsa Thanks, the tree command looks like it should do the trick!

But it does not work for me, I do not understand:

% rclone tree azure:rnd/test/README.md/
/
└── README.md

0 directories, 1 files
16

rnd is the name of the bucket. the real one.
(test is a folder)

Oh I am so sorry, I see I have wrangled some path indeed. I try to fix that. But the actual command are fine (by which I mean the issue is real), only some paste issue.

The bucket is rnd, inside is a test folder, inside are both a file and a folder named README.md

17

ok, good, and bonus, rclone can do that.

i do not use azure, so i am testing on s3, using the same bucket name and same folder name.

18

Also there is a related issue on github (for the 2nd subissue, the confusion between inside and aside, not the 1st, masking issue):

19

Ok thanks, I look on S3 on my side.

20

I confirm, the tree command works on S3, not on Azure:

(env) delahondes@victivallis scitq % rclone tree s3:rnd/test/README.md/                      
/
├── README.md
└── some_other_file.txt

0 directories, 2 files