Mysterious Ceph S3 bucket mount failure

Help and Support
1

What is the problem you are having with rclone?

I am trying to mount a Ceph RadosGW bucket using the S3 api and it fails under linux. However I am able to mount a specific prefix (or folder) that lies inside the bucket. I am also able to mount the full project that also includes the problematic bucket. I am also able to mount a similar bucket in the same project.

I am trying to get some hints as to why that specific bucket is not getting mounted but I am running out of ideas. I've looked carefully at the -vv logs and even some --dump logs, but did not find anything suspicious yet.

Run the command 'rclone version' and share the full output of the command.

$ rclone --version
rclone v1.71.0
- os/version: fedora 42 (64 bit)
- os/kernel: 6.16.12-200.fc42.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.6
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Ceph squid with RGW S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

mounting the MOH-Q bucket:

$ rclone  mount       c3g-data:MOH-Q/ ~/mnt/

then listing it:

$ls mnt/
ls: reading directory 'mnt/': Input/output error

But here are all the other tests that work

$ rclone    lsd     c3g-data:MOH-Q/
0 1999-12-31 19:00:00        -1 /
0 1999-12-31 19:00:00        -1 CM
0 1999-12-31 19:00:00        -1 CQ
0 1999-12-31 19:00:00        -1 CS
0 1999-12-31 19:00:00        -1 GC
0 1999-12-31 19:00:00        -1 HM
0 1999-12-31 19:00:00        -1 IQ
0 1999-12-31 19:00:00        -1 JG
0 1999-12-31 19:00:00        -1 MU
0 1999-12-31 19:00:00        -1 extras

I can mount a selected prefix:

$ rclone  mount       c3g-data:MOH-Q/MU ~/mnt/

and list

$ ls mnt/
MoHQ-MU-12  MoHQ-MU-33
MoHQ-MU-14  MoHQ-MU-34
MoHQ-MU-16  MoHQ-MU-35
MoHQ-MU-17  MoHQ-MU-36
MoHQ-MU-20  MoHQ-MU-44
MoHQ-MU-24  MoHQ-MU-8
MoHQ-MU-25

And the funkiest part, I can also mount the whole project.

$rclone  mount       c3g-data: ~/mnt/

and

$ ls mnt/MOH-Q/
CM  CS      GC  IQ  MOH-Q
CQ  extras  HM  JG  MU

I am also able to mount other buckets in the project that have the same bucket policy without problems.

Note that the bucket is quite big:

$ rclone  size       c3g-data:MOH-Q/
Total objects: 913.149k (913149)
Total size: 1.654 PiB (1862379856073980 Byte)

More than half of those objects are in the MU folder that mounts without any problems.

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[c3g-data]
type = s3
provider = Other
env_auth = false
access_key_id = XXX
secret_access_key = XXX
endpoint = https://objets.juno.calculquebec.ca
acl = private

A log from the command that you were trying to run with the -vv flag

That is where it get funny, the log are similar if for the failing mount and for the working example where I mount the whole project and list the bucket successfully:

$ rclone -vv mount       c3g-data:MOH-Q/ ~/mnt/
2025/10/31 16:13:09 DEBUG : rclone: Version "v1.71.0" starting with parameters ["rclone" "-vv" "mount" "c3g-data:MOH-Q/" "/home/poq/mnt/"]
2025/10/31 16:13:09 DEBUG : Creating backend with remote "c3g-data:MOH-Q/"
2025/10/31 16:13:09 DEBUG : Using config file from "/home/poq/.config/rclone/rclone.conf"
2025/10/31 16:13:09 DEBUG : fs cache: renaming cache item "c3g-data:MOH-Q/" to be canonical "c3g-data:MOH-Q"
2025/10/31 16:13:09 INFO  : S3 bucket MOH-Q: poll-interval is not supported by this remote
2025/10/31 16:13:09 DEBUG : S3 bucket MOH-Q: Mounting on "/home/poq/mnt/"
2025/10/31 16:13:09 DEBUG : Root: 
2025/10/31 16:13:09 DEBUG : >Root: node=/, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Lookup: name=".Trash"
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: Lookup: name="BDMV"
2025/10/31 16:13:09 DEBUG : /: Lookup: name=".xdg-volume-info"
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Lookup: name="autorun.inf"
2025/10/31 16:13:09 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2025/10/31 16:13:09 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2025/10/31 16:13:09 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2025/10/31 16:13:09 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Lookup: name=".Trash-1000"
2025/10/31 16:13:09 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Attr: 
2025/10/31 16:13:09 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:13:09 DEBUG : /: ReadDirAll: 
2025/10/31 16:13:09 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2025/10/31 16:13:09 DEBUG : /: Lookup: name="autorun.inf"
2025/10/31 16:13:09 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory

# then I run ls ~/mnt get the input/output error  described above and the mount command append this:


2025/10/31 16:14:53 DEBUG : /: Attr: 
2025/10/31 16:14:53 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2025/10/31 16:14:53 DEBUG : /: ReadDirAll: 
2025/10/31 16:14:53 DEBUG : /: >ReadDirAll: item=12, err=<nil>
2

perhaps that should be provider = Ceph

fwiw, best to test simple command such as rclone ls, rclone lsd before trying complex rclone mount

3

Just tested, same result with provider = Ceph. All the other simple rclone commands work normally (see rclone lsd c3g-data:MOH-Q/ above ).

Actually, the mount command was working in the past. The difference between last time that the mount worked is that the bucket grew form ~ 900 PiB to 1.6 PiB

4

Does Rclone looks at some metadata when mounting buckets? Maybe we've hit a limit... I am able to mount the bucket without a problem with other fuse base clients.