Empty folders and directory pemission with minio backend

What is the problem you are having with rclone?

  1. empty folders created on the mount do not get uploaded to minio backend, and distributed across to other rclone mount of the same backend

  2. using rclone in kubernetes environment with csi driver, where minio volumes are mounted by csi driver pods (running as root user) and in turn presented/mounted to kubernetes pods (running arbitrary restricted user) by kubelet, since only rclone mounting user(root) has access to the volume, "allow-others" mount option has to be used in order for application pods to use the volume without permission issue, this creates couple of problems/confusions

What is your rclone version (output from rclone version)

rclone v1.57.0

Which cloud storage system are you using? (eg Google Drive)

minio backed by local direct accessible block drives

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount :minio:pvc-34830760-7a05-4673-8410-f68ab8547d0a /tmp/mnt --daemon --no-check-certificate --no-modtime --cache-info-age=72h --cache-chunk-clean-interval=15m --dir-cache-time=5s --vfs-cache-mode=minimal --allow-non-empty=true --s3-provider=Minio --s3-access-key-id=<> --s3-secret-access-key=<> --s3-endpoint=svc.minio.svc.cluster.local:9000 --allow-others

The rclone config contents with secrets removed.

NA

A log from the command with the -vv flag

2021/12/20 18:45:32 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/12/20 18:45:49 DEBUG : /: Attr:
2021/12/20 18:45:49 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/12/20 18:45:49 DEBUG : /: Setattr: stat=Setattr [ID=0x12 Node=0x1 Uid=0 Gid=0 Pid=4318] mode=drwxrwxr-x handle=INVALID-0x0
2021/12/20 18:45:49 DEBUG : /: >Setattr: err=<nil>
2021/12/20 18:45:49 DEBUG : /: Attr:
2021/12/20 18:45:49 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>

i'm new to rclone and minio, having few questions:

for first issue, it seems to be related to the fact that folders in minio work as prefix to file objects, are there any solutions for apps to be able to use rclone minio mount effectively, without needing to create at least i.e. an empty file in order to get folder uploaded to backend and be seen by other mounts of the same backend, and also without needing to recreate the folder when last file in the folder gets removed?

for the second issue, since with "--allow-other", any user do have access to the mounted volumes in application pods, however, they are showing the users who mounted the volume in the driver pods, changing ownership/permissions after mount has no effect, basically disabled the meaning of file/dir attributes, is there a way that kubernetes fsgroup can be used here, so when minio volume is mounted to pods, the volume could be owned by that of the fsgroup?

third, is rclone mount with minio scale in terms of performance, any other tuning best practise can be applied with commands pasted here?

hello and welcome to the forum,

  • unlike most s3 tools including aws, rclone will not create an empty dir with s3 backend.
    https://rclone.org/commands/rclone_mount/#limitations

  • these flags are for the deprecated cache remote.
    --cache-info-age=72h
    --cache-chunk-clean-interval=15m

  • --allow-others is not a valid flag, i think you want
    --allow-other

fwiw, i always start with the simplest command, least number of flags and use as many defaults as possible, to establish a baseline performance number.

thanks, after mount, is updating folder permissions a valid operation with minio backend, base on this log?

2021/12/20 18:45:49 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/12/20 18:45:49 DEBUG : /: Setattr: stat=Setattr [ID=0x12 Node=0x1 Uid=0 Gid=0 Pid=4318] mode=drwxrwxr-x handle=INVALID-0x0

seemed to be having impression this is possible with other thread

Permissions are not a thing for cloud storage.

Are you trying to chmod something? That won't work.

1 Like

what would be the valid usage for --uid and --gid? mounted cloud storage as test:root, when switched over to test user, mount showing question marks:

sh-4.4# rclone mount :s3:pvc-34830760-7a05-4673-8410-f68ab8547d0a /tmp/mnt --daemon --no-check-certificate --no-modtime --dir-cache-time=5s --vfs-cache-mode=minimal --allow-non-empty=true --s3-provider=Minio --s3-access-key-id=1c5cac73058957 --s3-secret-access-key=a8f7915df261ba --s3-endpoint=https://svc.minio.svc.cluster.local:9000 --umask 000  --uid 1000 --gid=0

drwxrwxrwx. 1 test root     0 Dec 20 22:03 mnt

sh-4.4# su test
d?????????? ? ?    ?        ?            ? mnt

missing something here?

Fuse mounts require allow-other to let another user see it.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.