On s3, rclone should create persistent empty folders

What is the problem you are having with rclone?

rclone will create an empty folder and then rclone will delete that folder after the dir cache expires.
i do not understand why rclone would create a empty folder and then arbitrarily delete it a few minutes later.

if i create a folder using cloudberry explorer, that folder never disappears; rclone can see it and copy files to it.

so it would seem rclone should be able to create a persistent empty folder.

thanks

What is your rclone version (output from rclone version)

v1.51.0-264-g2696e369-fix-4104
winfsp 1.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

win10.1909.64

Which cloud storage system are you using? (eg Google Drive)

s3 wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

command does not matter, but in this case i am using a mount.

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

not sure a log is needed, as this is a general discussion of well known issues.

My understanding is directories don't really exist on bucket based remotes. There's nothing to create. The reason it goes away when the cache expires is it's simply held in a vfs structure.

What cloudberry does is create a zero length file called "directory/" and by convention tools (including rclone) ignore it.

Rclone doesn't ever create these directory markers at the moment. If you are doing a sync for example you end up doing a lot of extra transactions to create the directory markers you don't really need.

The only place directory markers are needed is to mark an empty directory.

Maybe rclone should attempt to do something clever like for every "Mkdir" create a directory marker in 10s if there weren't any other files written to the directory.

it seems strange, counter intuitive, that rclone has problems.

given that

  1. wasabi and aws have no problem with persistent empty folders

  2. using both the website and the clouberry explorer.

  3. rclone can lsd those persistent empty folders

rclone lsd wasabi: 
0 2020-05-10 10:41:14        -1 empty-folder-created-by-wasabi-website
0 2020-05-10 10:41:14        -1 empty.folder.created.by.cloudberry.explorer
rclone lsd aws:
0 2020-05-10 10:53:00        -1 empty.folder.created.by.aws.website
0 2020-05-10 10:53:00        -1 empty.folder.created.by.cloudberry.explorer
  1. rclone mount can copy files to those persistent empty folders.

  2. rclone can ls those folders.

rclone ls wasabi: 
1 empty-folder-created-by-wasabi-website/02.txt
1 empty.folder.created.by.cloudberry.explorer/01.txt
rclone ls aws:
1 empty.folder.created.by.aws.website/03.txt
1 empty.folder.created.by.cloudberry.explorer/04.txt

i understand the concern about extra transaction but
if a user creates a folder, rclone should behave as the user expects.

this is kind of the same issue as vfs/refresh that on s3 compatible storage does not get the modtime.
that rclone is making decisions to save transactions.

This should maybe be an option --s3-directory-markers ? Then the user can choose

that would be useful,
if you create a beta, let me know...

thanks

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.