Can Rclone remove empty dir in Minio now?

huanxp2 · September 2, 2024, 9:45am

What is the problem you are having with rclone?

I created an empty folder in Minio by following cmd:

./rclone mkdir s3:test/new/ --s3-directory-markers

then i want to delete it, but none of these commands work:

./rclone rmdirs s3:test/new/
./rclone rmdir s3:test/new/
./rclone purge s3:test/new/

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

os/version: openeuler 22.03 (64 bit)
os/kernel: 5.10.0-153.12.0.92.oe2203sp2.x86_64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.22.4
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Minio

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

./rclone rmdirs s3:test/new/
./rclone rmdir s3:test/new/
./rclone purge s3:test/new/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[s3]
type = s3
provider = Minio
access_key_id = XXX
secret_access_key = XXX
endpoint = http://hero-stg-minio.cnbita.com

A log from the command that you were trying to run with the `-vv` flag

# ./rclone rmdir s3:test/new/ -vv
2024/09/02 17:42:58 DEBUG : rclone: Version "v1.67.0" starting with parameters ["./rclone-67" "rmdir" "s3:test/new/" "-vv"]
2024/09/02 17:42:58 DEBUG : Creating backend with remote "s3:test/new/"
2024/09/02 17:42:58 DEBUG : Using config file from "/root/xxx/rclone.conf"
2024/09/02 17:42:58 DEBUG : Resolving service "s3" region "us-east-1"
2024/09/02 17:42:58 DEBUG : fs cache: renaming cache item "s3:test/new/" to be canonical "s3:test/new"
2024/09/02 17:42:58 INFO  : S3 bucket test path new: Removing directory
2024/09/02 17:42:58 DEBUG : 4 go routines active


# ./rclone purge s3:test/new/ -vv
2024/09/02 17:44:25 DEBUG : rclone: Version "v1.67.0" starting with parameters ["./rclone-67" "purge" "s3:test/new/" "-vv"]
2024/09/02 17:44:25 DEBUG : Creating backend with remote "s3:test/new/"
2024/09/02 17:44:25 DEBUG : Using config file from "/root/xxx/rclone.conf"
2024/09/02 17:44:25 DEBUG : Resolving service "s3" region "us-east-1"
2024/09/02 17:44:25 DEBUG : fs cache: renaming cache item "s3:test/new/" to be canonical "s3:test/new"
2024/09/02 17:44:25 DEBUG : S3 bucket test path new: bucket is versioned: false
2024/09/02 17:44:25 DEBUG : Waiting for deletions to finish
2024/09/02 17:44:25 DEBUG : 6 go routines active

kapitainsky · September 2, 2024, 9:53am

You have to always use this flag if you want to manage "empty directories" on S3 remote. The easiest is to add it to your configuration:

[s3]
type = s3
provider = Minio
access_key_id = XXX
secret_access_key = XXX
endpoint = http://hero-stg-minio.cnbita.com
directory_markers = true

Empty directories do not exist - actually no directories exist on S3. It is only convention to display objects using "directories" organisation.

huanxp2 · September 2, 2024, 11:58am

I have tried adding this flag to commands, but that's still not working

# ./rclone rmdir s3:test/new/ --s3-directory-markers -vv
2024/09/02 20:05:40 DEBUG : rclone: Version "v1.67.0" starting with parameters ["./rclone" "rmdir" "s3:test/new/" "--s3-directory-markers" "-vv"]
2024/09/02 20:05:40 DEBUG : Creating backend with remote "s3:test/new/"
2024/09/02 20:05:40 DEBUG : Using config file from "/root/xxx/rclone.conf"
2024/09/02 20:05:40 DEBUG : s3: detected overridden config - adding "{juk_h}" suffix to name
2024/09/02 20:05:40 DEBUG : Resolving service "s3" region "us-east-1"
2024/09/02 20:05:40 DEBUG : fs cache: renaming cache item "s3:test/new/" to be canonical "s3{juk_h}:test/new"
2024/09/02 20:05:40 INFO  : S3 bucket test path new: Removing directory
2024/09/02 20:05:40 DEBUG : 4 go routines active

after this, the empty object still exists

kapitainsky · September 2, 2024, 4:37pm

Indeed when I've tested it with other S3 provider rmdir does not work:) But purge does - of course with --s3-directory-markers flag only.

Can be because it is not really directory but empty object... There is no concept of directories in S3 storage and trying to gimmick their existence sometimes fails:)

huanxp2 · September 3, 2024, 1:49am

Thanks for your reply!
But purge doesn't work for my Minio either

kapitainsky · September 3, 2024, 4:20am

Post full debug log. FYI here is my test with iDrive (they use minio too I think):

$ rclone lsd iDrive:test-uk/
           0 2000-01-01 01:00:00        -1 test

$ rclone mkdir iDrive:test-uk/emptyDir --s3-directory-markers

$ rclone lsd iDrive:test-uk/
           0 2000-01-01 01:00:00        -1 emptyDir
           0 2000-01-01 01:00:00        -1 test

$ rclone purge iDrive:test-uk/emptyDir --s3-directory-markers -vv
2024/09/03 06:15:10 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "purge" "iDrive:test-uk/emptyDir" "-vv"]
2024/09/03 06:15:10 DEBUG : Creating backend with remote "iDrive:test-uk/emptyDir"
2024/09/03 06:15:10 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2024/09/03 06:15:10 DEBUG : Resolving service "s3" region "us-east-1"
2024/09/03 06:15:10 DEBUG : S3 bucket test-uk path emptyDir: bucket is versioned: false
2024/09/03 06:15:10 DEBUG : Waiting for deletions to finish
2024/09/03 06:15:11 DEBUG : "/" version false
2024/09/03 06:15:11 DEBUG : /: Deleting (id "")
2024/09/03 06:15:11 INFO  : /: Deleted
2024/09/03 06:15:11 DEBUG : 7 go routines active

$ rclone lsd iDrive:test-uk/
           0 2000-01-01 01:00:00        -1 test

might be also problem with minio... there are many configured strangely instances out there:)

huanxp2 · September 3, 2024, 6:18am

Here is my log:

# ./rclone lsd s3:test
           0 2000-01-01 08:00:00        -1 delete
           0 2000-01-01 08:00:00        -1 dstdir
           0 2000-01-01 08:00:00        -1 new
           0 2000-01-01 08:00:00        -1 newp

# ./rclone mkdir s3:test/todelete --s3-directory-markers

# ./rclone lsd s3:test
           0 2000-01-01 08:00:00        -1 delete
           0 2000-01-01 08:00:00        -1 dstdir
           0 2000-01-01 08:00:00        -1 new
           0 2000-01-01 08:00:00        -1 newp
           0 2000-01-01 08:00:00        -1 todelete

# ./rclone purge s3:test/todelete --s3-directory-markers -vv
2024/09/03 14:14:54 DEBUG : rclone: Version "v1.67.0" starting with parameters ["./rclone" "purge" "s3:test/todelete" "--s3-directory-markers" "-vv"]
2024/09/03 14:14:54 DEBUG : Creating backend with remote "s3:test/todelete"
2024/09/03 14:14:54 DEBUG : Using config file from "/root/xxx/rclone.conf"
2024/09/03 14:14:54 DEBUG : s3: detected overridden config - adding "{juk_h}" suffix to name
2024/09/03 14:14:54 DEBUG : Resolving service "s3" region "us-east-1"
2024/09/03 14:14:54 DEBUG : fs cache: renaming cache item "s3:test/todelete" to be canonical "s3{juk_h}:test/todelete"
2024/09/03 14:14:54 DEBUG : S3 bucket test path todelete: bucket is versioned: false
2024/09/03 14:14:54 DEBUG : Waiting for deletions to finish
2024/09/03 14:14:54 DEBUG : 6 go routines active

# ./rclone lsd s3:test
           0 2000-01-01 08:00:00        -1 delete
           0 2000-01-01 08:00:00        -1 dstdir
           0 2000-01-01 08:00:00        -1 new
           0 2000-01-01 08:00:00        -1 newp
           0 2000-01-01 08:00:00        -1 todelete

kapitainsky · September 3, 2024, 8:36am

Could you add --dump headers to the above command please? Maybe it will show enough info for somebody to see what the issue is.

kapitainsky · September 3, 2024, 8:46am

Also look at this:

Given that iDrive uses minio and rclone purge works then it is IMO minio server specific configuration. Maybe this link will give you some clues.

huanxp2 · September 3, 2024, 9:16am

Sure，thanks for your time!

# ./rclone purge s3:test/todelete --s3-directory-markers -vv --dump headers
2024/09/03 17:11:46 DEBUG : rclone: Version "v1.67.0" starting with parameters ["./rclone" "purge" "s3:test/todelete" "--s3-directory-markers" "-vv" "--dump" "headers"]
2024/09/03 17:11:46 DEBUG : Creating backend with remote "s3:test/todelete"
2024/09/03 17:11:46 DEBUG : Using config file from "/root/xxx/rclone.conf"
2024/09/03 17:11:46 DEBUG : s3: detected overridden config - adding "{juk_h}" suffix to name
2024/09/03 17:11:46 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/09/03 17:11:46 DEBUG : Resolving service "s3" region "us-east-1"
2024/09/03 17:11:46 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/09/03 17:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/03 17:11:46 DEBUG : HTTP REQUEST (req 0xc000f1fb00)
2024/09/03 17:11:46 DEBUG : HEAD /test/todelete HTTP/1.1
Host: hero-stg-minio.cnbita.com
User-Agent: rclone/v1.67.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240903T091146Z

2024/09/03 17:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/03 17:11:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/03 17:11:46 DEBUG : HTTP RESPONSE (req 0xc000f1fb00)
2024/09/03 17:11:46 DEBUG : HTTP/1.1 404 Not Found
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Date: Tue, 03 Sep 2024 09:11:46 GMT
Server: nginx/1.23.3
Vary: Origin
X-Amz-Request-Id: 17F1B0F6D0D67624
X-Xss-Protection: 1; mode=block
Content-Length: 0

2024/09/03 17:11:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/03 17:11:46 DEBUG : fs cache: renaming cache item "s3:test/todelete" to be canonical "s3{juk_h}:test/todelete"
2024/09/03 17:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/03 17:11:46 DEBUG : HTTP REQUEST (req 0xc000b4ea20)
2024/09/03 17:11:46 DEBUG : GET /test?versioning= HTTP/1.1
Host: hero-stg-minio.cnbita.com
User-Agent: rclone/v1.67.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240903T091146Z
Accept-Encoding: gzip

2024/09/03 17:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/03 17:11:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/03 17:11:46 DEBUG : HTTP RESPONSE (req 0xc000b4ea20)
2024/09/03 17:11:46 DEBUG : HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Tue, 03 Sep 2024 09:11:46 GMT
Server: nginx/1.23.3
Vary: Origin
X-Amz-Request-Id: 17F1B0F6D131C6F6
X-Xss-Protection: 1; mode=block

2024/09/03 17:11:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/03 17:11:46 DEBUG : S3 bucket test path todelete: bucket is versioned: false
2024/09/03 17:11:46 DEBUG : Waiting for deletions to finish
2024/09/03 17:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/03 17:11:46 DEBUG : HTTP REQUEST (req 0xc0006ed7a0)
2024/09/03 17:11:46 DEBUG : GET /test?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=todelete%2F HTTP/1.1
Host: hero-stg-minio.cnbita.com
User-Agent: rclone/v1.67.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240903T091146Z
Accept-Encoding: gzip

2024/09/03 17:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/03 17:11:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/03 17:11:46 DEBUG : HTTP RESPONSE (req 0xc0006ed7a0)
2024/09/03 17:11:46 DEBUG : HTTP/1.1 200 OK
Content-Length: 299
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Tue, 03 Sep 2024 09:11:46 GMT
Server: nginx/1.23.3
Vary: Origin
X-Amz-Request-Id: 17F1B0F6D167FC08
X-Xss-Protection: 1; mode=block

2024/09/03 17:11:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/03 17:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/03 17:11:46 DEBUG : HTTP REQUEST (req 0xc000526240)
2024/09/03 17:11:46 DEBUG : HEAD /test/todelete/ HTTP/1.1
Host: hero-stg-minio.cnbita.com
User-Agent: rclone/v1.67.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240903T091146Z

2024/09/03 17:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/03 17:11:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/03 17:11:46 DEBUG : HTTP RESPONSE (req 0xc000526240)
2024/09/03 17:11:46 DEBUG : HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/octet-stream
Date: Tue, 03 Sep 2024 09:11:46 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Last-Modified: Tue, 03 Sep 2024 06:14:19 GMT
Server: nginx/1.23.3
Vary: Origin
X-Amz-Request-Id: 17F1B0F6D1E81BBB
X-Xss-Protection: 1; mode=block
Content-Length: 0

2024/09/03 17:11:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/03 17:11:46 DEBUG : 6 go routines active

huanxp2 · September 3, 2024, 9:34am

I've just read this.
The same point is my Minio is a quite old release; but i can delete the empty dirs by Minio go sdk:

err := minio.Client.RemoveObject(context.Background(), bucket, objectKey, minio.RemoveObjectOptions{})

ncw · September 3, 2024, 11:53am

This may be a problem with rclone deleting the root directory.

If you rclone mount S3:test then I think you should be able to rmdir todelete.

huanxp2 · September 5, 2024, 12:19pm

Do you mean run:

# ./rclone mount s3:test/ test --daemon

and then:

# ./rclone rmdir test/todelete/

?
I've tried this, it works. But that seems a bit complicated in my case: using Minio sdk may be easier.

ncw · September 5, 2024, 12:30pm

This would probably work too

./rclone rmdirs s3:test --include /todelete/

I think this is actually a bug in the S3 backend which needs fixing

kapitainsky · September 5, 2024, 1:35pm

And given that I think all works with another minio installation it might be already fixed in newer minio versions.