"delete --rmdirs" attempts to delete non-empty directories on Google Drive

What is the problem you are having with rclone?

The rclone delete --rmdirs command attempts to delete directories even when they are not empty.

Run the command 'rclone version' and share the full output of the command.

# rclone version
rclone v1.65.0-beta.7473.ad83ff769
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-1113-aws (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.5
- go/linking: dynamic
- go/tags: none

Also tried v1.64.2

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone delete "gdrive:/userdata/voronin-i/outputs"  --modify-window=1s --min-age=30d --fast-list  --rmdirs  --retries 1 -vv 

The rclone config contents with secrets removed.

[gdrive]
type = drive
root_folder_id = <id>
team_drive = <id>
service_account_file = <path/to/creds.json>

A log from the command with the -vv flag

# rclone lsf -R  "gdrive:/userdata/voronin-i/outputs" 
txt2img-images/
txt2img-images/2023-11-09/
txt2img-images/2023-11-08/
txt2img-images/2023-11-08/00001-3825242360.png
txt2img-images/2023-11-08/00000-1153125023.png
txt2img-images/2023-11-09/00005-2522136706.png
txt2img-images/2023-11-09/00004-3907040751.png
txt2img-images/2023-11-09/00003-1485638230.png
txt2img-images/2023-11-09/00002-3020745549.png
txt2img-images/2023-11-09/00001-4293910767.png
txt2img-images/2023-11-09/00000-3643922645.png
# rclone delete "gdrive:/userdata/voronin-i/outputs"  --modify-window=1s --min-age=30d --fast-list  --rmdirs  --retries 1 -vv 
2023/11/09 16:53:15 DEBUG : --min-age 1M to 2023-10-10 16:53:15.484630982 +0300 MSK m=-2591999.754725830
2023/11/09 16:53:15 DEBUG : rclone: Version "v1.65.0-beta.7473.ad83ff769" starting with parameters ["./rclone" "delete" "gdrive:/userdata/voronin-i/outputs" "--modify-window=1s" "--min-age=30d" "--fast-list" "--rmdirs" "--retries" "1" "-vv"]
2023/11/09 16:53:15 DEBUG : Creating backend with remote "gdrive:/userdata/voronin-i/outputs"
2023/11/09 16:53:15 DEBUG : Using config file from "/home/voronin-i/.config/rclone/rclone.conf"
2023/11/09 16:53:16 DEBUG : fs cache: renaming cache item "gdrive:/userdata/voronin-i/outputs" to be canonical "gdrive:userdata/voronin-i/outputs"
2023/11/09 16:53:16 DEBUG : Waiting for deletions to finish
2023/11/09 16:53:17 DEBUG : fs cache: switching user supplied name "gdrive:/userdata/voronin-i/outputs" for canonical name "gdrive:userdata/voronin-i/outputs"
2023/11/09 16:53:18 DEBUG : removing 2 level 2 directories
2023/11/09 16:53:18 INFO  : txt2img-images/2023-11-09: Removing directory
2023/11/09 16:53:18 INFO  : txt2img-images/2023-11-08: Removing directory
2023/11/09 16:53:18 DEBUG : txt2img-images/2023-11-08: Rmdir: contains file: "00001-3825242360.png"
2023/11/09 16:53:18 ERROR : txt2img-images/2023-11-08: Failed to rmdir: directory not empty
2023/11/09 16:53:18 DEBUG : txt2img-images/2023-11-09: Rmdir: contains file: "00005-2522136706.png"
2023/11/09 16:53:18 ERROR : txt2img-images/2023-11-09: Failed to rmdir: directory not empty
2023/11/09 16:53:18 DEBUG : removing 1 level 1 directories
2023/11/09 16:53:18 INFO  : txt2img-images: Removing directory
2023/11/09 16:53:18 DEBUG : txt2img-images: Rmdir: contains file: "2023-11-09"
2023/11/09 16:53:18 ERROR : txt2img-images: Failed to rmdir: directory not empty
2023/11/09 16:53:18 ERROR : Attempt 1/1 failed with 4 errors and: failed to remove 3 directories: last error: directory not empty
2023/11/09 16:53:18 DEBUG : 10 go routines active
2023/11/09 16:53:18 Failed to delete with 4 errors: last error was: failed to remove 3 directories: last error: directory not empty

welcome to the forum,

rclone will only delete file that are older than 30 days.
cannot be sure, but based 2023-11-08, the file is newer than 30 days.
so rclone cannot delete that file, as a result, the dir is not empty and rclone cannot delete the non-empty dir.

safest way to test filters is to list.
and we would like to see the modtime of the source files

rclone lsl gdrive:/userdata/voronin-i/outputs --min-age=30d

# rclone lsl gdrive:/userdata/voronin-i/outputs --min-age=30d
#

All the files are less than 30 days old.
I expect rclone to not try to delete the folders that have these files.

you told rclone try to remove the folders using --rmdir
so rclone tried to remove the dir but could not because there are files newer than 30 days that could not be deleted.
imho, that seems to be the correct result.

i could be wrong, but this is not a bug but perhaps the wording of the log is a bit confusing?

in any event, let's see what other forum members think about it?

Docs (rclone delete) say "If you supply the --rmdirs flag, it will remove all empty directories along with it". This sounds like it shouldn't try to remove any non-empty ones.

I agree with that.

i think this is just an issue of wording in the log,

  1. when using --rmdir, rclone has to perform a function call -> Removing directory
  2. rclone lists files in the folder and see there is a file that cannot be deleted -> Rmdir: contains file: "00001-3825242360.png"
  3. rclone logs that it cannot delete the folder -> Failed to rmdir: directory not empty

perhaps when using rclone delete --rmdir --min-age=30d.
rclone should understand that if there are files newer than 30d, not to ERROR, but instead NOTICE

in any event, i could be wrong, let's see what other forum members think about it?

I would not call it a bug.. It is logs cosmetics not a program functional problem IMO.

Log level of "Failed to rmdir: directory not empty" maybe should be NOTICE not ERROR? Problem is different people might have different view on it:)

This is a bug.

  1. The behavior causes rclone to always terminate with a non-zero status code, making it more difficult to identify real errors.
  2. Here is the code that is intended to verify whether a directory is empty before attempting to delete it: https://github.com/rclone/rclone/blob/ad83ff769b03b0731757431a596823822d12ae29/fs/operations/operations.go#L1242 .

Initial troubleshooting suggests that the code fails to detect any file objects within the given filesystem tree, identifying only directories. As a result, it incorrectly presumes that all directories are empty. This issue might be due to the --min-age filter, which renders all files newer than 30 days "invisible" to the Rmdirs() function.

Update: running rclone rmdirs gdrive:/userdata/voronin-i/outputs --min-age=30d fails with the same errors, rclone rmdirs gdrive:/userdata/voronin-i/outputs (w/o --min-age=30d filter) works as expected.

Fair point. It indeed should not return error code.

Ohh well. One more on the 157 open bugs list

If you consider it important for your use I suggest you give it a go.... because it is one of this small things which would be nice to fix but not important enough to must fix today...

I mean, I could think of this either way as if you filter something out and it can't delete, I'd want to know that, but I'm quite sure other people might not as it really depends on the use case.

I thought filtering happens before to save calls/APIs/etc so that's most likely why it's like this as it's 'hard' if I remember ncw saying something about this one before.

That is exactly correct.

I think rclone rmdirs probably shouldn't be using the filters for its directory traversal. That is easy to fix - change false to true in this line.

However what if the user specified --include "/deleteOnlyStuffFromHere/**" which would work just fine at the moment, but with the proposed change we'd remove other directories.

Hmm...

Thank you, Nick. You've pinpointed the exact issue I'm dealing with. In the real-world scenario I'm facing, I need to consider both types of filters: min-age and include /exclude . As a temporary solution, I've split the rclone delete --rmdirs --min-age 30d --include some/*/dir command into two separate commands:

# rclone delete --min-age 30d --include some/*/dir
# rclone rmdirs --include some/*/dir

I think that will work.

Perhaps rclone rmdirs should give a warning if it is used with filters as correct use is quite hard.

Anyway apart from the non zero exit code it should be harmless and we could remove that by checking for fs.ErrorDirectoryNotEmpty on return from the rmdir and ignoring it or logging it which is what @kapitainsky suggested above.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.