S3 object deletion times

I am using the S3 backend and listing object versions using --s3-versions. I am interested not only in when an object version was created but also in when it was deleted. I think of the deletion time as when the version becomes non-current, which corresponds to the creation time of a new version or a deletion marker.

Is there currently a way to get this information from rclone? Does this seem like a reasonably common want?

For me, the most convenient implementation would be a flag that sets the deletion time as the object version's modification date. I understand this could be confusing, as the modification date is already used both for rclone's modification date and object birth time (when using --use-server-modtime).

(A quick recap to make sure we understand each other)

The version where the file is deleted will be the last version with a full prefix, so if your file was called file2.txt then its last version might be


This is the date that file2.txt was created.

Amazon does provide the information when the file was deleted in the delete markers.



You could probably do what you want in this function in the backend and adjust the object somehow.

It might be better to set a bit of metadata on the object say a dtime for deleted time.

What do you want to do with this info? Make a report? Or something else? That might change my thinking on how to output this info.

It might be easier to make a backend command to output this info directly.

We understand each other :). On AWS, information on the time of deletion is indeed available in the delete marker with the same prefix, not in the object version metadata itself. If an object is overridden, the "deletion time" of the old object version version can be the creation of the new version.

Setting a bit of metadata (dtime) would indeed make a lot of sense and probably be a better option compared to setting modification times. It would be great though if it were possible to get dtime without also retrieving (rclone) modification times. Asking rclone to retrieve this metadata (e.g. by setting --metadata) slows down the list operation a lot (at least two orders of magnitude).

My use case is to analyze the contents of a bucket (distribution of file sizes, growth over time, etc). For this I export a list of object versions using rclone lsjson. I am connecting to buckets residing on MinIO; on AWS I would probably use Amazon S3 Inventory.

Thanks for pointing me to the relevant piece of source code. I don't quite understand the code structure yet, but I do get from this that I should be able to see the delete markers themselves if I would be able to show "hidden" files somehow. Is that possible?

Give this a go

v1.65.0-beta.7469.ee1c4d9d0.fix-s3-version-delete on branch fix-s3-version-delete (uploaded in 15-30 mins)

It makes delete markers visible with the --s3-version-deleted flag or version_deleted=true in the config.

From the docs:

Show deleted file markers when using versions.

This shows deleted file markers in the listing when using versions. These will appear as 0 size files. The only operation which can be performed on them is deletion.

Deleting a delete marker will reveal the previous version.

Deleted files will always show with a timestamp.

This does mean you can now undelete files by deleting the delete marker which there was no way of doing in rclone previously!

1 Like

@semafoor did that work?

Apologies for taking so very long to get back to you.

The listing works perfectly and allows me to do the analytics I wanted to do. Many thanks!

Unfortunately, I could not get removing a delete marker to work. Maybe I am doing something wrong? I would expect my attempts below to work, as deleting object versions works like this.

export RCLONE_S3_REGION=eu-west-1
export RCLONE_S3_ENV_AUTH=true
./rclone version
# rclone v1.65.0-beta.7469.ee1c4d9d0.fix-s3-version-delete
# - os/version: darwin 14.1.1 (64 bit)
# - os/kernel: 23.1.0 (arm64)
# - os/type: darwin
# - os/arch: arm64 (ARMv8 compatible)
# - go/version: go1.21.4
# - go/linking: dynamic
# - go/tags: cmount

./rclone ls :s3:$bucket/tmp --s3-versions --s3-version-deleted
#        0 testfile-v2023-11-21-211605-000
# 69409026 testfile-v2023-11-21-211543-000
#        0 testfile-v2023-11-20-233459-000
# this is correct: there is a single object version, and two delete markers, one of which is current

./rclone ls :s3:$bucket/tmp
# no results, as expected, since the current object version is a delete marker

./rclone deletefile :s3:$bucket/tmp/testfile-v2023-11-21-211605-000 --s3-versions --s3-version-deleted
# 2023/11/21 22:24:18 ERROR : Attempt 1/3 failed with 1 errors and: :s3:.../tmp/testfile-v2023-11-# 21-211605-000 is a directory or doesn't exist: object not found
# 2023/11/21 22:24:18 ERROR : Attempt 2/3 failed with 1 errors and: :s3:.../tmp/testfile-v2023-11-# 21-211605-000 is a directory or doesn't exist: object not found
# 2023/11/21 22:24:18 ERROR : Attempt 3/3 failed with 1 errors and: :s3:.../tmp/testfile-v2023-11-# 21-211605-000 is a directory or doesn't exist: object not found
# 2023/11/21 22:24:18 Failed to deletefile: :s3:.../tmp/testfile-v2023-11-21-211605-000 is a directory or doesn't exist: object not found

./rclone delete :s3:$bucket/tmp/testfile-v2023-11-21-211605-000 --s3-versions --s3-version-deleted
# no result, does not remove delete marker

./rclone delete :s3:$bucket/tmp/testfile-v2023-11-21-211543-000 --s3-versions
# no result, does remove the object version

I agree that being able to undelete files using rclone would be a cool new feature!


I think this is a bug. Try this!

v1.65.0-beta.7527.3c9ff4d5e.fix-s3-version-delete on branch fix-s3-version-delete (uploaded in 15-30 mins)

:partying_face: This version works perfectly!

Thanks for testing!

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.66

1 Like

Thanks for implementing this! Seeing how well you manage this open-source project is inspiring.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.