# S3 object deletion times

**URL:** <https://forum.rclone.org/t/s3-object-deletion-times/42781>\
**Category:** Feature\
**Created:** [November 8, 2023, 10:00am UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781 "2023-11-08T10:00:50Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![semafoor](https://forum.rclone.org/letter_avatar_proxy/v4/letter/s/ce7236/32.png) [@semafoor](https://forum.rclone.org/u/semafoor)\
**Post date:** [November 8, 2023, 10:00am UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/1 "2023-11-08T10:00:50Z")

</div>

I am using the S3 backend and listing object versions using `--s3-versions`. I am interested not only in when an object version was created but also in when it was deleted. I think of the deletion time as when the version becomes non-current, which corresponds to the creation time of a new version or a deletion marker.

Is there currently a way to get this information from rclone? Does this seem like a reasonably common want?

For me, the most convenient implementation would be a flag that sets the deletion time as the object version's modification date. I understand this could be confusing, as the modification date is already used both for rclone's modification date and object birth time (when using `--use-server-modtime`).

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [November 8, 2023, 11:47am UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/2 "2023-11-08T11:47:20Z")

</div>

(A quick recap to make sure we understand each other)

The version where the file is deleted will be the last version with a full prefix, so if your file was called `file2.txt` then its last version might be

```
file2-v2022-07-07-165438-000.txt

```

This is the date that `file2.txt` was created.

Amazon does provide the information when the file was deleted in the delete markers.

Eg

```auto
  <DeleteMarker>
    <Key>hello</Key>
    <VersionId>AFTrN_QHemMvPM505pYuqo3_MOAFKzdW</VersionId>
    <IsLatest>true</IsLatest>
    <LastModified>2023-03-21T10:57:57.000Z</LastModified>
    <Owner>
      <ID>bdda7f5ee2c061a28f824d1890f570ff4e32f13811267fa8c19ac4dd2a7bd6c5</ID>
    </Owner>
  </DeleteMarker>

```

You could probably do what you want in this function in the backend and adjust the object somehow.

> <https://github.com/rclone/rclone/blob/ad83ff769b03b0731757431a596823822d12ae29/backend/s3/s3.go#L3657>

> [@semafoor](#):
>
> For me, the most convenient implementation would be a flag that sets the deletion time as the object version's modification date. I understand this could be confusing, as the modification date is already used both for rclone's modification date and object birth time (when using `--use-server-modtime`).

It might be better to set a bit of metadata on the object say a `dtime` for deleted time.

What do you want to do with this info? Make a report? Or something else? That might change my thinking on how to output this info.

It might be easier to make a backend command to output this info directly.

---

<div class="post-metadata">

**Author:** ![semafoor](https://forum.rclone.org/letter_avatar_proxy/v4/letter/s/ce7236/32.png) [@semafoor](https://forum.rclone.org/u/semafoor)\
**Post date:** [November 8, 2023, 12:56pm UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/3 "2023-11-08T12:56:28Z")

</div>

We understand each other :). On AWS, information on the time of deletion is indeed available in the delete marker with the same prefix, not in the object version metadata itself. If an object is overridden, the "deletion time" of the old object version version can be the creation of the new version.

Setting a bit of metadata (`dtime`) would indeed make a lot of sense and probably be a better option compared to setting modification times. It would be great though if it were possible to get `dtime` without also retrieving (rclone) modification times. Asking rclone to retrieve this metadata (e.g. by setting `--metadata`) slows down the list operation a _lot_ (at least two orders of magnitude).

My use case is to analyze the contents of a bucket (distribution of file sizes, growth over time, etc). For this I export a list of object versions using `rclone lsjson`. I am connecting to buckets residing on MinIO; on AWS I would probably use [Amazon S3 Inventory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html).

Thanks for pointing me to the relevant piece of source code. I don't quite understand the code structure yet, but I do get from this that I should be able to see the delete markers themselves if I would be able to show "hidden" files somehow. Is that possible?

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [November 8, 2023, 3:32pm UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/4 "2023-11-08T15:32:40Z")

</div>

> [@semafoor](#):
>
> Thanks for pointing me to the relevant piece of source code. I don't quite understand the code structure yet, but I do get from this that I should be able to see the delete markers themselves if I would be able to show "hidden" files somehow. Is that possible?

Give this a go

[v1.65.0-beta.7469.ee1c4d9d0.fix-s3-version-delete](https://beta.rclone.org/branch/fix-s3-version-delete/v1.65.0-beta.7469.ee1c4d9d0.fix-s3-version-delete/) on branch [fix-s3-version-delete](https://github.com/rclone/rclone/tree/fix-s3-version-delete) (uploaded in 15-30 mins)

It makes delete markers visible with the `--s3-version-deleted` flag or `version_deleted=true` in the config.

From the docs:

> Show deleted file markers when using versions.
> 
> This shows deleted file markers in the listing when using versions. These will appear as 0 size files. The only operation which can be performed on them is deletion.
> 
> Deleting a delete marker will reveal the previous version.
> 
> Deleted files will always show with a timestamp.

This does mean you can now undelete files by deleting the delete marker which there was no way of doing in rclone previously!

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [November 20, 2023, 5:55pm UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/5 "2023-11-20T17:55:22Z")

</div>

@semafoor did that work?

---

<div class="post-metadata">

**Author:** ![semafoor](https://forum.rclone.org/letter_avatar_proxy/v4/letter/s/ce7236/32.png) [@semafoor](https://forum.rclone.org/u/semafoor)\
**Post date:** [November 21, 2023, 9:29pm UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/6 "2023-11-21T21:29:21Z")

</div>

Apologies for taking so very long to get back to you.

The listing works perfectly and allows me to do the analytics I wanted to do. Many thanks!

Unfortunately, I could not get removing a delete marker to work. Maybe I am doing something wrong? I would expect my attempts below to work, as deleting object versions works like this.

```bash
export RCLONE_S3_REGION=eu-west-1
export RCLONE_S3_ENV_AUTH=true
export RCLONE_S3_PROVIDER=AWS
./rclone version
# rclone v1.65.0-beta.7469.ee1c4d9d0.fix-s3-version-delete
# - os/version: darwin 14.1.1 (64 bit)
# - os/kernel: 23.1.0 (arm64)
# - os/type: darwin
# - os/arch: arm64 (ARMv8 compatible)
# - go/version: go1.21.4
# - go/linking: dynamic
# - go/tags: cmount

./rclone ls :s3:$bucket/tmp --s3-versions --s3-version-deleted
# 0 testfile-v2023-11-21-211605-000
# 69409026 testfile-v2023-11-21-211543-000
# 0 testfile-v2023-11-20-233459-000
# this is correct: there is a single object version, and two delete markers, one of which is current

./rclone ls :s3:$bucket/tmp
# no results, as expected, since the current object version is a delete marker

./rclone deletefile :s3:$bucket/tmp/testfile-v2023-11-21-211605-000 --s3-versions --s3-version-deleted
# 2023/11/21 22:24:18 ERROR : Attempt 1/3 failed with 1 errors and: :s3:.../tmp/testfile-v2023-11-# 21-211605-000 is a directory or doesn't exist: object not found
# 2023/11/21 22:24:18 ERROR : Attempt 2/3 failed with 1 errors and: :s3:.../tmp/testfile-v2023-11-# 21-211605-000 is a directory or doesn't exist: object not found
# 2023/11/21 22:24:18 ERROR : Attempt 3/3 failed with 1 errors and: :s3:.../tmp/testfile-v2023-11-# 21-211605-000 is a directory or doesn't exist: object not found
# 2023/11/21 22:24:18 Failed to deletefile: :s3:.../tmp/testfile-v2023-11-21-211605-000 is a directory or doesn't exist: object not found

./rclone delete :s3:$bucket/tmp/testfile-v2023-11-21-211605-000 --s3-versions --s3-version-deleted
# no result, does not remove delete marker

./rclone delete :s3:$bucket/tmp/testfile-v2023-11-21-211543-000 --s3-versions
# no result, does remove the object version

```

I agree that being able to undelete files using rclone would be a cool new feature!

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [November 23, 2023, 10:02am UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/7 "2023-11-23T10:02:41Z")

</div>

> [@semafoor](#):
>
> The listing works perfectly and allows me to do the analytics I wanted to do. Many thanks!

Great!

> [@semafoor](#):
>
> Unfortunately, I could not get removing a delete marker to work.

I think this is a bug. Try this!

[v1.65.0-beta.7527.3c9ff4d5e.fix-s3-version-delete](https://beta.rclone.org/branch/fix-s3-version-delete/v1.65.0-beta.7527.3c9ff4d5e.fix-s3-version-delete/) on branch [fix-s3-version-delete](https://github.com/rclone/rclone/tree/fix-s3-version-delete) (uploaded in 15-30 mins)

---

<div class="post-metadata">

**Author:** ![semafoor](https://forum.rclone.org/letter_avatar_proxy/v4/letter/s/ce7236/32.png) [@semafoor](https://forum.rclone.org/u/semafoor)\
**Post date:** [November 27, 2023, 11:11am UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/9 "2023-11-27T11:11:09Z")

</div>

> [@ncw](#):
>
> [v1.65.0-beta.7527.3c9ff4d5e.fix-s3-version-delete](https://beta.rclone.org/branch/fix-s3-version-delete/v1.65.0-beta.7527.3c9ff4d5e.fix-s3-version-delete/) on branch [fix-s3-version-delete](https://github.com/rclone/rclone/tree/fix-s3-version-delete) (uploaded in 15-30 mins)

🥳 This version works perfectly!

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [November 29, 2023, 9:46am UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/10 "2023-11-29T09:46:33Z")

</div>

Thanks for testing!

I've merged this to master now which means it will be in [the latest beta](https://beta.rclone.org/) in 15-30 minutes and released in v1.66

---

<div class="post-metadata">

**Author:** ![semafoor](https://forum.rclone.org/letter_avatar_proxy/v4/letter/s/ce7236/32.png) [@semafoor](https://forum.rclone.org/u/semafoor)\
**Post date:** [November 29, 2023, 3:52pm UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/11 "2023-11-29T15:52:33Z")

</div>

Thanks for implementing this! Seeing how well you manage this open-source project is inspiring.

---

<div class="post-metadata">

**Author:** ![system](https://forum.rclone.org/uploads/default/original/2X/3/39ad3235f6ffec76ccd0c81616bd55697b4c300d.png) [@system](https://forum.rclone.org/u/system)\
**Post date:** [January 28, 2024, 3:52pm UTC](https://forum.rclone.org/t/s3-object-deletion-times/42781/12 "2024-01-28T15:52:48Z")

</div>

Closed on January 28, 2024, 3:52pm UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.