From my research delete markers only apply to buckets with versioning enabled or suspended. When I created the bucket I enabled versioning prior to adding the files using rclone. I’ve since disabled versioning but that means it’s now “suspended”. To resolve this I may need to create a new bucket and move all of my files. Could we possibly add a flag to rclone when doing a move or delete on a versioning enabled/suspended bucket to delete the file by version ID so the delete marker is not created?
Here is S3 documentation on versioning and delete markers.
Using Versioning
Versioning-enabled buckets enable you to recover objects from accidental deletion or overwrite. For example:
*If you delete an object, instead of removing it permanently, Amazon S3 inserts a delete marker, which becomes the current object version. You can always restore the previous version. For more information, see Deleting Object Versions.
Managing Objects in a Versioning-Suspended Bucket
When you suspend versioning, existing objects in your bucket do not change. What changes is how Amazon S3 handles objects in future requests. The topics in this section explain various object operations in a versioning-suspended bucket.
Adding Objects to Versioning-Suspended Buckets
Once you suspend versioning on a bucket, Amazon S3 automatically adds a null version ID to every subsequent object stored thereafter (using PUT, POST, or COPY) in that bucket.
If a null version is already in the bucket and you add another object with the same key, the added object overwrites the original null version.
If there are versioned objects in the bucket, the version you PUT becomes the current version of the object. The following figure shows how adding an object to a bucket that contains versioned objects does not overwrite the object already in the bucket. In this case, version 111111 was already in the bucket. Amazon S3 attaches a version ID of null to the object being added and stores it in the bucket. Version 111111 is not overwritten.
Deleting Objects from Versioning-Suspended Buckets
If versioning is suspended, a DELETE request:
*Can only remove an object whose version ID is null
Doesn’t remove anything if there isn’t a null version of the object in the bucket.
*Inserts a delete marker into the bucket.
Working with Delete Markers
A delete marker is a placeholder (marker) for a versioned object that was named in a simple DELETE request. Because the object was in a versioning-enabled bucket, the object was not deleted. The delete marker, however, makes Amazon S3 behave as if it had been deleted.
Delete markers accrue a nominal charge for storage in Amazon S3. The storage size of a delete marker is equal to the size of the key name of the delete marker. A key name is a sequence of Unicode characters. The UTF-8 encoding adds from 1 to 4 bytes of storage to your bucket for each character in the name.
Removing Delete Markers
To delete a delete marker, you must specify its version ID in a DELETE Object versionId request. If you use a DELETE request to delete a delete marker (without specifying the version ID of the delete marker), Amazon S3 does not delete the delete marker, but instead, inserts another delete marker.