Hello, I have a netapp storage grid, in order to use an ILM rule to purge old files I need to set x-amz-meta-creation-time. Is that possible or is there a similar flag I can set when doing the upload? I searched and can't find any info so far, I have 2000 TB of data (30 M files) that I need to move which will take a few months, need to figure out the best way to do this (can't edit each one in the S3 bucket). Rclone keeps the timestamps which is great, but the bucket properties ignore it, I can resort to a script that looks at the timestamps if there isn't an easier way that I'm missing.
Hi Nick, I think it's just a netapp thing, it's a bucket property that gives you 3 options for cleaning up old files, injest, last access, or user defined.
x-amz-meta-creation-time needs to be the same as the mod-time, we have existing data going back 13 months and need to roll off the oldest, but netapp is ignoring the timestamps and views them all as the day you moved them. They are being copied from Isilon NAS storage.
Here's the options the ILM gives you, can't believe it can't just look at the timestamp...
This is a tiny patch which just duplicates the modtime into the creation-time for uploads.
diff --git a/backend/s3/s3.go b/backend/s3/s3.go
index c49d9c9b7..327922ca2 100644
--- a/backend/s3/s3.go
+++ b/backend/s3/s3.go
@@ -2036,6 +2036,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
metadata := map[string]*string{
metaMtime: aws.String(swift.TimeToFloatString(modTime)),
}
+ metadata["creation-time"] = metadata[metaMtime]
// read the md5sum if available for non multpart and if
// disable checksum isn't present.
I'm not sure I want to merge that as-is as it is a bit special purpose, but see if it works for you!
It might be that it doesn't as the time rclone writes is a floating point number - that is easy to fix though.
Thanks! I tried it and see the creation time set for my October file, just have to wait and see if netapp plays nice:o) We have a test bucket set to 2 days so I'll know whenever it runs it's policy.
Hi Nick, one favor please... can you nix the floating point number part of it for me? The purge policy didn't work last night, the file attribute is exactly what the doc calls for minus the precision.
Hi Nick, I can't imagine anyone would code their system to be case sensitive, but have you ever heard of anyone having an issue with camel case for the keys? All the netapp docs have lowercase for everything, I still am not seeing it delete as expected. Is it possible to make the keys all lowercase?
"To record the object creation time, so that you can use the User Defined Creation Time option for the reference time in an ILM rule, you need to store the value in a user-defined header named x-amz-meta-creation-time. For example: x-amz-meta-creation-time=1443399726."
Hey Nick, I think i'm good here, the case doesn't matter, I can see the date in the bucket through an admin tool, I think we just don't have the bucket rule setup correctly yet. I'll let you know if this starts working, the admin guy isn't back for another week.
Can you help me with the help text? Here is a start - please amend for accuracy!
The first line is the help when you do rclone help flags - the complete text is what you see in the docs
--s3-set-creation-time
Set x-amz-creation-time for use with netapp storage grid
If set this flag will set x-amz-creation-time metadata when uploading objects. This will be set to the modified time of the uploaded object in seconds since the epoch.
This flag is understood by netapp storage grid and can be used to XXX?
hey Nick, sorry, just noticed this post. I can try to explain a bit with an example: Thanks for the help, this works perfect!
This flag is understood by netapp storage grid and can be used to purge files based on this time vs. the injestion time (the time they were actually moved to S3). When migrating old files to S3 they all appear "new" and a policy based on injestion time to delete files older than say 365 days would never delete files that are actually older than 365 days.
From the netapp manual:
"To record the object creation time, so that you can use the User Defined Creation Time option for the reference time in an ILM rule, you need to store the value in a user-defined header named x-amz-meta-creation-time. For example: x-amz-meta-creation-time=1443399726."
