I would like to copy contents of a bucket to a local filsystem.
I have two folders that the data needs to end up in.
Archive is ro once data is in there but InUse gets data deleted from it quickly.
I need the data to end up in both so I thought I'd run copy to InUse with --compare-dest=Archive
Then copy to Archive but what's the guarantee that the second run won't find more files on s3 and therefor makes InUse not to get them in the next run. (Unfortunately the min-age param only accepts relative time like 5seconds and not a unix timestamp that could be propagated to both commands)
How should I set this up?
Is my best chance really to lsd the source bucket and the dest Archive folder, do a diff
And then do two copy operation based on the files , first InUse than Archive?
Thanks in advance