Chunks remaining after completed S3 upload

Hello!

After syncing a directory of around 80k files (biggest one about 6 GiB) to a S3 bucket the storage provider (Scaleway) still shows ~2,500 "parts" (that's chunks, I guess) in the bucket. Now, shouldn't this number be zero after an upload is finished and the parts/chunks have been assembled back into files?

Is this possibly an unwanted situation, caused by several aborted multipart uploads that haven't properly cleaned up after themselves?
Additionally, this only affects the bucket I have filled via rclone sync. My other buckets, for example used by longhorn S3 backup show zero parts next to the object count.

Thanks!

hello and welcome to the forum,

when posting, there should have been a template of questions to be answered.
including to post a rclone debug log, as that should show what is going on.

Yes that is most likely the cause.

Check out rclone cleanup to fix the problem.

Thank you, but after some more investigation I think there is no problem to fix. First I used rclone cleanup (which found nothing to clean up) and then I did another multipart upload without interruptions, ending up with the same "x objects, y parts" display at Scaleway (with y being the number of chunks rclone did upload).

From their documentation I got the impression those "y parts" would still need to be assembled into objects, and was worried why this hasn't happened. However, all files are present and intact in the bucket. So that "parts" count seems to be just a (rather pointless?) piece of extra information without any consequence to the overall integrity of the bucket contents.

(As for my longhorn buckets with zero parts ... they simply don't contain objects large enough to be broken into chunks.)

So, thanks again I guess and consider this solved please.

1 Like