Rclone size with max-age as filter

[dlluner (David Luner)]

I'm running

• rclone v1.53.1
  ** os/arch: linux/amd64
  ** go version: go1.15
• IBMCOS / s3

COS bucket has 1.5M objects and I run "daily" incrementals with (likely slightly redundant)

• --update
• --use-server-modtime
• --no-traverse
• --size-only
• --no-update-modtime
• -- max-age 3d

and it runs in 30-45 minutes. For reference, a complete copy (with server-side operations) takes 3-4 hours.

[dlluner (David Luner)]

[Hit return too fast]

If I do a

rclone size <bucket>

it takes about 5-10 minutes. But a

rclone size <bucket> --max-age 3d

it takes far longer (I haven't let it finish, but it has run for at least an hour) and I can't understand why.

Is there something with the way max-age works?

If there's no ready explanation, I'll run a trace.

[Animosity022]

Run the command with the debug log as asked for in the help and support template and you'll see

[ncw (Nick Craig-Wood)]

@dlluner

it takes about 5-10 minutes. But a

rclone size <bucket> --max-age 3d

it takes far longer (I haven't let it finish, but it has run for at least an hour) and I can't understand why.

This is because rclone, to get the modified time, needs to do a HEAD request on each object.

If you just want to use the time the object was uploaded then use --use-server-modtime instead then rclone will skip the HEAD request and it will run much quicker.