Rclone size with max-age as filter

I'm running

  • rclone v1.53.1
    ** os/arch: linux/amd64
    ** go version: go1.15
  • IBMCOS / s3

COS bucket has 1.5M objects and I run "daily" incrementals with (likely slightly redundant)

  • --update
  • --use-server-modtime
  • --no-traverse
  • --size-only
  • --no-update-modtime
  • -- max-age 3d

and it runs in 30-45 minutes. For reference, a complete copy (with server-side operations) takes 3-4 hours.

[Hit return too fast]

If I do a

rclone size <bucket>

it takes about 5-10 minutes. But a

rclone size <bucket> --max-age 3d

it takes far longer (I haven't let it finish, but it has run for at least an hour) and I can't understand why.

Is there something with the way max-age works?

If there's no ready explanation, I'll run a trace.

Run the command with the debug log as asked for in the help and support template and you'll see :slight_smile:

This is because rclone, to get the modified time, needs to do a HEAD request on each object.

If you just want to use the time the object was uploaded then use --use-server-modtime instead then rclone will skip the HEAD request and it will run much quicker.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.