Smartest way to do regular syncing

What is the problem you are having with rclone?

Having 10s of millions of objects impossible to hourly sync.
My question is similar to this:

I wonder is there a way to not start the check from the beginning, is there a kind of flag mechanism with some metadata that fastens the check and don't scan all the objects, only scans the modified/flagged objects?

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0-DEV

  • os/version: centos 8.5.2111 (64 bit)
  • os/kernel: 4.18.0-348.7.1.el8_5.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.12
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Ceph S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --no-check-certificate=true --size-only --transfers=128 --checkers=128 -P user:bucket user2:bucket2

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

used latest from official centos repo, no redacted option

A log from the command that you were trying to run with the -vv flag

not necessary

You are using really ancient version of rclone - it is now 1.64.2 - you have to update it.

The fastest way for hourly "sync" (after initial full sync) would be to run:

rclone copy --min-age 1h --no-traverse source: destination:

it won't scan destination and only copy what has been updated within the last hour in the source.

and run rclone sync e.g. once a day outside of office hours to remove from destination any files deleted from source etc.

it's not max-age=1h so if something changed in the last hour. As I understand min-age it means older than 1 hour.

You are right. It should be:

rclone copy --max-age 1h --no-traverse source: destination:

I make this mistake often.

Anyway always good to test with -vv --dry-run before running anything in real.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.