Smartest way to do regular syncing

[y_x]

What is the problem you are having with rclone?

Having 10s of millions of objects impossible to hourly sync.
My question is similar to this:

Fastest way to check for changes in 2.5+ Million files?

Hello! I am using the latest rclone version (linux/amd64) and I want to sync a directory on my NAS to GDrive which consists of over 2.5 million files and was wondering what the fastest way to check for changes to the files through all of them. /opt/bin/rclone sync "/share/CACHEDEV1_DATA/Username/" "gcrypt:shared/Username" --backup-dir "gcrypt:shared/Backups/Username/"`date -I`/`date +%HH%MM` --log-file /share/CACHEDEV1_DATA/rclone/sync.log -v --tpslimit 10 --transfers 10 --checkers 10 --max-ba…

I wonder is there a way to not start the check from the beginning, is there a kind of flag mechanism with some metadata that fastens the check and don't scan all the objects, only scans the modified/flagged objects?

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0-DEV

• os/version: centos 8.5.2111 (64 bit)
• os/kernel: 4.18.0-348.7.1.el8_5.x86_64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.16.12
• go/linking: dynamic
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Ceph S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --no-check-certificate=true --size-only --transfers=128 --checkers=128 -P user:bucket user2:bucket2

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

used latest from official centos repo, no redacted option

A log from the command that you were trying to run with the -vv flag

not necessary

[kapitainsky]

@y_x

rclone v1.57.0-DEV

You are using really ancient version of rclone - it is now 1.64.2 - you have to update it.

The fastest way for hourly "sync" (after initial full sync) would be to run:

rclone copy --min-age 1h --no-traverse source: destination:

it won't scan destination and only copy what has been updated within the last hour in the source.

and run rclone sync e.g. once a day outside of office hours to remove from destination any files deleted from source etc.

[y_x]

@kapitainsky

--min-age 1h

it's not max-age=1h so if something changed in the last hour. As I understand min-age it means older than 1 hour.

[kapitainsky]

You are right. It should be:

rclone copy --max-age 1h --no-traverse source: destination:

I make this mistake often.

Anyway always good to test with -vv --dry-run before running anything in real.