How to sync kafka data using rclone from file storage to bucket

What is the problem you are having with rclone?

I want to sync kafka data from one region cluster to another region. For which I want to sync data from one region to OCI bucket and then from there to another region filesystem. But since kafka data modifies continuesly how to handle?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

OCI storage bucket

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

Paste config here

A log from the command that you were trying to run with the -vv flag

Paste  log here
1 Like

Design your system to support filesystem snapshots - use BTRFS/ZFS/LVM. Then you can make sure that synced data does not change during sync work.

Here some read for thought.