Sync between remotes consistently

Hi there,

I'm trying to sync from an FTP server to a minio storage, and start some process when the sync finished. Is there a way to signal when rclone finished the sync?

Currently, the process which creates the files into the FTP server creates a file called ImportFinished.txt, when they finished writing the files, and it removes that file when they start to resync the files in the FTP server.

I'd be happy either:

  • if I could specify to sync the ImportFinished.txt file last
  • or if rclone syncs all the files to a temp directory, and moves that directory at the end of the sync
  • or if I could specify to start the sync only when some file exists on the source, and create some other file when the sync is finished.

Thanks,
Vilmos

How about running two rclone commands one, the first with --exclude "/ImportFinished.txt" and one with --include "/ImportFinished.txt"?

You can do this manually by doing something like

rclone copy ftp: minio:bucket/temp
rclone move minio:bucket/temp minio:bucket/final

The move on minio isn't instant though as each file will need to be server side copied, so that probably isn't ideal

You can test to see if a file exists using rclone lsf ftp: | grep ImportFinished

You can create files with rclone touch ftp:SyncFinished2.txt

I hope that is of some help!

1 Like

Thanks, that's gonna work for us.

1 Like