Rclone copy using regex expression using include multiple expression for file name

Help and Support
35

Let me try to explain bit more in details.

  1. I have files copied in folder "/opt/SP/minio/datasync/demo_skip"
  2. from folder "/opt/SP/minio/datasync/demo_skip" I am copying data to minio bucket using rclone.
    rclone copy /opt/SP/minio/datasync/demo_skip s3_bucket:/demo_skip/
  3. this folder " /opt/SP/minio/datasync/demo_skip" have files retention period 30-60 days .
  4. what happening from bucket s3_bucket:/demo_skip/ application consume the files and deleted from "s3_bucket:/demo_skip/" and it is empty .
  5. one cron joob running in background copy the files again from /opt/SP/minio/datasync/demo_skip to s3_bucket:/demo_skip/ now application find the same data again and processed it :frowning:
    is there any way we can we can stop rclone to copy the files once it is done copying to bucket again and again.?

i tried to do export list of copied files in one specific file like below.

rclone md5sum /opt/SP/minio/datasync/demo_skip >> /tmp/filescopied

#cat /tmp/filescopied
d41d8cd98f00b204e9800998ecf8427e test1
d41d8cd98f00b204e9800998ecf8427e test4
d41d8cd98f00b204e9800998ecf8427e test8
c625c9b8ac50605c1da62b6e2cf23268 test3
d41d8cd98f00b204e9800998ecf8427e test9
d41d8cd98f00b204e9800998ecf8427e opt/SP/edwdata/ab_data_mount/main/serial/VFDE/public/DWH_PUB/main/mozart/incoming/DWH.2MMO.SERVICE_AGREEMENT.20210704.A901
60b8b876699360cba8fe70ae0e1ab909 test2
f888fd81cb9426a1aac0bb69919a12a1 opt/SP/edwdata/ab_data_mount/main/serial/VFDE/public/DWH_PUB/main/mozart/incoming/DWH.2MMO.SERVICE_AGREEMENT.20210702.A901
f888fd81cb9426a1aac0bb69919a12a1 DWH.2MMO.SERVICE_AGREEMENT.20210702.A901

now I am using rclone copy with --exclude-from flag
rclone copy --exclude-from=/tmp/filescopied /opt/SP/minio/datasync/demo_skip s3_bucket:/demo_skip -vvv -P
but its not work , when I redireect only file list to filescopied file then it work but I wanted to do delta check if any changes in source files then only it will trigger copy same file to bucket.

can you help how I can set logic to see checksum and filename using --exlude-from=/tmp/filescopied and then rclone decide whether it need to copy or ignore the already transferred files

let me know if you understand or any query
thanks in advance for your help :slight_smile: