Rclone copy using regex expression using include multiple expression for file name

siddiquis3 · October 19, 2021, 1:34pm

yes correct in addition in case source files change in size or updates then it should copy the file to dst.

asdffdsa · October 19, 2021, 2:16pm

then i think that my suggestion should work.

on source: rclone copy sftp:/incoming /datasync --compare-dest=/alreadyprocessed
on dest: application processes files in /datasync. do not delete the processed files.
on dest: mv /datasync/ /alreadyprocessed/
goto 1.

siddiquis3 · October 19, 2021, 2:58pm

Let me test this and update you
Thanks for your help so far

asdffdsa · October 20, 2021, 6:52pm

i see that you marked this as solved, great!

curious as to how my suggestion worked out for you?

siddiquis3 · October 21, 2021, 4:34am

I follow the same process you mentioned ,

rclone copy /datasync/ minio:bucket/ --compare-dest=/alreadyprocessed

Then
2. rclone move minio:bucket /alreadyprocessed

So by comparing /alreadyprocessed to sftp/incomming it is not copying the files which is already in present in /alreadyprocessed and copying when the new files there or change in file
3. Then step 1 again
Only thing due to comparisons we need to keep double space one for local directory and one for intermediate directory

system · October 24, 2021, 4:35am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.