Minimize duplicate uploads when transferring using two VPS

Is there any way to minimize duplicate uploads of the same files when I use two VPS to transfer from A to B?

Yes, as VPS1 makes a copy of the file A:test/test1-5.zip B:test/test1-5.zip, VPS2 server did the test6+ transfer.

Is it a problem you are trying to solve or just sharing ideas?

I have a problem that two VPS are trying to copy from A to B the same file. With VPS1 copying it before VPS2 started copying the same file

For example, I have test1-10.zip files of 5GB each.

rclone ls A:
test1.zip 5GB
test2.zip 5GB
test3.zip 5GB
(...)
test10.zip 5GB

On VPS1 run the script:

rclone copy A: B: --order-by-name, asc

On VPS2 run the script:

rclone copy A: B: --order-by name, desc

Problem: How to force VPS2 not to copy files that VPS1 copied ???

OK I use filters for cases like this but your approach is fine too.

create file filter.txt with this content:

+ /Dir/[a-b]**/**
+ /Dir/[A-B]**/**
+ /Dir/[0-2]**/**
- **

This will copy all folders from Dir starting with AaBb012

then:

rclone copy source: destination: --filter-from filter.txt

You can modify filter to suit your needs.

try with-vv -dry-run before you run for real

So on one machine you can copy everything starting from A to L and on other one from K to Z for example.

You do not have to worry about any overlap. If file1 is already copied than other copy simply will skip it.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.