Max size of –filter-from file

(Please pardon the repost from Help and Support - no response there)

I'm working on a rev of rclonesync that batches up all of the Path2 to Path1 copies into one rclone --filter-from file and invokes a single rclone copy Path2 Path1 --filter-from filter_copy_P2P1 . The filter file is formatted as:

+ /path/file1
+ /path/fileABC
+ /fileDEF
... (many thousands?)
- **

I'll do the same for deletes on Path1.

Questions...

  1. Is there a max practical size of the --filter-from file, such as of number of lines?
  2. The current code invokes individual, sequential rclone copies of files to be transferred from Path2 to Path1. Should this new method be significantly more efficient by taking advantage of rclone's built in multi-threaded transfer features?
  3. The current code invokes an rclone sync Path1 Path2 as the final step (after Path1 has been updated for all combined changes). Since I'll know the changes to be copied to and deleted on Path2 would it be more efficient to similarly build copy and delete filter files and issue rclone copy Path1 Path2 --filter-from filter_copy_P1P2 , and rclone delete Path2 --filter-from filter_delete_P2 commands instead of the rclone sync?

Rclone converts each line into a regexp and matches them one by one. Given enough lines this will become slow. However go is fast so I think it will take quite a few before you can notice! Even if it did take 1 second then it only does that once per file.

That said you might be better off using --files-from-raw which is designed for lists of files.

Yes, you'll get concurrency like this.

It probably depends on the number of files but if it small then using --files-from-raw will be much quicker.

You could potentially pass that list to rclone sync too I think that should work.

rclonesync V3.0 is now posted. This version makes use of --files-from-raw, and shows a dramatic improvement in some scenarios. Benchmark data is in the README.md.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.