Max size of --filter-from file

What is the problem you are having with rclone?

I'm working on a rev of rclonesync that batches up all of the Path2 to Path1 copies into one rclone --filter-from file and invokes a single rclone copy Path2 Path1 --filter-from filter_copy_P2P1. The filter file is formatted as:

+ path/file1
+ path/fileABC
+ fileDEF
...
- **

I'll do the same for deletes on Path1.

Questions...

  1. Is there a max practical size of the --filter-from file, such as of number of lines?
  2. The current code invokes individual, sequential rclone copies of files to be transferred from Path2 to Path1. Will this new method be significantly more efficient by taking advantage of rclone's built in multi-threaded transfer features?
  3. The current code invokes an rclone sync Path1 Path2 as the final step (after Path1 has been updated for all combined changes). Since I'll know the changes to be copied to and deleted on Path2 would it be more efficient to similarly build copy and delete filter files and issue rclone copy Path1 Path2 --filter-from filter_copy_P1P2, and rclone delete Path2 --filter-from filter_delete_P2 commands instead of the rclone sync?

What is your rclone version (output from rclone version)

1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Centos 7.8 64b and Windows 10 64b

Which cloud storage system are you using? (eg Google Drive)

all

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy PathA PathB --filter-from FilesToCopy

The rclone config contents with secrets removed.

na

A log from the command with the -vv flag

na

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.