The starting of the transfers takes a very long time, which becomes visible with a files-from file with 10M entries.
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-1045-aws (x86_64)
rclone copy s3:a s3:b -P --files-from=segment1M --no-traverse --ignore-existing -vv rclone copy s3:a s3:b -P --files-from=segment10M --no-traverse --ignore-existing -vv
[s3] type = s3 provider = AWS env_auth = false access_key_id = … secret_access_key = …
2021/11/04 08:35:05 DEBUG : rclone: Version "v1.58.0-beta.5848.454574e2c" starting with parameters ["rclone" "copy" "s3:a" "s3:b" "-P" "--files-from=segment1M" "--no-traverse" "--ignore-existing" "-vv"] 2021/11/04 08:35:05 DEBUG : Creating backend with remote "s3:a" 2021/11/04 08:35:05 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf" 2021/11/04 08:35:05 DEBUG : Creating backend with remote "s3:b"
It's clear that it takes a moment to read the file, but that should be done in under a minute. What I observe is the following:
With 1M entries it takes about 5 minutes after the last debug message for the checking to start. There's no output, not much CPU usage, no diskusage during that time, so it's unclear what's happening in those 5 minutes (after loading the file and before starting checking (and then later transfer)).
With a file that has 10M entries it took 40 minutes for the first checks to start happening.
It seems like there is something linear to the number of entries in the files-from file happening, which does not generate any log output.