I need to copy 100TB+ from S3 to Local Disk. When the files hit the destination they will be picked up by another process and moved. Will rclone have issues when running the copy command if destination files are removed after each file is fully copied?
I wasn’t sure if the checks would get thrown off.
Thanks!
Hmm, potentially if there is a retry it will copy all the files again which is probably not what you want, so using --retries 1 would fix that. That will mean that if one file doesn’t get copied (say a transient network error) it might get lost.
I think I’d wait until the rclone process has finished before processing the output files.