We have a rclone sync that has been running to copy files from one bucket to another in a different region. It has been running for about 4 days on a 80TB. I was reading more on chunk and concurrent settings and was wondering if I should tweak the config to get more speed. We are running on an xlarge EC2 with 16GB of memory. I am wondering if I am going to possibly loose anything if I try and change config and restart. Any thoughts? Is it worth changing?
You will lose any transfers that are in progress. Depending on how big your files are that might be a few seconds uploading or hours of uploading…
You’ll also lose the directory listing which may take some time depending on how many files you’ve got.
For an S3 transfer I’d recommend using --fast-list to minimize the number transactions and --checksum (so rclone doesn’t read the modtime metadata which costs an extra transaction).
1 Like