What is the problem you are having with rclone?
When I run my rclone transfer from an HTTP source, I am getting just 1.5 mb/s and I need to get this running faster. My upload is taking 3-5 min / 400 mb file and I really need to get that all the way down to 3-5 sec.
My baseline burst GB/s on the EC2 instance is 5. I'm hoping to find the optimal setting to get this copy function close to this speed. I cannot easily use clone as the directory contains a bunch of files with irregular strings I don't want in AWS.
I have looked into using a chunker overlay and have 1 configured in my .config file (called overlay- see below) to run on my source, but this doesn't seem to be boosting my upload either. I have chunker configured to chunk for anything 1M+ in size.
What is your rclone version (output from
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Amazon Linux 2 x64
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg
rclone copy /tmp remote:tmp)
rclone copy --ignore-existing -v prs_idx, aws_prs --buffer-size=512M --tpslimit=40 --transfers=20 --tpslimit-burst=10
The rclone config contents with secrets removed.
[horelS3] type = s3 env_auth = false access_key_id = XXX secret_access_key = XXX #region = other-v2-signature endpoint = https://pando-rgw01.chpc.utah.edu location_constraint = [AWS test] type = s3 provider = AWS env_auth = false access_key_id = XXX secret_access_key = XXX region = us-west-1 [NOMADs] type = http url = https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/ [NCEP] type = http url = https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/ [AWS Grib] type = s3 provider = AWS env_auth = true region = us-east-1 [overlay] type = chunker remote = NCEP chunk_size = 1M hash_type = md5
A log from the command with the
2020/11/11 23:09:53 INFO : Transferred: 265.996M / 385.768 MBytes, 69%, 1.479 MBytes/s, ETA 1m21s Transferred: 0 / 1, 0% Elapsed time: 3m0.5s Transferring: * hrrr.t21z.wrfprsf15.grib2: 68% /385.768M, 1.528M/s, 1m18s 2020/11/11 23:10:53 INFO : Transferred: 352.996M / 385.768 MBytes, 92%, 1.471 MBytes/s, ETA 22s Transferred: 0 / 1, 0% Elapsed time: 4m0.5s Transferring: * hrrr.t21z.wrfprsf15.grib2: 91% /385.768M, 1.448M/s, 22s 2020/11/11 23:11:15 INFO : hrrr.t21z.wrfprsf15.grib2: Copied (new) 2020/11/11 23:11:15 INFO : Transferred: 385.768M / 385.768 MBytes, 100%, 1.473 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 4m22.5s 2020/11/11 23:11:15 INFO : Starting HTTP transaction limiter: max 40 transactions/s with burst 10 2020/11/11 23:11:15 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Checks: 1 / 1, 100% Elapsed time: 0.5s