I am having trouble uploading large files to S3. I have managed to sync all the pictures with no problems. The larger video files fail to upload. They get to nearly 100% then fail.
C:\Users\Karl>rclone --version rclone v1.57.0 - os/version: Microsoft Windows 10 Pro 2009 (64 bit) - os/kernel: 10.0.19043.1348 (x86_64) - os/type: windows - os/arch: amd64 - go/version: go1.17.2 - go/linking: dynamic - go/tags: cmount
Sync to Amazon S3 bucket located in Ireland (I am located UK).
rclone copy F:\XXXXX\XXXXX\XXXXX\ S3:\\XXXXX\XXXXXX\ --progress --bwlimit 10M
Nb. I am using rclone encryption here.
Error with mp4 files. There is one error, but if I left the script to continue, each video would fail Same problems persists if I remove bwlimit.
S3 upload 400 bad request
Of note, the offending video file is 145 MB. I am under the allowable put request limit:
“You can send a
PUTrequest to upload an object of up to 5 GB in a single operation. For more information, see the
PutObjectexample in the AWS CLI Command Reference.” REF
My fibre bandwidth:
I’ve tried this on different days for a few months.
Next I tried multipart uploads with rclone. Used the rclone manual for recommended settings:
--s3-upload-concurrencywill increase throughput (8 would be a sensible value) and increasing
--s3-chunk-sizealso increases throughput (16M would be sensible). Increasing either of these will use more memory. The default values are high enough to gain most of the possible performance without using too much memory. ”
rclone copy F:\XXXX\XXXXX\XXXXX\ S3:\\XXXXXX\XXXX\ --progress --s3-chunk-size 16M --s3-upload-concurrency 8
This resulted in the same errors. I left it running for a while and the script reported fail uploads of 5 files (all video mp4 files)
Is there anything I am missing here?