What is the problem you are having with rclone?
Despite usage of "--ignore-checksum --s3-disable-checksum --s3-no-check-bucket --s3-no-head --no-check-dest", multi-part uploads still error out with the old "Forbidden: Forbidden" error that plagued older versions before the --s3-no-head feature was implemented. With the combination of --s3-no-check-bucket --s3-no-head --no-check-dest, put-only buckets work well, but for some reason rclone will still attempt something with multipart uploads that causes it to retry the upload numerous times.
A workaround is to use --retries 1, then verify the file is there simply by calling "rclone ls" and manually inspecting the result to see the file is there and is the correct size (again, HEAD is not allowed on put-only buckets, which are typical in high-security organizations).
Run the command 'rclone version' and share the full output of the command.
rclone v1.64.0
- os/version: clear-linux-os 1 (64 bit)
- os/kernel: 6.6.4-1389.native (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Amazon S3 AWS
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy -P --retries 1 --ignore-checksum --s3-disable-checksum --s3-no-check-bucket --s3-no-head --no-check-dest --transfers 20 --s3-upload-concurrency 20 --s3-chunk-size 64M bigfile.dat remoteS3:mybucket/files/
The rclone config contents with secrets removed.
multiple PUT-only buckets -- let me know how to share this. calling rclone config shows an interactive prompt, not a pastable nugget.
A log from the command with the -vv flag
2024-01-23 14:19:00 ERROR : bigfile.dat: Failed to copy: multi-thread copy: failed to find object after copy: Forbidden: Forbidden
status code: 403, request id: F3D5HTD61QP5XCE2, host id: qMiM7+8PV80PKxzvfi3rnyDogEtdUuhdyqYVTMN5nrx+nZH6GgP0ziePox4zap7hn/7ePF6EhEo=
2024-01-23 14:19:00 ERROR : Attempt 1/1 failed with 1 errors and: multi-thread copy: failed to find object after copy: Forbidden: Forbidden
status code: 403, request id: F3D5HTD61QP5XCE2, host id: qMiM7+8PV80PKxzvfi3rnyDogEtdUuhdyqYVTMN5nrx+nZH6GgP0ziePox4zap7hn/7ePF6EhEo=
Transferred: 18.171 GiB / 18.171 GiB, 100%, 1.005 GiB/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 18.1s
2024/01/23 14:19:00 Failed to copy: multi-thread copy: failed to find object after copy: Forbidden: Forbidden
status code: 403, request id: F3D5HTD61QP5XCE2, host id: qMiM7+8PV80PKxzvfi3rnyDogEtdUuhdyqYVTMN5nrx+nZH6GgP0ziePox4zap7hn/7ePF6EhEo=
The old thread is here: Prevent HEAD on Amazon S3 family - #5 by ncw