Problem Summary
I have two issues, but the main one I'd like assistance with is that since upgrading to 1.59.0
part of the sync is failing with the error:
multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 10000, inclusive
I'll provide the full logs below. But nothing else has changed config wise, or on the S3 side. The previous version was 1.58.0
rclone version'
rclone v1.59.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-122-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.3
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone sync --check-first --fast-list --ignore-existing --transfers 40 /mnt/backup/my/backups/ my_s3:my-archive/backups/
But this is part of a bash script, and this is the full section of the script trying to run the sync.
logger -t "RCLONE" "rclone START for /mnt/backup/my/backups/"
if ! rclone sync --check-first --fast-list --ignore-existing --transfers 40 /mnt/backup/my/backups/ my_s3:my-archive/backups/ 2>&1 | tee >(logger -t "RCLONE");
then
# rclone failed
logger -t "RCLONE" "rclone FAILED for /mnt/backup/my/backups/"
backup_status=1
fi
logger -t "RCLONE" "rclone FINISH for /mnt/backup/my/backups/"
The rclone config contents with secrets removed.
[my_s3]
type = s3
provider = AWS
env_auth = false
access_key_id = <redacted>
secret_access_key = <redacted>
region = eu-west-2
location_constraint = eu-west-2
acl = private
upload_concurrency = 8
A log from the command with the -vv
flag
Unfortunately as this is a live system (and this error has made us go over our data transfer quota) I cannot run this at the moment. But below are the logs that my script had written to syslog.
Aug 02 02:00:02 tooting RCLONE[2166904]: rclone START for /mnt/backup/my/backups/
Aug 02 03:29:09 tooting RCLONE[2166908]: 2022/08/02 03:29:09 ERROR : CBB_DBSVR/CBB_Database/DBSVR/large.database/20220730200006/large.database_full_20220730200006.bak: Failed to copy: multipart upload failed to>
Aug 02 03:29:09 tooting RCLONE[2166908]: status code: 400, <redacted>, host id: <redacted>
Aug 02 03:29:09 tooting RCLONE[2166908]: 2022/08/02 03:29:09 ERROR : S3 bucket my-archive path backups: not deleting files as there were IO errors
Aug 02 03:29:09 tooting RCLONE[2166908]: 2022/08/02 03:29:09 ERROR : S3 bucket my-archive path backups: not deleting directories as there were IO errors
Aug 02 03:29:09 tooting RCLONE[2166908]: 2022/08/02 03:29:09 ERROR : Attempt 1/3 failed with 1 errors and: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 10000, inclusive
Aug 02 03:29:09 tooting RCLONE[2166908]: status code: 400, <redacted>, host id: <redacted>
Aug 02 04:57:30 tooting RCLONE[2166908]: 2022/08/02 04:57:30 ERROR : CBB_DBSVR/CBB_Database/DBSVR/large.database/20220730200006/large.database_full_20220730200006.bak: Failed to copy: multipart upload failed to>
Aug 02 04:57:30 tooting RCLONE[2166908]: status code: 400, <redacted>, host id: <redacted>
Aug 02 04:57:30 tooting RCLONE[2166908]: 2022/08/02 04:57:30 ERROR : S3 bucket my-archive path backups: not deleting files as there were IO errors
Aug 02 04:57:30 tooting RCLONE[2166908]: 2022/08/02 04:57:30 ERROR : S3 bucket my-archive path backups: not deleting directories as there were IO errors
Aug 02 04:57:30 tooting RCLONE[2166908]: 2022/08/02 04:57:30 ERROR : Attempt 2/3 failed with 1 errors and: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 10000, inclusive
Aug 02 04:57:30 tooting RCLONE[2166908]: status code: 400, <redacted>, host id: <redacted>
Aug 02 06:25:58 tooting RCLONE[2166908]: 2022/08/02 06:25:58 ERROR : CBB_DBSVR/CBB_Database/DBSVR/large.database/20220730200006/large.database_full_20220730200006.bak: Failed to copy: multipart upload failed to>
Aug 02 06:25:58 tooting RCLONE[2166908]: status code: 400, <redacted>, host id: <redacted>
Aug 02 06:25:58 tooting RCLONE[2166908]: 2022/08/02 06:25:58 ERROR : S3 bucket my-archive path backups: not deleting files as there were IO errors
Aug 02 06:25:58 tooting RCLONE[2166908]: 2022/08/02 06:25:58 ERROR : S3 bucket my-archive path backups: not deleting directories as there were IO errors
Aug 02 06:25:58 tooting RCLONE[2166908]: 2022/08/02 06:25:58 ERROR : Attempt 3/3 failed with 1 errors and: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 10000, inclusive
Aug 02 06:25:58 tooting RCLONE[2166908]: status code: 400, <redacted>, host id: <redacted>
Aug 02 06:25:58 tooting RCLONE[2166908]: 2022/08/02 06:25:58 Failed to sync: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 10000, inclusive
Aug 02 06:25:58 tooting RCLONE[2166908]: status code: 400, <redacted>, host id: <redacted>
Aug 02 06:25:58 tooting RCLONE[2192745]: rclone FINISH for /mnt/backup/my/backups/