Wasabi sync 500 errors

What is the problem you are having with rclone?

getting intermittent 500 errors when syncing an s3 bucket to Wasabi us-east-1

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.62.2

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-40-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Source: generic S3 on-prem service
Destination: Wasabi us-east-1

The command you were trying to run (eg rclone copy /tmp remote:tmp)

 rclone --config /export/wasabi-sync/rclone.conf sync audio:audio wasabi-audio:audio --transfers 16 --checkers 16 --progress --exclude=/temp

The rclone config contents with secrets removed.

type = s3
provider = Other
access_key_id = XXXXXXX
secret_access_key = XXXXXXXXXX
endpoint = s3.home.local

type = s3
provider = Wasabi
access_key_id = XXXXXXXX
secret_access_key = XXXXXXXX
endpoint = s3.wasabisys.com

A log from the command with the -vv flag

2023/06/02 17:24:16 Failed to sync with 30 errors: last error was: march failed with 29 error(s): first error: InternalError: We encountered an internal error.  Please retry the operation again later.
        status code: 500, request id: DF87CF5BCDBC0A80, host id: fGzjcJOrV8/9OSsI1mrGoYxrjX5GaHREnR3DSbceWHfSj3I5Y0CTg7FSaFJYfJlxdenRuDUN+r2d

I had a similar issue with terraform which was fixed by updating the client to send authenticated headers.

I should note this is a large bucket with over 130TB of data and takes several hours to sync

Could this be a similar issue?

A 500 error is a server problem at Wasabi. Trying again is the best thing to try next .

If the errors continue then I'd report them to Wasabi.

Hi Nick

thanks for the response, agreed it's most likely an issue on the Wasabi side but I wonder if its related to the article I posted about some Wasabi servers needing authentication and an update to rclone is needed to support this


i have never experienced that issue with rclone+wasabi.

you are sure the issue is with wasabi, not the on=prem s3 server?

have you contacted wasabi tech support?
i found a wasabi bug, they have yet to fix it but they were quick to test and confirm the bug.

hey, thanks for the information. I'm not sure if this error is related, the error I get points to the destination. Based on this it looked similar to the issues with that terraform plugin

2023-06-05 19:18:55 ERROR : audio: error reading destination directory: InternalError: We encountered an internal error.  Please retry the operation again later.
        status code: 500, request id: BD84686882BE92CB:A, host id: sJIDk3deVqwZDy38Ein6q8rprTGmsl6z+bgJu5pPIKm3KFABOCIZ98yTKg7jQzgoq2rY4TnOG+fK

with 130TB at wasabi, you should be able to get a decent repsonse from wasabi.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.