Hello Guys
I usually upload about 50GB by day for Wasabi S3 account, and almost always it spent about 3 hours, and now it is spending about 12 hours to upload
How can I know if the problem is my link or Wasabi limitation?
I've made some internet speed tests and it looks my link is OK.
I hear S3 should be plenty fast, so assuming you have the bandwidth for it it sounds like you should be getting more. There are a lot of things that could be going on here...
Does your transfers include very many smaller files? If so you may want to set a significantly higher --transfers
Maybe it would be a good idea to give us a copypaste of how the transfer looks currently when you use -progress and that might give us some clues.
It would also be a good idea to share your config file (redact passwords and secrets first) and tell us what command you are running.
Hello @thestigma.
I usually upload big files (with 5gb or 10gb) (Bacula Volumes)
The commando that I'm using is:
rclone copyto /backup/Volume-VM-0023 wasabi:backup/Volume-VM-0024 --stats=1000m -v
The config is:
[wasabi]
type = s3
env_auth = false
access_key_id = XXXXXXXXXXXXXXXXXXXXXXXX
secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXX
region = us-east-1
endpoint = s3.wasabisys.com
location_constraint =
acl = private
server_side_encryption =
storage_class =
comment = email@domain.com.br
A file with 10mb output command:
rclone copyto file_10mb.txt wasabi:wanderlei/file_10mb.txt -v 2019/08/27 14:51:05 INFO : S3 bucket wanderlei: Bucket "bucket_name" created with ACL "private"
2019/08/27 14:51:26 INFO : file_10mb.txt: Copied (new)
2019/08/27 14:51:26 INFO :
Transferred: 10M / 10 MBytes, 100%, 486.687 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 21s
Ok, and you don't do anything special in your mount command? (any extra flags ect)
I'm not super familiar with S3, but this definitely looks abnormal.
Could you please repeat the test but use -vv so we can get full debug output?
You may also want to use --log-file mylogfilename.txt
I'm kind of suspecting the debug-level log will reveal a lot of lot of low-level error responses from the server, but that's what we need to check out.
Following my log file
2019/08/27 16:04:43 DEBUG : rclone: Version "v1.49.0" starting with parameters ["C:\\Dados\\rclone\\rclone.exe" "copyto" "C:\\Users\\wanderlei\\Desktop\\file_10mb.txt" "wasabi:wanderlei/file_10mb.txt" "-vv" "--log-file" ".\\mylogfilename.txt"]
2019/08/27 16:04:43 DEBUG : Using config file from "C:\\Dados\\rclone\\rclone.conf"
2019/08/27 16:04:45 DEBUG : file_10mb.txt: Couldn't find file - need to transfer
2019/08/27 16:04:45 INFO : S3 bucket wanderlei: Bucket "wanderlei" created with ACL "private"
2019/08/27 16:05:36 DEBUG : file_10mb.txt: MD5 = f1c9645dbc14efddc7d8a322685f26eb OK
2019/08/27 16:05:36 INFO : file_10mb.txt: Copied (new)
2019/08/27 16:05:36 INFO :
Transferred: 10M / 10 MBytes, 100%, 197.295 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 51.9s
2019/08/27 16:05:36 DEBUG : 5 go routines active
2019/08/27 16:05:36 DEBUG : rclone: Version "v1.49.0" finishing with parameters ["C:\\Dados\\rclone\\rclone.exe" "copyto" "C:\\Users\\wanderlei\\Desktop\\file_10mb.txt" "wasabi:wanderlei/file_10mb.txt" "-vv" "--log-file" ".\\mylogfilename.txt"]
I've opened a ticket on Wasabi and it looks that the problem is there.
"I looked into your account logs and I can confirm that we are experiencing some load-related ingest congestion that is affecting throughput for some of our customers. We are taking steps this week to this address this condition and we will update you once we have made some progress in resolving the problem."
Well your log does indicate no obvious problems, so I was a little stuck on what the problem was here - except possible just some bad wifi-connection or something. Given that support message I'd say it's worth giving this a little time to hopefully resolve itself before putting more time into troubleshooting it.
Let's hope it's that easy 