Rclone mount (Wasabi) delayed loading of files

[CP_SLC (Chirag Patel)]

Out of our 1GB terabyte nvme drive, recently we ran into a problem we're almost 800 GB of the files were being used by our screenshots, since the storage space was going to go beyond our current capacity; we decided we will move it to the cloud.

We had two options, 1) S3 or 2) Wasabi. We choose Wasabi due to ofcourse much lower pricing point.

Currently we are running into an interesting problem, whenever the screenshots are uploaded from our system, write seems to be super fast. Almost instantaneously. Which we are really happy about. However when our system tries to read the file back, it seems to be taking more than a minute each time.

I'm hoping someone can help us troubleshoot this a little bit more maybe they are using a wrong Mount command, but I'm not 100% sure.

Other options that we are exploring is Oracle object storage or perhaps backBlaze.

What is the problem you are having with rclone?

rclone mount seems to load files really slow.

What is your rclone version (output from rclone version)

rclone v1.56.2

Which cloud storage system are you using? (eg Google Drive)

Wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --allow-other --buffer-size 10G --fast-list --vfs-cache-mode writes --vfs-write-back 1s wasabi:bucket-name screenshots/ --daemon

The rclone config contents with secrets removed.

[wasabi]
type = s3
provider = Wasabi
access_key_id = 
secret_access_key = 
region = us-east-1
endpoint = s3.wasabisys.com
acl = public-read
chunk_size = 50Mi
disable_checksum = true
upload_concurrency = 20

A log from the command with the -vv flag

Not collecting logs atm.

[CP_SLC (Chirag Patel)]

Luckily I heard back from Wasabi very fast turns out, we are hitting the max GET request quota. I suspect this is because we are checking if the file exists first and then if not, then we are uploading them. Or if it exists, downloading it from the cloud to local to be served.

Is there anything we can do this make rclone mount minimize the number of requests?

[asdffdsa (jojothehumanmonkey)]

hi,

• rclone does not download files on it own. something else is doing that.
  have you figured out what is downloading so many files per second to trigger the quota response from wasabi?

• how many files are being uploaded/downloaded per second?

• is there a hard quota limit or perhaps you mean pacer messages? and yes, they do have great tech support.

• to understand what is going on, need to see the exact response from wasabi.
  --- enable wasabi bucket logging
  --- use a rclone debug log.

• what is the use-case for the rclone mount instead of rclone sync

• have you tried any of these?
  https://rclone.org/s3/#avoiding-head-requests-to-read-the-modification-time