Rclone mount of AWS S3 in macOS incurring significant costs

Help and Support
1

What is the problem you are having with rclone?

rclone mount command for AWS S3 buckets are incurring significant costs in MacOS with MacFUSE. This never happened in Linux.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

  • os/version: darwin 14.4.1 (64 bit)
  • os/kernel: 23.4.0 (arm64)
  • os/type: darwin
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.22.1
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

AWS - S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount "$remote:$bucket" "$mount_point" \
                    --cache-dir ~/.cache/rclone \
                    --vfs-cache-mode minimal \
                    --vfs-read-chunk-size "300M" \
                    --vfs-read-chunk-size-limit 0 \
                    --dir-cache-time "20s" \
                    --poll-interval "18s" \
                    --daemon \
                    --umask 002 \
                    --log-file ~/.log/rclone.log \
                    --log-level NOTICE

The rclone config contents with secrets removed.

[aws-account]
type = s3
provider = AWS
region = ap-southeast-2
access_key_id=<key-id>
secret_access_key=<access-key>
session_token=<token>

A log from the command with the -vv flag

The log only documents errors especially when token expires

2024/05/17 09:26:18 ERROR : /: Dir.Stat error: ExpiredToken: The provided token has expired.
	status code: 400, request id: <request id>
, host id: <host id>

The cost breakdown for a single day is provided below
image

2

It has nothing to do with rclone. And is not a bug.

Most likely you have program(s) on your system reading a lot of files. In additions you amplify the problem but not using full cache.

Possible culprits on macOS might be Finder generating files' thumbnails, spotlight indexing etc. Can be also some 3rd party stuff. It is impossible to guess.

rclone on macOS or Linux behaves the same. What is different is software you are using.

3

Thank you. Any recommendations on reducing API calls other than --vfs-cache-mode ? Does --fast-list and --checksum help? I have seen that being recommended for sync but not sure if it is helpful in the mount setting.

4

Yes it might help a bit but in your case this is probably not the key issue. If the same mount command on Linux does not generate so many API calls then on macOS it is something else.

Identify what program is reading so much and either not use it or change its settings. For example in Finder you can disable thumbnails generations for the start.

Also maybe do not mount all S3 bucket but only folder you are working on? All depends what it is for.

1 Like
5

change --log-level NOTICE to --log-level DEBUG
the rclone debug log will show which apps are accessing the mount and what is being accessed.

the first item is ListBucket, that could be from using a low value for --dir-cache-time
the lower the value, the more api calls.

does not apply to S3 remotes. the debug log would show that.

might use defaults or reduce the chunk size.

does nothing on a mount

might try --vfs-cache-mode=full, then rclone will cache the downloaded chunks and not have to re-download.

have you considered switching to a provider that does not charge for api calls, does not charge for outbound transfer?
given that you are using session token, might consider wasabi for some or all of your data.

1 Like
6

Unfortunately its for my work and we are locked into AWS at the moment. I will use these suggestions and report back in a few days regarding cost.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.