Can `rclone serve s3` handle more than one client?

Help and Support
1

What is the problem you are having with rclone?

The server generated by the rclone serve s3 command appears to be capable of serving only a single client at a time.
To illustrate, when listing a folder with a considerable number of files with rclone ls rclone_s3_server:/my-files, it is not possible to reconnect to the server and run, for example, rclone copy -P doc.pdf rclone_s3_server:/my-files until the previous command is completed.

Run the command 'rclone version' and share the full output of the command.

rclone 1.68.1

  • os/version: gentoo 2.15 (64 bit)
  • os/kernel: 6.1.67-gentoo-whatbox (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.1
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3 + local filesystem

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone serve s3 --auth-key XXX --auth-proxy XXX --vfs-cache-mode writes --cache-dir cache --vfs-write-back 10s --vfs-cache-max-age 10m --vfs-cache-max-size 1000G --dir-cache-time 100s --server-read-timeout 4h --server-write-timeout 8h --transfers 36 --addr 127.0.0.1:8080 -v

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[rclone_s3_server]
type = s3
provider = Rclone
access_key_id = XXX
secret_access_key = XXX
endpoint = XXX
acl = private

A log from the command that you were trying to run with the -vv flag

It just gets stuck indefinitely.
2

hi,

  1. "not possible to reconnect to the server ... until the previous command is completed."
  2. "It just gets stuck indefinitely."

those seem to be two different outcomes??

3

My previous statement was not sufficiently clear.
What I intended to convey was that any subsequent action must await the completion of the previous one.
It looks like the operation is single-threaded.

4

It isn't supposed to be single threaded.

Probably there is a lock taken on the root directory while it is listing which is causing the problem.

Does that fit the symptoms?

5

I'm not sure. That might be it. I did notice, though, that I can complete uploads that have already been started.

6

I couldn't find any mention of a lock in cmd/serve/s3.
It looks like there are mutexes only in the code of the gofakes3/s3mem backend, which Rclone doesn't seem to use.

7

There are mutexes in the VFS layer which are probably causing this. Likely the directory lock is held.

8

Even unauthenticated user requests are locked up until the previous action is finished. I'm not sure if this is related to the auth-proxy directive, which I use to create backends on the fly.

9

I don't think there is locking in the Auth proxy itself, but maybe how it is called.

If you can get a go routine dump at the point it is locking the Auth calls that will show what is going on. Easiest way is to kill the S3 serve with SIGQUIT or use the RC (see the RC docs)

10

You're spot on.
The issue resides in gofakes3.go:AddAuthKeys().

11

We've narrowed down which lock which is great.

Looking at the gofakes code this lock is being held throughout the http transaction which is wrong and causing the problem as the lock is held through the very long directory listing.

I've had a go at fixing this - can you give this a try please

v1.69.0-beta.8379.34749b86e.fix-serve-s3-locking on branch fix-serve-s3-locking (uploaded in 15-30 mins)

12

This fixes it. Thanks!

13

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.69

14

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.