Rclone pratically not serving from the cache

Help and Support
1

Hey folks,

What is the problem you are having with rclone?

Files not getting downloaded from the cache.
I'm mounting my remote S3 Storage using rclone mount like this:

rclone mount cdn:media /srv/webdav/ \
--use-server-modtime \
--no-modtime \
--read-only \
--umask 0000 \
--buffer-size 32M \
--dir-cache-time 43200s \
--write-back-cache \
--vfs-cache-max-age 43200s \
--vfs-cache-mode full \
--vfs-read-ahead 50M \
--vfs-read-chunk-size 50M \
--cache-dir /cache/media \
--max-read-ahead 512Ki \
--transfers 100 \
--checkers 100 \
--drive-chunk-size 128M \
--volname media \
--daemon

I also see that after I downloaded some files, these are getting stored under /cache/media. Nice!
But each time when I download the file again it gets served from the remote source, not from the cache... Why that?
The backend I'm using here is S3.

What is the problem you are having with rclone?

rclone v1.59.1

  • os/version: debian 11.4 (64 bit)
  • os/kernel: 5.4.0-124-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.5
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (e.g. Google Drive)

S3 Storage (Minio - StorJ Gateway-ST on same host)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

my testing cmd: rclone copy --progress --transfers 100 cdn:media /tmp
-> Rclone speeds itself looking fine, but the performance I get when sending the files trough NGINX is just horrible, thats why I wanted to use the cache. Using rclone directly gives me about 40 MB/s. As soon as I send the files trough nginx, I get around 1-2 MB/s

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1

  • os/version: debian 11.4 (64 bit)
  • os/kernel: 5.4.0-124-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.5
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

StorJ Gateway-ST, basically minio

The command you were trying to run (eg rclone copy /tmp remote:tmp)

No direct rclone cmd related

The rclone config contents with secrets removed.

[cdn]
type = s3
provider = Other
access_key_id = %hidden%
secret_access_key = %hidden%
region = minio
endpoint = http://127.0.0.1:7777
acl = private
chunk_size = 64M

A log from the command with the -vv flag

Nothing necessary here.

Can someone give me advice onto this behavior?

Thanks in advance

2

You missed all the bits in the help and support template so it's hard to answer that question.

That's now how the cache works based on the flags you've shared. So you'd want to share all the details so we can help out.

1 Like
3

@Animosity022 Just found the Issue. In nginx config make sure you set tcp_nodelay off;
Not sure why it is like that but the performance is now as wished. But im still unsure if the cache is in use or not. need to check again

4

NGinx really has nothing to do with rclone and how it serves from the cache.

Again, you'd have to share a log of what you mean by utilizing the help and support template.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.