Huge range requests for mounted S3 bucket, other issues

What is the problem you are having with rclone?

I'm testing out partial downloads and uploads on a mounted S3 bucket and I'm encountering various issues:

  • Creating a zip file directly in the mounted directory fails without vfs-cache write cache because when zip tests whether the file is seekable by seeking to offset 0, rclone falsely reports success. This then causes zip to fail at a later stage when it is trying to rewrite the local file headers. As a workaround I instead have zip write to a fifo that is piped to a file in the mounted directory. With the workaround the fifo causes the initial seek test to fail, which then stops zip from seeking at later stages.

  • Performing a 1 megabyte read using dd on a large +1GB file results in a GET request for the first 128 megabytes. Instructing dd to skip the first 3 megabytes and then read 1 megabyte on the same file results in a GET request for the first 128 megabytes followed by a GET request for another 128 megabytes 3 megabytes into the file. I have no idea which flags to use to fix this behavior.

  • Seems not possible to wipe the directory cache, no matter what I try. Mounting a bucket and listing its contents always returns the cached result without actually reading anything from remote.

Run the command 'rclone version' and share the full output of the command.

rclone v1.55.1-DEV

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.15.5
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount -vvv --dir-cache-time 0s --daemon --s3-endpoint <endpoint> --s3-access-key-id <id> --s3-secret-access-key <secret> :s3:/<bucket> <folder>

The rclone config contents with secrets removed.

2023/05/28 09:37:20 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one

A log from the command with the -vv flag

2023/05/28 09:07:39 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2023/05/28 09:07:39 DEBUG : rclone: Version "v1.55.1-DEV" starting with parameters ["rclone" "mount" "-vvv" "--dir-cache-time" "0s" "--max-transfer" "1M" "--daemon" "--s3-endpoint" "<endpoint>" "--s3-access-key-id" "<id>" "--s3-secret-access-key" "<secret>" ":s3:/<bucket>" "<folder>"]
2023/05/28 09:07:39 DEBUG : Creating backend with remote ":s3:/<bucket>"
2023/05/28 09:07:39 DEBUG : :s3: detected overridden config - adding "{O0Qc2}" suffix to name
2023/05/28 09:07:39 DEBUG : fs cache: renaming cache item ":s3:/<folder>" to be canonical ":s3{O0Qc2}:<folder>"
2023/05/28 09:07:39 DEBUG : rclone: Version "v1.55.1-DEV" finishing with parameters ["rclone" "mount" "-vvv" "--dir-cache-time" "0s" "--max-transfer" "1M" "--daemon" "--s3-endpoint" "<endpoint>" "--s3-access-key-id" "<id>" "--s3-secret-access-key" "<secret>" ":s3:/<bucket>" "<folder>"]

BTW - you are using very old beta version of rclone (v1.55.1-DEV)... it is now v1.62.2. I would suggest to verify potential issues you experience with the latest rclone version before diving into deeper investigation.

The last one is by design and you can tweak it:

Have a look at rclone mount documentation. For this particular case:

vfs-chunked-reading

your mount command:

rclone mount -vvv --dir-cache-time 0s --daemon --s3-endpoint <endpoint> --s3-access-key-id <id> --s3-secret-access-key <secret> :s3:/<bucket> <folder>

is using default --vfs-cache-mode off and in this mode file can't be opened for both read AND write, you have to use --vfs-cache-mode full

All detail of different caching modes in the doc:

## VFS File Caching

BTW - you are using very old version of rclone (v1.55.1-DEV)... it is now v1.62.2. I would suggest to verify potential issues you experience with the latest rclone version before diving into deeper investigation.

I'm aware, I'm not using CentOS because I want to.

The last one is by design and you can tweak it:

Since I am not using vfs, I was not expecting vfs options to have any effect. I tried --vfs-read-chunk-size 0 which results in a GET for the entire 1 gigabyte test file followed by a second gigabyte GET request starting 3 megabytes in. Using --vfs-read-chunk-size 1M causes more reasonable 1 megabyte requests at expected offsets. I don't know why rclone insists on downloading the first N bytes no matter what. Can this behavior be altered?

your mount command is using default --vfs-cache-mode off and in this mode file can't be opened for both read AND write, you have to use --vfs-cache-mode full.

I am either reading or writing, not both. I can also use rclone rcat to write files directly:

zip - file | rclone rcat <s3 options>

Reading with rclone cat however in my case is not possible as zip does not support reading from stdin nor can piped arguments be seeked:

strace unzip -l <(rclone cat <s3 options>)            
Archive:  /dev/fd/63
stat("/dev/fd/63", {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
open("/dev/fd/63", O_RDONLY)            = 3
lseek(3, 0, SEEK_SET)                   = -1 ESPIPE (Illegal seek)
read(3, "", 0)                          = 0
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /dev/fd/63 or
        /dev/fd/63.zip, and cannot find /dev/fd/63.ZIP, period.

So I have to use rclone mount to read files.

I am intentionally avoiding VFS because it keeps copies on disk which I cannot afford (insufficient space) in this particular application.

You can upgrade regardless of the OS. Just install the latest version and don't use a package manager.

Are you on some very old CentOS? I have CentOS 7 and latest rclone works no problem.

I don't think installing a newer version will help. I can live with rclone downloading an extra meg or two, I have workarounds for uploading files and I can force fetching the bucket contents with rclone lsf. Thanks.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.