Rclone mount of crypt remote on s3 cannot handle large files

What is the problem you are having with rclone?

I am using a crypt remote (using a S3 remote as backend store) as a file storage. I use rclone mount to make files available to my servers. Previously I used this setup with Backblaze S3 as a backend. Now I wanted to move providers from Backblaze to Intercolo (https://intercolo.de). I moved all data from Backblaze to Intercolo using rclone copy. When testing functionality using Intercolo as the backend, I discovered that reading large files from this rclone mount-mounted filesystem does not work properly. Coying such a file using rclone copy does work and I could confirm that the files are intact.

It seems like this setup does not play nice with this provider. I would appreciate any ideas for a workaround or how to narrow down the cause for this.

Run the command 'rclone version' and share the full output of the command.

# rclone --version
rclone v1.70.0-beta.8638.4d38424e6
- os/version: debian 12.10 (64 bit)
- os/kernel: 6.1.0-32-arm64 (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.24.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Intercolo (https://intercolo.de) S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount -vv bucket-crypt: /mnt/debug

...

cat /mnt/debug/[...]/s3-test/1152.file >/dev/null
cat: /mnt/debug/[...]/s3-test/1152.file: Input/output error

The rclone config contents with secrets removed.

[intercolo]
type = s3
provider = Other
access_key_id = X
secret_access_key = X
endpoint = https://de-fra.i3storage.com

[bucket-crypt]
type = crypt
remote = intercolo:BUCKET
filename_encryption = obfuscate
directory_name_encryption = true
password = X

A log from the command with the -vv flag

Please note that the rclone version in the log linked below is v.1.69.1, as this is the version I used originally. I re-tested with the beta as shown above with the same result.

https://gist.github.com/trekkie3k/0f8f9ce351219cb4bda066a61004d189

Thanks everyone for taking an interest!

welcome to the forum,

tl;dr - intercolo claims to support rclone, so should contact them.
and ask them to consider https://rclone.org/sponsor/


best to do testing on intercolo:, not bucket-crypt:


i did some quick testing, using v1.67.0 and v1.69.1
and i can replicate the issue when downloading using rclone copy.
so, not sure need to test rclone mount.

DEBUG : file.ext: Reopening on read failure after offset 3145728 bytes: retry 1/10: unexpected EOF


for a deeper look at the api calls, --dump=headers

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.