`rclone mount` not following `--low-level-retries` and instead returning errors immediately to the application

Suspected Bug
1

What is the problem you are having with rclone?

When creating new files and writing to them on a directory mounted via rclone mount, low-level errors are not leading to retries, and instead are being immediately returned as I/O errors to the program doing the writing.

Expected behavior AFAIK was for the write operations that generated these errors to be retried up to --low-level-retries times (default 3).

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: debian 11.7 (64 bit)
  • os/kernel: 6.1.0-0.deb11.7-cloud-arm64 (aarch64)
  • os/type: linux
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Box, under crypt, under chunker (see below for full config)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount CHUNKER_ENCRYPTED_REMOTE: CHUNKER_ENCRYPTED_REMOTE_MOUNT_POINT --buffer-size 512m --dir-cache-time 5m0s --max-read-ahead 256m --vfs-cache-mode off --vfs-cache-max-size 1g --rc --rc-addr 127.0.0.1:5572

The rclone config contents with secrets removed.

[BASE_REMOTE]
type = box
token = {"access_token":"REDACTED","token_type":"bearer","refresh_token":"REDACTED","expiry":"REDACTED"}

[ENCRYPTED_BASE_REMOTE]
type = crypt
remote = BASE_REMOTE:
password = REDACTED
filename_encoding = base32768

[CHUNKER_ENCRYPTED_REMOTE]
type = chunker
remote = ebox:
chunk_size = 4G
name_format = *.rcc###
hash_type = sha1all

A log from the command with the -vv flag

It's impossible for me to generate a full -vv log for the command as it's very long-running (over 6 days now and counting) and as the errors happen very infrequently (just 2 errors so far in many thousands of files), this would generate gigabytes of logs and I don't have the disk space to store them in the VPS where all of this is running.

Here is what the above rclone mount command prints to its stderr when each of these 2 errors happened:

Error #1:

2023/07/30 08:09:26 ERROR : CHUNKER_ENCRYPTED_REMOTE_MOUNT_POINT/REDACTED1/REDACTED2/REDACTED3/REDACTED4/REDACTED5/REDACTED6: WriteFileHandle.New Rcat failed: Post "https://upload.box.com/api/2.0/files/content": read tcp REDACTED_IP:45290->74.112.186.128:443: read: connection reset by peer
2023/07/30 08:09:26 ERROR : CHUNKER_ENCRYPTED_REMOTE_MOUNT_POINT/REDACTED1/REDACTED2/REDACTED3/REDACTED4/REDACTED5/REDACTED6: WriteFileHandle.Flush error: Post "https://upload.box.com/api/2.0/files/content": read tcp REDACTED_IP:45290->74.112.186.128:443: read: connection reset by peer
2023/07/30 08:09:26 ERROR : IO error: Post "https://upload.box.com/api/2.0/files/content": read tcp REDACTED_IP:45290->74.112.186.128:443: read: connection reset by peer

Error #2:

2023/07/30 13:26:52 ERROR : CHUNKER_ENCRYPTED_REMOTE_MOUNT_POINT/REDACTED1/REDACTED2/REDACTED3/REDACTED4/REDACTED5/REDACTED7: WriteFileHandle.New Rcat failed: Post "https://upload.box.com/api/2.0/files/content": read tcp REDACTED_IP:53482->74.112.186.128:443: read: connection reset by peer
2023/07/30 13:26:52 ERROR : CHUNKER_ENCRYPTED_REMOTE_MOUNT_POINT/REDACTED1/REDACTED2/REDACTED3/REDACTED4/REDACTED5/REDACTED7: WriteFileHandle.Flush error: Post "https://upload.box.com/api/2.0/files/content": read tcp REDACTED_IP:53482->74.112.186.128:443: read: connection reset by peer
2023/07/30 13:26:52 ERROR : IO error: Post "https://upload.box.com/api/2.0/files/content": read tcp REDACTED_IP:53482->74.112.186.128:443: read: connection reset by peer

And here's what the writing program (GNU tar v.1.34, in case it makes a difference) printed in result:

Error #1:

tar: CHUNKER_ENCRYPTED_REMOTE_MOUNT_POINT/REDACTED1/REDACTED2/REDACTED3/REDACTED4/REDACTED5/REDACTED6: Cannot close: Input/output error

Error #2:

tar: CHUNKER_ENCRYPTED_REMOTE_MOUNT_POINT/REDACTED1/REDACTED2/REDACTED3/REDACTED4/REDACTED5/REDACTED7: Cannot close: Input/output error

Please feel free to contact me if you need more info and/or if there's a viable way to generate a more complete log without using a lot of disk space.

Ditto if I should go ahead and create an issue for that on rclone' s github.

2

Rclone can't retry uploads with --vfs-cache-mode off - it sends the data straight to the cloud storage system and doesn't store it.

If you use --vfs-cache-mode writes you'll get retries and in general a much more resilient upload.

Those are the tradeoffs unfortunately.

1 Like
3

Many thanks again @ncw. I did not know --vfs-cache-mode off had an impact on error retrying -- now that you explained it's totally obvious, but before your explanation I would have a hard time figuring that out as it's quite unintuitive.

BTW, is this mentioned in the docs somewhere? At least in the most obvious place (eg, in the part documenting --low-level-retries option) I can't find it. If this indeed isn't in the docs yet, I can add it and send a PR your way.

4

It is on the mount page:

5

Hummmrmrm... just checked it and:

image
image983×372 41.6 KB

AFAICS it fails to mention anything related to --low-level-retries -- it only mentions the case where the entire upload fails (not sure that's the case).

Wouldn't it be better if it was more explicit, and specially also mentioned right next to the documentation for the --low-level-retries option?

6

By all means, submit a PR if you'd like it more clear.

For me, I feel it is but always better to be more explicit:

image

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.