Hey there. I've been working with the "Combine" backend through rclone mount and I noticed that the root directory of the combine backend is writable. This introduces some issues, especially when combined with vfs caching. If possible, I think it would be wise to make the root of the combine mount read-only.
While writing to the root is strange, the problems become amplified when using a vfs cache mode that involves writing (e.g. writes or full). When a vfs cache is enabled, my log will get flooded with errors as rclone tries to upload the file to the non-existent backend root. This becomes especially a problem when I mount that folder on my Linux machine from macOS, where macOS will decide to place .DS_Store files in that directory, immediately causing log spam.
As near as I can tell, there is no viable solution or workaround for this. --filter-from and --exclude-from seem to be unable to filter .DS_Store files from being uploaded, let alone any files created at the root of the combine remote. The "vfs: add exclusions for vfs write back - fixes #2109" PR would perhaps provide a workaround, but it would probably be best to just make this root read-only or completely ignored outside of listing the mounts inside. Currently if you set --read-only on the combined mount, the sub-backends will also be read-only.
EDIT: More thoughts/feasibility
I'm not sure how feasible it would be to have a true "read-only" structure that has the root folder as non-writable. I'm not super familiar with
rcloneinternals, but mounting with--read-onlyjust makes the Linux fuse mount read-only. Having the fine-grained control that I would perhaps be proposing here seems like it would be messing with filesystem permissions which seems sticky and I'm not surerclonedeals with currently. Perhaps the real solution would be based on the "vfs: add exclusions for vfs write back - fixes #2109" PR. Alternatively, for thecombinebackend the sub-mounts could be discrete fuse mounts, but that might be overly complicated..
Log behavior
When vfs caching is disabled the logs will say:
2024/03/25 04:46:49 ERROR : test: Failed to copy: combine for remote "test": directory not found
2024/03/25 04:46:49 ERROR : test: WriteFileHandle.New Rcat failed: combine for remote "test": directory not found
2024/03/25 04:46:49 ERROR : test: WriteFileHandle.Flush error: combine for remote "test": directory not found
before the file gets deleted.
When vfs caching is enabled the logs will flood with:
2024/03/25 04:26:03 ERROR : test: Failed to copy: combine for remote "test": directory not found
2024/03/25 04:26:03 ERROR : test: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: combine for remote "test": directory not found
2024/03/25 04:26:13 ERROR : test: Failed to copy: combine for remote "test": directory not found
2024/03/25 04:26:13 ERROR : test: vfs cache: failed to upload try #2, will retry in 20s: vfs cache: failed to transfer file from cache to remote: combine for remote "test": directory not found
and I will need to stop the mount process and delete the cache dir.
Version Info:
rclone v1.65.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-101-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.6
- go/linking: static
- go/tags: none
This seems to be half-way between a Feature Request and a Bug Report. I'm not sure what category it belongs in. Currently I can't think of a good way to workaround the log spam issue if I want to enable mount caching.
EDIT/More info:
I created a test file in the root of the combined mount for the above logs with a simple echo "hey" > /combined_mount/test. The error seems to pop up after I do a ls -la /combined_mount
This problem with uploading to the root of a combine mount cropped up in a separate post on these forums, although in a post with a separate topic: