Nfsmount uses enormous amounts of memory

Nasado · January 24, 2025, 5:23am

What is the problem you are having with rclone?

rclone nfsmount crashes after accessing a number of files very quickly. According to the system logs, it's killed by the kernel for trying to allocate a little under 72 GiB of memory.

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.0

os/version: darwin 15.2 (64 bit)
os/kernel: 24.2.0 (arm64)
os/type: darwin
os/arch: arm64 (ARMv8 compatible)
go/version: go1.23.4
go/linking: dynamic
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

None; see my previous (resolved as user error) post for a description of what I'm doing.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

clone --config rclone.conf nfsmount --vfs-cache-mode writes --cache-dir .rclone-vfs-cache Mount: マウント </dev/null

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[Data]
type = union
upstreams = Local:/Users/nasado/収集/物置き/Hot/001 Local:/Users/nasado/収集/物置き/Hot/002
create_policy = mfs

[Local]
type = local
case_sensitive = true

[Mount]
type = combine
upstreams = 一般=Data:General 私的=Private-Data:

[Private-Data]
type = crypt
remote = Data:Private
password = XXX

A log from the command that you were trying to run with the `-vv` flag

As I mentioned in my previous post, rclone nfsmount -vv emits megabytes of log per second. For what it's worth, here's the tail end:

2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: Open: 
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: OpenFile: flags=O_RDONLY, perm=----------
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: Open: flags=O_RDONLY
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: >Open: fd=一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3 (r), err=<nil>
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: >OpenFile: fd=一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3 (r), err=<nil>
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: >Open: 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3 (r), err=<nil>
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.openRange at 0 length 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ReadFileHandle.seek from 0 to 6750208 (fs.RangeSeeker)
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.RangeSeek from 0 to 6750208 length -1
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.Read at -1 length 4096 chunkOffset 6750208 chunkSize 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.openRange at 6750208 length 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.Read at 6754304 length 8192 chunkOffset 6750208 chunkSize 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.Read at 6762496 length 16384 chunkOffset 6750208 chunkSize 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.Read at 6778880 length 32768 chunkOffset 6750208 chunkSize 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.Read at 8810496 length 1048576 chunkOffset 6717440 chunkSize 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.Read at 10874880 length 1048576 chunkOffset 6684672 chunkSize 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.Read at 6811648 length 65536 chunkOffset 6750208 chunkSize 134217728
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: Lstat: 
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: >Lstat: fi=一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3, err=<nil>
2025/01/23 23:17:34 DEBUG : nfs: request: RPC #749985323 (nfs.ReadDirPlus)
2025/01/23 23:17:34 DEBUG : nfs: FromHandle: handle=DA73F60A3B6F4354980F24CCD0797C6B
2025/01/23 23:17:34 DEBUG : 一般/objects/1f/V7/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3/BLAKE2B256E-s24258351--718588a5c1e534e1870d1d80e6346892f146854795c87a4e0d8b4dcfdbe323d6.mp3: ChunkedReader.Read at 6877184 length 131072 chunkOffset 6750208 chunkSize 134217728
2025/01/23 23:17:34 DEBUG : nfs: >FromHandle: path=["一般" "objects" "02" "2K" "BLAKE2B256E-s59542--ce7ff277e9dbcd71c8f8db9ed540ab2938de3cc637866691e887a60040e5482e.wem"], err=<nil>
2025/01/23 23:17:34 DEBUG : nfs: FromHandle: handle=DA73F60A3B6F4354980F24CCD0797C6B
2025/01/23 23:17:34 DEBUG : nfs: >FromHandle: path=["一般" "objects" "02" "2K" "BLAKE2B256E-s59542--ce7ff277e9dbcd71c8f8db9ed540ab2938de3cc637866691e887a60040e5482e.wem"], err=<nil>
2025/01/23 23:17:34 DEBUG : 一般/objects/02/2K/BLAKE2B256E-s59542--ce7ff277e9dbcd71c8f8db9ed540ab2938de3cc637866691e887a60040e5482e.wem: ReadDir: 
2025/01/23 23:17:34 DEBUG : 一般/objects/02/2K/BLAKE2B256E-s59542--ce7ff277e9dbcd71c8f8db9ed540ab2938de3cc637866691e887a60040e5482e.wem: OpenFile: flags=O_RDONLY, perm=----------
2025/01/23 23:17:34 DEBUG : 一般/objects/02/2K/BLAKE2B256E-s59542--ce7ff277e9dbcd71c8f8db9ed540ab2938de3cc637866691e887a60040e5482e.wem: >OpenFile: fd=一般/objects/02/2K/BLAKE2B256E-s59542--ce7ff277e9dbcd71c8f8db9ed540ab2938de3cc637866691e887a60040e5482e.wem/ (r), err=<nil>
./mount.sh: line 8: 10191 Killed: 9               /opt/homebrew/bin/rclone --config rclone.conf nfsmount -vv --vfs-cache-mode writes --cache-dir .rclone-vfs-cache Mount: マウント < /dev/null

And here are the relevant system log entries (the only ones referencing rclone):

low swap: killing largest compressed process with pid 10191 (rclone) and size 73000 MB

[R6121] DNSServiceCreateConnection STOP PID[10191](rclone)

Nasado · January 24, 2025, 11:09pm

Update: I did the obvious thing and ctrl+F'd the nfsmount docs for "memory". After trying a few things, it turns out the issue is the VFS file buffer; disabling it brings memory usage down to a much more reasonable 80 MB.

system · January 27, 2025, 11:09pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.