Mounted Crypt Remote filling up windows cache disk and failing on very large transfers

Help and Support
1

What is the problem you are having with rclone?

Is there a way to get rclone mount to work with a crypt remote on windows and NOT fill up the cache disk completely when transferring more data then the local disk can hold?

First Problem: When using rclone mount to a crypt remote to a smb remote and --vfs-cache-mode write its possible to totally fill the the host system, causing writes to fail.

Second Problem: The same, except without --vfs-cache-mode specified, windows file explorer can't copy files to the crypt Remote

Scenario:

A: 10TiB NAS with SMB mount, 1 GbE Network

B: 100GiB Windows computer, 1GbE
C: attached USB Storage device of 1TiB to B

Using rclone copy works without a issue from C to A

Using rclone mount WITH --vfs-cache-mode write AND using windows file explorer to copy the contents from C to A, will work for awhile, totally fill up the disk on B, then the copy will fail with no space left on device. Pausing the transfer manually lets the network catch up and free up cache space to prevent this from happening, but for very large USB disks it would be ideal to be able to run this process unattended

Using rclone mount WITHOUT --vfs-cache-mode specified AND using windows file explorer to copy the contents from C to A will fail immediately with O_TRUNC errors.

I'm trying to create an environment where people who are familiar with windows can self serve and copy their LARGE collection of USB hard drives to a central NAS without requiring them to use the command line tools. Right now I've setup rclone mount as a windows startup task so its a always available network disk for them.

I'm happy for file transfers to go slowly and block while waiting for the local disk cache to empty out to the network, as long as it doesn't require manual intervention from a human.

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.4651 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Synology SMB NAS on local network

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount demo_crypt_remote: y: --file-perms=0777 --vfs-cache-mode write —vfs-cache-max-age 1m --network-mode --rc --no-console

or 

rclone mount demo_crypt_remote: y: --file-perms=0777  --network-mode --rc --no-console

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[demo_crypt_remote]
type = crypt
password = PASSWORD
password2 = SALT
remote = demo_smb:/home/demo_crypt_remote

[demo_smb]
type = smb
user = demo
domain = WORKGROUP
host = demo.localhost
pass = SMBPASSWORD

A log from the command that you were trying to run with the -vv flag


rclone mount demo_crypt_remote: y: --file-perms=0777  -vv --network-mode --rc 
(Windows file copy a file from a USB drive, to the crypt remote)

2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: CreateEx: flags=0x502, mode=0700
2024/07/15 17:14:37 DEBUG : testdor2/RSI-Setup-1.6.10.exe: Open: flags=O_RDWR|O_CREATE|O_EXCL
2024/07/15 17:14:37 DEBUG : testdor2: Added virtual directory entry vAddFile: "RSI-Setup-1.6.10.exe"
2024/07/15 17:14:37 DEBUG : testdor2/RSI-Setup-1.6.10.exe: >Open: fd=testdor2/RSI-Setup-1.6.10.exe (w), err=<nil>
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >CreateEx: errc=0, fh=0x1
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Getattr: fh=0x1
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Getattr: fh=0x1
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Truncate: size=164328496, fh=0x1
2024/07/15 17:14:37 ERROR : testdor2/RSI-Setup-1.6.10.exe: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Truncate: errc=-1
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Getattr: fh=0x1
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Write: ofst=0, fh=0x1
2024/07/15 17:14:37 ERROR : testdor2/RSI-Setup-1.6.10.exe: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Write: n=-1
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Getattr: fh=0x1
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Write: ofst=1048576, fh=0x1
2024/07/15 17:14:37 DEBUG : testdor2/RSI-Setup-1.6.10.exe: waiting for in-sequence write to 1048576 for 1s
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /: Opendir:
2024/07/15 17:14:37 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/07/15 17:14:37 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/07/15 17:14:37 DEBUG : /: >Opendir: errc=0, fh=0x2
2024/07/15 17:14:37 DEBUG : /: Releasedir: fh=0x2
2024/07/15 17:14:37 DEBUG : /: >Releasedir: errc=0
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /: Opendir:
2024/07/15 17:14:37 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/07/15 17:14:37 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/07/15 17:14:37 DEBUG : /: >Opendir: errc=0, fh=0x2
2024/07/15 17:14:37 DEBUG : /: Releasedir: fh=0x2
2024/07/15 17:14:37 DEBUG : /: >Releasedir: errc=0
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /: Opendir:
2024/07/15 17:14:37 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/07/15 17:14:37 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/07/15 17:14:37 DEBUG : /: >Opendir: errc=0, fh=0x2
2024/07/15 17:14:37 DEBUG : /: Releasedir: fh=0x2
2024/07/15 17:14:37 DEBUG : /: >Releasedir: errc=0
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /: Opendir:
2024/07/15 17:14:37 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/07/15 17:14:37 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/07/15 17:14:37 DEBUG : /: >Opendir: errc=0, fh=0x2
2024/07/15 17:14:37 DEBUG : /: Releasedir: fh=0x2
2024/07/15 17:14:37 DEBUG : /: >Releasedir: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /testdor2: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /testdor2: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2: Opendir:
2024/07/15 17:14:37 DEBUG : /testdor2: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/07/15 17:14:37 DEBUG : /testdor2: >OpenFile: fd=testdor2/ (r), err=<nil>
2024/07/15 17:14:37 DEBUG : /testdor2: >Opendir: errc=0, fh=0x2
2024/07/15 17:14:37 DEBUG : /testdor2: Readdir: ofst=0, fh=0x2
2024/07/15 17:14:37 DEBUG : /testdor2: >Readdir: items=2, errc=0
2024/07/15 17:14:37 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:37 DEBUG : /testdor2: >Getattr: errc=0
2024/07/15 17:14:37 DEBUG : /testdor2: Releasedir: fh=0x2
2024/07/15 17:14:37 DEBUG : /testdor2: >Releasedir: errc=0
2024/07/15 17:14:38 DEBUG : /testdor2: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:38 DEBUG : /testdor2: >Getattr: errc=0
2024/07/15 17:14:38 DEBUG : /testdor2: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:38 DEBUG : /testdor2: >Getattr: errc=0
2024/07/15 17:14:38 DEBUG : /testdor2: Opendir:
2024/07/15 17:14:38 DEBUG : /testdor2: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/07/15 17:14:38 DEBUG : /testdor2: >OpenFile: fd=testdor2/ (r), err=<nil>
2024/07/15 17:14:38 DEBUG : /testdor2: >Opendir: errc=0, fh=0x2
2024/07/15 17:14:38 DEBUG : /testdor2: Releasedir: fh=0x2
2024/07/15 17:14:38 DEBUG : /testdor2: >Releasedir: errc=0
2024/07/15 17:14:38 DEBUG : /testdor2: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:38 DEBUG : /testdor2: >Getattr: errc=0
2024/07/15 17:14:38 DEBUG : testdor2/RSI-Setup-1.6.10.exe: aborting in-sequence write wait, off=1048576
2024/07/15 17:14:38 DEBUG : testdor2/RSI-Setup-1.6.10.exe: failed to wait for in-sequence write to 1048576
2024/07/15 17:14:38 ERROR : testdor2/RSI-Setup-1.6.10.exe: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Write: n=-29
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Getattr: errc=0
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Getattr: errc=0
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Getattr: fh=0x1
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >Getattr: errc=0
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: OpenEx: flags=0x0
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/07/15 17:14:38 DEBUG : testdor2/RSI-Setup-1.6.10.exe: Open: flags=O_RDONLY
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: Write: ofst=2097152, fh=0x1
2024/07/15 17:14:38 DEBUG : testdor2/RSI-Setup-1.6.10.exe: waiting for in-sequence write to 2097152 for 1s
2024/07/15 17:14:38 DEBUG : testdor2/RSI-Setup-1.6.10.exe: >Open: fd=testdor2/RSI-Setup-1.6.10.exe (r), err=<nil>
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >OpenFile: fd=testdor2/RSI-Setup-1.6.10.exe (r), err=<nil>
2024/07/15 17:14:38 DEBUG : /testdor2/RSI-Setup-1.6.10.exe: >OpenEx: errc=0, fh=0x2
2024/07/15 17:14:39 DEBUG : testdor2/RSI-Setup-1.6.10.exe: aborting in-sequence write wait, off=2097152
2

welcome to the forum,

might try --vfs-cache-min-free-space but not really a solution

to write files to the mount, need to use --vfs-cache-mode

and might want to check out my howto guide about using smb with rclone
https://forum.rclone.org/t/how-to-access-smb-samba-with-rclone/42754

3

Thank you for the kind welcome, it's good to be here.

That you for that guide, I didn't think about using windows itself for the SMB mount, good idea - I'll test it out!

I did think about using the SMB drive as the cache location for the crypt remote, but not only would it triple the network usage, it would defeat the purpose of using the crypt remote, even if "temporarily"

I suppose I want a way to "block" network operations on the vfs system when its filled up the cache size limit, to let it drain to the NAS.

4

not sure what you mean about self serve?
the end-user would copy the files from usb to the synbox and after that, access the files from the synbox?

about copy their LARGE collection of USB hard drives
might write a simple script to use rclone copy which does not use cache.

5

That could work, but this group of users is very command line phobic, so even a small script would be very brittle, and I don't want to have to remote in each time they switch USB drives.

These are windows users, they understand windows file manager. They know about drives in the sidebar. They can copy and paste their files to the network share without assistance. That is why I'd like to find a paradigm that fits their current workflow and domain knowledge. They have about 60 USB drives scattered around, so there is going to be alot of switching, and while I wish them the best on this journey I don't want to manage it directly for them.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.