Huge memory usage (>10GB) when upload a single large file (16GB) in WebDAV

teadrinker2015 · December 6, 2023, 2:17pm

What is the problem you are having with rclone?

Huge memory usage (>10GB) when upload a single large file (16GB) in WebDAV backend (Aliyun+Alist). Crash out of memory.
I tried --cache-db-purge and wait for a week for the cache to be expired but still didn't work.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0

os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
os/kernel: 10.0.19045.3693 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.21.4
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Aliyun+Alist (tool for translating aliyun API to WebDAV)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone mount ali:/ J: --no-modtime --vfs-cache-mode full --vfs-cache-max-age 100h --vfs-cache-max-size 1G --vfs-disk-space-total-size 6T

Previously I was using: (I changed cache-size from 16G to 1G as above, neither could work)

rclone mount ali:/ J: --no-modtime --vfs-cache-mode full --vfs-cache-max-age 100h --vfs-cache-max-size 16G --vfs-disk-space-total-size 6T

The rclone config contents with secrets removed.

[ali]
type = webdav
url = http://0.0.0.0:5244/dav/ali
vendor = sharepoint-ntlm
user = admin
pass = *** ENCRYPTED ***

A log from the command with the `-vv` flag

rclone-out-of-memory.log

teadrinker2015 · December 6, 2023, 2:22pm

I forgot to mention: After encountered this problem, I upgraded rclone from 1.64 to 1.65, and result in the same behaviour.

asdffdsa · December 6, 2023, 2:48pm

welcome to the forum,

not sure the exact reason for the high memory usage.
i use mount+webdav, windows and linux, never had issues with high memory usage.

that flag does nothing, as it was used in the deprecated cache remote.

that is for the vfs file cache which is stored at vfs cache: root is "C:\\Users\\tom\\AppData\\Local\\rclone"
so that has nothing to do with memory.

rclone mount has two caches, vfs dir cache uses memory, vfs file cache uses local storage.

check out my summary of the two caches:
https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34

ncw · December 6, 2023, 3:00pm

Could it have been this tool which was using all the memory? Did you check the process list to see if rclone was using lots of memory or not?

teadrinker2015 · December 6, 2023, 3:19pm

Does that mean the memory is only subject to the complexity of directory structure?

Actually I have a crypt remote inside the root folder which is related to this problem. (This problem happened originally inside the crypt one. I moved it to uncrypt remote later to simplify the question.) I belive the uncrypt remote (we are talking about) has the same amount of files as the crypt one, acording to the crypting method.

I made a test, copying 4G file into the remote (no matter crypt or not), and observed the same symptom. The memory is correlated with the file size. (4~5G)

teadrinker2015 · December 6, 2023, 3:23pm

I'm uploading a single file of 4G to a crypt remote, which is using the mentioned remote as a backend.

asdffdsa · December 6, 2023, 3:50pm

let's keep this simple. test without crypt remote, ok?

yes, that is how i understand it.

i am not able to replicate, uploading a 5GiB file to webdav.

teadrinker2015 · December 6, 2023, 4:12pm

The problem is I cannot make the simpler uncrypted remote pass the first minute because of crash out of memory:(
Is there any command except --cache-db-purge that can completely reset the cache? I have already given up my files in that cache.

I'm trying to borrow an another windows laptop (I only have a macbook around here), and another WebDAV backend server to replicate this. It may take a long time to reply, appologize.

asdffdsa · December 6, 2023, 4:25pm

no.
tho not recommended, in this case, manually delete the file from the cache.

ncw · December 6, 2023, 6:25pm

Just wondering why you are using this vendor?

I think it is causing the problem. When I look at your log I see

github.com/Azure/go-ntlmssp.Negotiator.RoundTrip({{0x27ff220?, 0xc0006ce230?}}, 0xc0009d6000)
	github.com/Azure/go-ntlmssp@v0.0.0-20221128193559-754e69321358/negotiator.go:52 +0x2bd fp=0xc0007aac68 sp=0xc0007aa990 pc=0x191645d
github.com/Azure/go-ntlmssp.(*Negotiator).RoundTrip(0xc0007aad48?, 0x418f48?)
	<autogenerated>:1 +0x31 fp=0xc0007aac90 sp=0xc0007aac68 pc=0x1918a31

So it looks very muchlike the go-ntlmssp is keeping the body of the file in memory.

If I look at the code I see

github.com

Azure/go-ntlmssp/blob/754e69321358ada85ce213a4ec971d3e4d1bfdf7/negotiator.go#L49-L59


      
          	// Save request body
          	body := bytes.Buffer{}
          	if req.Body != nil {
          		_, err = body.ReadFrom(req.Body)
          		if err != nil {
          			return nil, err
          		}
          
          		req.Body.Close()
          		req.Body = ioutil.NopCloser(bytes.NewReader(body.Bytes()))
          	}

Which is definitely storing the entire body in memory and explains your out of memory.

So in summary

this looks like a bug in github.com/Azure/go-ntlmss
why did you choose that vendor? Does vendor = other work?

I made an upstream issue here: ntlmssp.Negotiator is storing the entire body in memory and causing out of memory errors · Issue #44 · Azure/go-ntlmssp · GitHub

teadrinker2015 · December 6, 2023, 6:56pm

Confirmed. Changing vendor from sharepoint-ntlm to other indeed solved my problem. Thanks so much for your support.

That is weird. I remember I did select other for the vendor during config. And when I'm struggling with my (weak) windows NAS with rclone 1.65 installed to reproduce, the config shows other correctly... Anyway, its maybe an old version's issue.

ncw · December 7, 2023, 10:49am

Great

I'll follow up the bug with the go-ntlmssp library which will help with the rclone users who are using this library.

system · December 10, 2023, 10:50am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.