Unsure if cause of memory leak or not

Help and Support
1

What is the problem you are having with rclone?

I have been getting random out of memory errors with VirtualAlloc failing to grab more memory when using rclone rc vfs/forget to clear the dir cache after iterating through a lot of dirs. Worried that for some reason forget isn't actually causing rclone to fully invalidate the cache and its causing a memory leak of some sorts

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: Microsoft Windows 10 Pro 2009 (64 bit)
  • os/kernel: 10.0.22000.613 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount "archiveenc:" X: -vvvvvvvv --vfs-cache-mode full --cache-dir Z:\rclone --vfs-read-chunk-size 200M --vfs-cache-max-age 30m --vfs-cache-poll-interval 35m --volname 'The Archive' --attr-timeout 1h --drive-skip-gdocs --drive-skip-shortcuts --dir-cache-time 5h --cache-dir 'Z:\rclone' --rc --rc-web-gui --rc-user=admin --rc-pass=admin --transfers 2 --order-by 'size,ascending'

The rclone config contents with secrets removed.

[archive]
type = drive
scope = drive
token = 

[archiveenc]
type = crypt
remote = archive:Encrypted
filename_encryption = standard
directory_name_encryption = true

A log from the command with the -vv flag

Log is gone now as it was in a used terminal window and I restarted the mount before I had the thought that I've seen that error more than once lately and to make this post
2

hi,
--- really need to see a debug log, not terminal output.
--- best to update to latest stable v1.58.0
--- duplicate flags
--cache-dir Z:\rclone
--cache-dir 'Z:\rclone'

3

Didn't notice the duplicate argument, thanks

4

Go programs use garbage collection so memory leaks are less likely (but still possible!).

Check out this FAQ entry also which has some tips.

How much memory is rclone using? Do you see its memory usage growing over time?

5

It was a good 1GB+ but I recently found out it may be Windows Terminal with the issue rather than rclone itself as I am currently helping to try to debug the issue. Ill keep this posted if anything changes. While I have this open, ever thought about a on disk dir cache? maybe a map file of some sorts that just gets scanned at mount start as a sort of cache and updated as time goes on? If I'm understanding things correctly the dir cache is currently kept only in ram

6

Yes I have that on the list of planned enhancements. It would make a lot of difference to rclone's startup performance and make it use less RAM so would be well worth while.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.