I’m encountering some pretty high memory usage when performing a sync from the local filesystem to a remote encrypted fs (which itself wraps an azure blob storage account). I suspect this may be because I have quite a lot of smallish files to sync. Key info:
Command being run: rclone --config /path/to/my/version/controlled/rclone.conf --checksum --retries 20 --suffix 2018-10-04 --backup-dir azure-encrypted:/backups.history/ --quiet --bwlimit 384k sync /home/backup-user/backups/ azure-encrypted:/backups
Rclone Version: 1.42
Total size of files being sync’ed: 855GB
Number of files being sync’ed: 1,098,342 (1098342 without locale-specific separators)
The backup is using 5.325GB of physical RAM (12.408GB virtual), which seems pretty extreme. Is it caching the file checksums in memory or something?
Any help/insights would be most appreciated
That does seem like a lot of RAM! A sync shouldn’t really use very much memory at all.
First I’d try with 1.43.1 or even better the latest beta.
Then you could try memory profiling rclone which would at least tell us what is using up the memory.
Just tried the same command with v1.43-136-g87e1efa9-beta and it ballooned up to 10GB in about an hour. Will run it now with the profiler enabled and post the results.
Profiling indicates a lot of leaks in XML related code?
Showing nodes accounting for 2268.21MB, 84.14% of 2695.62MB total
Dropped 117 nodes (cum <= 13.48MB)
Showing top 10 nodes out of 104
flat flat% sum% cum cum%
960.02MB 35.61% 35.61% 960.02MB 35.61% encoding/xml.copyValue
608.13MB 22.56% 58.17% 611.13MB 22.67% github /ncw/rclone/vendor/github.com/Azure/azure-storage-blob-go/2018-03-28/azblob.(*Metadata).UnmarshalXML
168.52MB 6.25% 64.43% 224.02MB 8.31% github /ncw/rclone/backend/azureblob.(*Fs).newObjectWithInfo
117.02MB 4.34% 68.77% 117.02MB 4.34% strings.Join
81MB 3.00% 71.77% 81MB 3.00% github /ncw/rclone/vendor/github.com/rfjakob/eme.tabulateL
79.50MB 2.95% 74.72% 79.50MB 2.95% strings.ToUpper
74.02MB 2.75% 77.47% 74.02MB 2.75% github /ncw/rclone/fs/asyncreader.newBuffer
72MB 2.67% 80.14% 153MB 5.68% github /ncw/rclone/vendor/github.com/rfjakob/eme.Transform
55.50MB 2.06% 82.20% 55.50MB 2.06% encoding/base64.(*Encoding).EncodeToString
52.50MB 1.95% 84.14% 52.50MB 1.95% encoding/base32.(*Encoding).DecodeString
Thanks for doing the profiling - very helpful
That does look like a bug, probably in the azure-storage-blob-go library rather than rclone…
Can you put the above into a new issue on github and I’ll get the azure blob maintainer sandeepkru to take a look at it - thanks. (Put a link to the forum post too please.)