Foritus
(Foritus)
October 6, 2018, 1:45pm
1
Hello!
I’m encountering some pretty high memory usage when performing a sync from the local filesystem to a remote encrypted fs (which itself wraps an azure blob storage account). I suspect this may be because I have quite a lot of smallish files to sync. Key info:
Command being run: rclone --config /path/to/my/version/controlled/rclone.conf --checksum --retries 20 --suffix 2018-10-04 --backup-dir azure-encrypted:/backups.history/ --quiet --bwlimit 384k sync /home/backup-user/backups/ azure-encrypted:/backups
Rclone Version: 1.42
Total size of files being sync’ed: 855GB
Number of files being sync’ed: 1,098,342 (1098342 without locale-specific separators)
The backup is using 5.325GB of physical RAM (12.408GB virtual), which seems pretty extreme. Is it caching the file checksums in memory or something?
Any help/insights would be most appreciated
ncw
(Nick Craig-Wood)
October 6, 2018, 4:50pm
2
That does seem like a lot of RAM! A sync shouldn’t really use very much memory at all.
First I’d try with 1.43.1 or even better the latest beta .
Then you could try memory profiling rclone which would at least tell us what is using up the memory.
Foritus
(Foritus)
October 6, 2018, 6:01pm
3
Just tried the same command with v1.43-136-g87e1efa9-beta and it ballooned up to 10GB in about an hour. Will run it now with the profiler enabled and post the results.
Foritus
(Foritus)
October 6, 2018, 6:35pm
4
Profiling indicates a lot of leaks in XML related code?
(pprof) text
Showing nodes accounting for 2268.21MB, 84.14% of 2695.62MB total
Dropped 117 nodes (cum <= 13.48MB)
Showing top 10 nodes out of 104
flat flat% sum% cum cum%
960.02MB 35.61% 35.61% 960.02MB 35.61% encoding/xml.copyValue
608.13MB 22.56% 58.17% 611.13MB 22.67% github /ncw/rclone/vendor/github.com/Azure/azure-storage-blob-go/2018-03-28/azblob.(*Metadata).UnmarshalXML
168.52MB 6.25% 64.43% 224.02MB 8.31% github /ncw/rclone/backend/azureblob.(*Fs).newObjectWithInfo
117.02MB 4.34% 68.77% 117.02MB 4.34% strings.Join
81MB 3.00% 71.77% 81MB 3.00% github /ncw/rclone/vendor/github.com/rfjakob/eme.tabulateL
79.50MB 2.95% 74.72% 79.50MB 2.95% strings.ToUpper
74.02MB 2.75% 77.47% 74.02MB 2.75% github /ncw/rclone/fs/asyncreader.newBuffer
72MB 2.67% 80.14% 153MB 5.68% github /ncw/rclone/vendor/github.com/rfjakob/eme.Transform
55.50MB 2.06% 82.20% 55.50MB 2.06% encoding/base64.(*Encoding).EncodeToString
52.50MB 1.95% 84.14% 52.50MB 1.95% encoding/base32.(*Encoding).DecodeString
ncw
(Nick Craig-Wood)
October 6, 2018, 10:55pm
5
Thanks for doing the profiling - very helpful
That does look like a bug, probably in the azure-storage-blob-go library rather than rclone…
Can you put the above into a new issue on github and I’ll get the azure blob maintainer sandeepkru to take a look at it - thanks. (Put a link to the forum post too please.)
Foritus
(Foritus)
October 7, 2018, 12:00am
6
1 Like