uptime: 1 week 2 days
Nick, looks like the version you gave me is stable. The script to clean up the memory and old folders helped also. Thank you very much!
Uploaded so far: images=36238921; size=20.28 TB
I have one question. In log file I see errors like:
Failed to copy: context canceled
corrupted on transfer: MD5 hash differ "d41d8cd98f00b204e9800998ecf8427e" vs "e3d4bc86f501d5e7e108711612b52e23"
vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: corrupted on transfer: MD5 hash differ "d41d8cd98f00b204e9800998ecf8427e" vs "e3d4bc86f501d5e7e108711612b52e23"
Failed to copy: -> github.com/Azure/azure-pipeline-go/pipeline.NewError, github.com/Azure/azure-pipeline-go@v0.2.3/pipeline/error.go:157
Are they important and may corrupt the final files?
2251570 root 20 0 7496M 5999M 12340 S 12.6 18.7 85h23:40 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-
rclone version
rclone v1.54.0-beta.5061.705d00b8b.fix-azureblob-memory
- os/arch: linux/amd64
- go version: go1.15.6
go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.052.pb.gz
File: rclone
Type: inuse_space
Time: Feb 7, 2021 at 2:57pm (EST)
Showing nodes accounting for 4737.06MB, 97.26% of 4870.35MB total
Dropped 158 nodes (cum <= 24.35MB)
flat flat% sum% cum cum%
1679.64MB 34.49% 34.49% 2964.84MB 60.88% context.WithDeadline
1462.63MB 30.03% 64.52% 1462.63MB 30.03% context.(*cancelCtx).Done
1221.59MB 25.08% 89.60% 1254.70MB 25.76% time.AfterFunc
132.44MB 2.72% 92.32% 132.44MB 2.72% time.startTimer
77.51MB 1.59% 93.91% 77.51MB 1.59% github.com/rclone/rclone/vfs.newDir
38.50MB 0.79% 94.70% 65.50MB 1.34% context.WithCancel
30.50MB 0.63% 95.33% 30.50MB 0.63% github.com/rclone/rclone/fs.NewDir (inline)
25.50MB 0.52% 95.85% 25.50MB 0.52% encoding/xml.copyValue
24.51MB 0.5% 96.36% 24.51MB 0.5% github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
19.23MB 0.39% 96.75% 84.74MB 1.74% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
13.50MB 0.28% 97.03% 112.83MB 2.32% time.NewTimer
9MB 0.18% 97.21% 4017.56MB 82.49% github.com/rclone/rclone/backend/azureblob.(*Fs).Put
1MB 0.021% 97.23% 57.51MB 1.18% context.propagateCancel
1MB 0.021% 97.25% 54.18MB 1.11% github.com/rclone/rclone/cmd/mount.(*Dir).Create
0.50MB 0.01% 97.26% 211.34MB 4.34% github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
0 0% 97.26% 241.09MB 4.95% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 97.26% 241.09MB 4.95% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 97.26% 241.09MB 4.95% bazil.org/fuse/fs.(*Server).serve
0 0% 97.26% 2963.84MB 60.85% context.WithTimeout
0 0% 97.26% 29.50MB 0.61% encoding/xml.(*Decoder).Decode (inline)
0 0% 97.26% 29.50MB 0.61% encoding/xml.(*Decoder).DecodeElement
0 0% 97.26% 29.50MB 0.61% encoding/xml.(*Decoder).unmarshal
0 0% 97.26% 29.50MB 0.61% encoding/xml.(*Decoder).unmarshalPath
0 0% 97.26% 29.50MB 0.61% encoding/xml.Unmarshal
0 0% 97.26% 4517.34MB 92.75% github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
0 0% 97.26% 4517.34MB 92.75% github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
0 0% 97.26% 1552.49MB 31.88% github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
0 0% 97.26% 165.34MB 3.39% github.com/Azure/azure-storage-blob-go/azblob.(*copier).close
0 0% 97.26% 96.01MB 1.97% github.com/Azure/azure-storage-blob-go/azblob.(*copier).write
0 0% 97.26% 96.01MB 1.97% github.com/Azure/azure-storage-blob-go/azblob.(*copier).writer
0 0% 97.26% 4157.95MB 85.37% github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
0 0% 97.26% 165.34MB 3.39% github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.CommitBlockList
0 0% 97.26% 96.01MB 1.97% github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock
0 0% 97.26% 99.54MB 2.04% github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
0 0% 97.26% 1521.46MB 31.24% github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
0 0% 97.26% 4517.34MB 92.75% github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
0 0% 97.26% 4517.34MB 92.75% github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
0 0% 97.26% 4517.34MB 92.75% github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
0 0% 97.26% 211.34MB 4.34% github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
0 0% 97.26% 4157.95MB 85.37% github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
0 0% 97.26% 165.34MB 3.39% github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.CommitBlockList
0 0% 97.26% 96.01MB 1.97% github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock
0 0% 97.26% 99.54MB 2.04% github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
0 0% 97.26% 30.03MB 0.62% github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
0 0% 97.26% 1551.49MB 31.86% github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
0 0% 97.26% 117.51MB 2.41% github.com/rclone/rclone/backend/azureblob.(*Fs).List
0 0% 97.26% 121.51MB 2.49% github.com/rclone/rclone/backend/azureblob.(*Fs).list
0 0% 97.26% 99.54MB 2.04% github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
0 0% 97.26% 117.51MB 2.41% github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
0 0% 97.26% 4298.58MB 88.26% github.com/rclone/rclone/backend/azureblob.(*Object).Update
0 0% 97.26% 210.84MB 4.33% github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
0 0% 97.26% 27.01MB 0.55% github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
0 0% 97.26% 4151.85MB 85.25% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
0 0% 97.26% 4157.95MB 85.37% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
0 0% 97.26% 1521.46MB 31.24% github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
0 0% 97.26% 156.75MB 3.22% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
0 0% 97.26% 4401.61MB 90.38% github.com/rclone/rclone/fs.pacerInvoker
0 0% 97.26% 1519.96MB 31.21% github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
0 0% 97.26% 92.01MB 1.89% github.com/rclone/rclone/fs/list.DirSorted
0 0% 97.26% 4278.08MB 87.84% github.com/rclone/rclone/fs/operations.Copy
0 0% 97.26% 4256.46MB 87.40% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 97.26% 142.62MB 2.93% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 97.26% 4401.61MB 90.38% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 97.26% 156.75MB 3.22% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 97.26% 157.75MB 3.24% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 97.26% 156.75MB 3.22% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 97.26% 41.18MB 0.85% github.com/rclone/rclone/vfs.(*File).Open
0 0% 97.26% 2933.46MB 60.23% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 97.26% 3017.46MB 61.96% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 97.26% 2933.46MB 60.23% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 97.26% 2933.46MB 60.23% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 97.26% 1521.46MB 31.24% net/http.(*Client).Do (inline)
0 0% 97.26% 1521.46MB 31.24% net/http.(*Client).do
0 0% 97.26% 1519.96MB 31.21% net/http.(*Client).send
0 0% 97.26% 1519.96MB 31.21% net/http.(*Transport).RoundTrip (inline)
0 0% 97.26% 1519.96MB 31.21% net/http.(*Transport).roundTrip
0 0% 97.26% 112.83MB 2.32% net/http.(*persistConn).roundTrip
0 0% 97.26% 1519.96MB 31.21% net/http.send