I have also tried the profiler. when looking at top it start at 6GB memeory and then slowly go down to about 1,5Gb when the rclone is OOM killed. I am not sure how to read the bellow profile. If anyone can have a look and see if anything is out of the ordinary.
rclone sync --rc --no-update-modtime --checksum --dry-run --checkers=6 COS_US_SOUTH:ibm-docs-dev/POWER10 COS_US_SOUTH:ibm-docs-stage/POWER10 -v &
pod memory requests 2G limit 5GB
jenkins@jenkins-agent-9zb66:~/agent$ go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /home/jenkins/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.010.pb.gz
File: rclone
Type: inuse_space
Time: Oct 31, 2022 at 2:44pm (UTC)
Showing nodes accounting for 1558.08MB, 98.23% of 1586.19MB total
Dropped 88 nodes (cum <= 7.93MB)
flat flat% sum% cum cum%
1040.66MB 65.61% 65.61% 1481.18MB 93.38% github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.XMLToStruct
296.01MB 18.66% 84.27% 296.01MB 18.66% github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.(*XMLNode).findNamespaces (inline)
53.50MB 3.37% 87.64% 129.51MB 8.16% encoding/xml.(*Decoder).Token
53.01MB 3.34% 90.98% 53.01MB 3.34% github.com/rclone/rclone/backend/s3.(*Fs).newObjectWithInfo
39MB 2.46% 93.44% 76MB 4.79% encoding/xml.(*Decoder).rawToken
37MB 2.33% 95.78% 37MB 2.33% encoding/xml.(*Decoder).name
15MB 0.95% 96.72% 15MB 0.95% encoding/xml.CharData.Copy (inline)
12MB 0.76% 97.48% 12MB 0.76% reflect.(*structType).Field
11.40MB 0.72% 98.20% 64.41MB 4.06% github.com/rclone/rclone/backend/s3.(*Fs).listDir.func1
0.50MB 0.032% 98.23% 1579.10MB 99.55% github.com/rclone/rclone/backend/s3.(*Fs).list
0 0% 98.23% 37MB 2.33% encoding/xml.(*Decoder).nsname
0 0% 98.23% 1509.69MB 95.18% github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run
0 0% 98.23% 1509.69MB 95.18% github.com/aws/aws-sdk-go/aws/request.(*Request).Send
0 0% 98.23% 1509.19MB 95.15% github.com/aws/aws-sdk-go/aws/request.(*Request).sendRequest
0 0% 98.23% 1509.19MB 95.15% github.com/aws/aws-sdk-go/private/protocol/restxml.Unmarshal
0 0% 98.23% 1508.18MB 95.08% github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.UnmarshalXML
0 0% 98.23% 27.01MB 1.70% github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.parse
0 0% 98.23% 27.01MB 1.70% github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.parseList
0 0% 98.23% 27.01MB 1.70% github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.parseStruct
0 0% 98.23% 1509.69MB 95.18% github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsWithContext
0 0% 98.23% 1579.10MB 99.55% github.com/rclone/rclone/backend/s3.(*Fs).List
0 0% 98.23% 53.01MB 3.34% github.com/rclone/rclone/backend/s3.(*Fs).itemToDirEntry
0 0% 98.23% 1509.69MB 95.18% github.com/rclone/rclone/backend/s3.(*Fs).list.func1
0 0% 98.23% 1579.10MB 99.55% github.com/rclone/rclone/backend/s3.(*Fs).listDir
0 0% 98.23% 1509.69MB 95.18% github.com/rclone/rclone/fs.pacerInvoker
0 0% 98.23% 1579.10MB 99.55% github.com/rclone/rclone/fs/list.DirSorted
0 0% 98.23% 1579.10MB 99.55% github.com/rclone/rclone/fs/march.(*March).makeListDir.func1
0 0% 98.23% 768.83MB 48.47% github.com/rclone/rclone/fs/march.(*March).processJob.func1
0 0% 98.23% 810.27MB 51.08% github.com/rclone/rclone/fs/march.(*March).processJob.func2
0 0% 98.23% 1509.69MB 95.18% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 98.23% 1509.69MB 95.18% github.com/rclone/rclone/lib/pacer.(*Pacer).call
jenkins@jenkins-agent-9zb66:~/agent$ go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
http://localhost:5572/debug/pprof/heap: Get "http://localhost:5572/debug/pprof/heap": read tcp 127.0.0.1:52990->127.0.0.1:5572: read: connection reset by peer
failed to fetch any source profiles
[1]+ Killed rclone sync --rc --no-update-modtime --checksum --dry-run --checkers=6 COS_US_SOUTH:ibm-docs-dev/POWER10 COS_US_SOUTH:ibm-docs-stage/POWER10 -v