The associated forum post URL from Issues · rclone/rclone · GitHub
What is the problem you are having with rclone?
My minio bucket information: There are a total of 168GB files, approximately 12 million files, each of which is 10240 bytes
Server information for installing rclone: CPU: 12, memory: 16g
But every time I use the sync function, there is a memory overflow(OOM) and my process is killed,
Run the command 'rclone version' and share the full output of the command.
rclone v1.64.0-DEV
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-169-generic (x86_64)
- os/type: linux
- os/arch: amd64
Which cloud storage system are you using? (eg Google Drive)
S3(MINIO)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone sync minio:minio-226/ ceph:test/minio --transfers 50 --use-mmap --checkers 50
I have also used the following parameters before, but none of them were very ideal
--cache-chunk-no-memory
--s3-max-upload-parts
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
rlcone.conf
[minio]
type = s3
provider = Minio
access_key_id = <redacted>
secret_access_key = <redacted>
endpoint = <redacted>
acl = private
[ceph]
type = s3
provider = Ceph
access_key_id = <redacted>
secret_access_key = <redacted>
endpoint = <redacted>
acl = private
A log from the command that you were trying to run with the -vv flag
rclone sync -vv minio:minio-226/ ceph:test/minio --transfers 50 --checkers 50 --use-mmap
2024/01/23 10:58:38 DEBUG : rclone: Version "v1.64.0-DEV" starting with parameters ["rclone" "sync" "-vv" "minio:minio-226/" "ceph:test/minio" "--transfers" "50" "--checkers" "50" "--use-mmap"]
2024/01/23 10:58:38 DEBUG : Creating backend with remote "minio:minio-226/"
2024/01/23 10:58:38 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/23 10:58:38 DEBUG : name = "minio", root = "minio-226/", opt = &s3.Options{Provider:"Minio", EnvAuth:false, AccessKeyID:"", SecretAccessKey:"", Region:"", Endpoint:"", STSEndpoint:"", LocationConstraint:"", ACL:"private", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:5242880, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:false, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2024/01/23 10:58:38 DEBUG : Resolving service "s3" region "us-east-1"
2024/01/23 10:58:38 DEBUG : fs cache: renaming cache item "minio:minio-226/" to be canonical "minio:minio-226"
2024/01/23 10:58:38 DEBUG : Creating backend with remote "ceph:test/minio"
2024/01/23 10:58:38 DEBUG : name = "ceph", root = "test/minio", opt = &s3.Options{Provider:"Ceph", EnvAuth:false, AccessKeyID:"", SecretAccessKey:"", Region:"", Endpoint:"", STSEndpoint:"", LocationConstraint:"", ACL:"private", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:5242880, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:false, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2024/01/23 10:58:38 DEBUG : Resolving service "s3" region "us-east-1"
2024/01/23 11:21:25 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 1m0.3s
dmesg|grep oom
[54994.684774] rclone invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[54994.684797] oom_kill_process.cold+0xb/0x10
[54994.685012] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[54994.685180] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-17.scope,task=gbsync,pid=84278,uid=0
[54994.685224] Out of memory: Killed process 84278 (rclone) total-vm:16583660kB, anon-rss:13995836kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:31232kB oom_score_adj:0