What is the problem you are having with rclone?
"rclone sync" runs out of memory.
I'm trying to sync 2 IBM Cloud CoS buckets with ±40TB of data in 133M files. When I run rclone, it slowly starts eating more and more memory and eventually eats all memory(62GB) available to OS and after that crashes. It is running on VM with 64GB of RAM and rclone is only process except OS processes running on this system. I tried this with --transfers=4 and behaviour was same. See below my current command I use.
What is your rclone version (output from rclone version)
rclone v1.56.0
- os/version: debian 10.10 (64 bit)
- os/kernel: 4.19.0-17-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.5
- go/linking: static
- go/tags: none
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Debian Buster - 64bit - updated.
Which cloud storage system are you using? (eg Google Drive)
S3 - IBM Cloud Object Storage.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone --transfers=64 -u --use-mmap --buffer-size=0 sync cos-ro:cos-ctera-prod cos-rw:cos-ctera-prod-new
The rclone config contents with secrets removed.
[cos-ro]
type = s3
provider = IBMCOS
access_key_id = ...
secret_access_key = ...
endpoint = s3.private.osl01.cloud-object-storage.appdomain.cloud
location_constraint = eu-flex
acl = private
[cos-rw]
type = s3
provider = IBMCOS
access_key_id = ...
secret_access_key = ...
endpoint = s3.private.eu-de.cloud-object-storage.appdomain.cloud
location_constraint = eu-smart
acl = private
A log from the command with the -vv flag
root@rclone:~# rclone --transfers=64 -u --use-mmap --buffer-size=0 -vv sync cos-ro:cos-ctera-prod cos-rw:cos-ctera-prod-new
2021/08/04 09:49:14 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "--transfers=64" "-u" "--use-mmap" "--buffer-size=0" "-vv" "sync" "cos-ro:cos-ctera-prod" "cos-rw:cos-ctera-prod-new"]
2021/08/04 09:49:14 DEBUG : Creating backend with remote "cos-ro:cos-ctera-prod"
2021/08/04 09:49:14 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/08/04 09:49:14 DEBUG : Creating backend with remote "cos-rw:cos-ctera-prod-new"
2021/08/04 09:49:15 DEBUG : blocks/00000011/02230fd7a0070b422039118264a6ecc927a41d90-2: Sizes identical
2021/08/04 09:49:15 DEBUG : blocks/00000011/02230fd7a0070b422039118264a6ecc927a41d90-2: Destination mod time is within 1ns of source and files identical, skipping
2021/08/04 09:49:15 DEBUG : blocks/00000011/037906090d5f085c9a860c7a2b00eb537d755957-3: Sizes identical
2021/08/04 09:49:15 DEBUG : blocks/00000011/037906090d5f085c9a860c7a2b00eb537d755957-3: Destination mod time is within 1ns of source and files identical, skipping
2021/08/04 09:49:15 DEBUG : blocks/00000011/059d3e435b70b29e038e4f442ad6aacd145d11b5-5: Sizes identical
...
It is processing many files and when it runs our of memory it crashes. Is there any command I can use to tell rclone to not use more memory than x?
Thank you for all your help.