What is the problem you are having with rclone? Im runing rclone in Cloud in kubernetes in Jenkins agent container. I have a job that sync many files for different products. This jenkins agent pod had 2GB memory limit. One of the products has grown to about 870k files in COS S3. and all of a sudden …

[image] Thorien: So it means that the checkers check severa folders at the same time and fill up the memory with the hughe lists of files? Correct the checkers concurrently checks the folders, and each checker needs to hold the entire folder listing form both remotes in memory to do the compa…

I have tried to get rid of --no-update-modtime --metadata and added --fast-list. Same result it ends after about a minute without no errors in log. I wonder if the process hits some memory limit and is killed by the kubernetes pod.

Hi Andrej, Your observations share some similarities with this thread (depending on the characteristics of your data/folders/remotes): https://forum.rclone.org/t/huge-memory-usage-when-copying-between-s3-like-services/33687 I therefore suggest you try: rclone sync --dry-run --checkers=1 COS_US_S…

Thanks for the lead. I have tried lower checkers=1 and it was running for long time. (It takes 3,5h to check whole product key) so I cancelled it and was looking for the sweet spot. And I saw how with checkers=6 and --fast-list it fills 5,6GB of memory in about 1 minute and half. whole product has…

with rclone sync --no-update-modtime --checksum --dry-run --checkers=5 COS_US_SOUTH:ibm-docs-stage/"$PRODUCT_KEY" COS_US:ibm-docs-prod/"$PRODUCT_KEY" -v I was able to run the job in 5 minutes and it checked 870k files. It used little more then 4GB of memory in the peak but did not crashed. Custom…

Thanks a lot I wonder how much memory will the transfers need. Most of the files are kilobytes in size, few are more then a MB. So I assume the buffer and streaming of transfers=6 wont go over hundreds of MB.

[image] Thorien: with rclone sync --no-update-modtime --checksum --dry-run --checkers=5 COS_US_SOUTH:ibm-docs-stage/"$PRODUCT_KEY" COS_US:ibm-docs-prod/"$PRODUCT_KEY" -v I was able to run the job in 5 minutes and it checked 870k files. It used little more then 4GB of memory in the peak but did…

I have run memory test on the biggest folder in the structure jenkins@jenkins-agent-5ns68:~/agent$ rclone test memory COS_US_SOUTH:ibm-docs-dev/****/p10eai 2022/10/31 14:20:26 NOTICE: 82632 objects took 299458512 bytes, 3624.0 bytes/object 2022/10/31 14:20:26 NOTICE: System memory changed from 4248…

I have also tried the profiler. when looking at top it start at 6GB memeory and then slowly go down to about 1,5Gb when the rclone is OOM killed. I am not sure how to read the bellow profile. If anyone can have a look and see if anything is out of the ordinary. rclone sync --rc --no-update-modtime …

I have assigned little bit more memory to the jenkins agent pod and the rclone has passed but last profile I gathered was this before the rclone finished rclone sync --rc --no-update-modtime --checksum --dry-run --checkers=6 COS_US_SOUTH:ibm-docs-dev/POWER10 COS_US_SOUTH:ibm-docs-stage/POWER10 -v & …

Perfect and tough call for me. The 3.6K per object and the high memory usage (65+18=83%) by the xmlutil in the aws sdk does indeed challenge my view on normal rclone memory usage; but I am not enough expert to say if this is normal for (some of the) S3 backend(s) - or we are looking at a potential …

rclone forum

Rclone OOM killed during sync files check error 137

Help and Support

show post in topic

Home
Categories
Guidelines
Terms of Service
Privacy Policy

Powered by Discourse, best viewed with JavaScript enabled