What is the problem you are having with rclone?
services cant sync files
this bucket size 12TB，approximately 8000000 files.
After I execute the command, the memory will continue to grow, but the files will not be transferred. Finally, the memory overflows and the process ends.
However, when transferring a single file, it can succeed.
order is : rclone sync -vv ceph:ecm2bucket oss:ecm2bucket --bwlimit 1G > "./sync.log" 2>&1 &
Is it 8M files in one directory?
The way rclone works at the moment is that it will load the whole directory into RAM so you'll need roughly 8GB of RAM for that. (Maybe half that, maybe twice that).
How much RAM have you got?
Thanks for your response.My English is very poor,But I still want to explain.
At first，i set memory to 20GB. but the program didn't work. Then I expanded the memory to 100GB and the program began to run.After calculation, the program consumes about 30GB of memory. Does this have anything to do with the name length of the file?
That sounds about what I expect.
Yes it does, but the length of the file name will only have a small effect on the sizes most likely as there is about 1k of overhead which isn't the file name.