Rclone is using a huge amount of memory when sync to iDrive

[Animosity022]

Tons of posts about the file objects and memory use:

Folder with millions of files

Hi everyone, I’ve been trying to sync a bucket from S3 to Wasabi with no success. There are some folders with a few hundred files (images, mostly) which do get synced successfully, but one folder in particular with about 6 millions images does not sync a single file even after hours running. I feel I must be doing something wrong or not using all the recommended parameters for that kind of situation. Can anyone please give me a hint or point me to the right direction? I have tried a lot of co…

How to move 2 millions of files?

What is the problem you are having with rclone? I need to move 2.3 Millions of files from Oracle Cloud Gen1 (Swift) to Oracle Cloud Bucket Gen2 (S3 Compatible). The commands Sync, Move and Copy don't start the transfer, how can I do that? What is your rclone version (output from rclone version) Rclone Download v1.51.0 Which OS you are using and how many bits (eg Windows 7, 64 bit) Oracle Linux 7.7 64bit Which cloud storage system are you using? (eg Google Drive) Oracle Cloud Gen 1 (Swift) to…

Long delay when copying millions of tiny objects

Hi all, I am seeing a long idle time between submitting the copy command and start of the actual transfer. Source and Target are two different S3-compatible Cloud Object Storages. There are millions of tiny (few KB) keys with the specified source prefix. The destination bucket also contains millions of objects, but none share the prefix that I am trying to copy. Delay before rclone uploads files, a newbie question looks similar, but my objects are tiny and calculation for the downloaded keys s…