Trying to optimize speed and compute power, flood our OCI Object Storage backend and migrate 150TB | 1.5 Billion objects between regions. We have 3 top level folders/prefixes, and a ton of folders and data within those 3 folders. I'm trying to copy/migrate the data to another region (Ashburn to Phoenix). My issue here is I have 1.5 Billion objects. I decided to split the workload up into 3 VMs (each one is an A2.Flex, 56 ocpu (112 cores) with 500Gb Ram on 56 Gbps NIC's), each VM runs against one of the prefixed folders. I don't notice a large amount of my cpu & ram being utilized, backend support is barely seeing my listing operations (which are supposed to finish in approx 7hrs - hopefully).
Based on these specs, what would be the most efficient way I can list and transfer this amount of objects? I see a variety of different post regarding millions of files in single directories, or deeply nested ones. How do we figure out the proper way to do listing without running out of memory, maximize concurrent processes and flood the VM + OCI Object storage service? Any guidance is greatly appreciated!
rclone v1.69.1
os/version: oracle 8.10 (64 bit)
os/kernel: 5.15.0-304.171.4.el8uek.aarch64 (aarch64)
os/type: linux
os/arch: arm64 (ARMv8 compatible)
go/version: go1.24.0
go/linking: static
go/tags: none
OCI - Oracle Cloud Infrastructure
rclone copy /tmp remote:tmp)rclone copy <source>:<source_bucket> <destination>:<destionation_bucket> --transfers=5000 --checkers=5000 --fast-list
[ociashburn]
type = oracleobjectstorage
provider = instance_principal_auth
namespace = XXX
compartment = XXX
region = us-ashburn-1
[ociphoenix]
type = oracleobjectstorage
provider = instance_principal_auth
namespace = XXX
compartment = XXX
region = us-phoenix-1
-vv flagNo logs as of right now, early testing.
You might be interested in trying the latest rclone beta which includes some improvements for such cases:
opened 08:44AM - 25 Jul 24 UTC
bug
## Background - Amazon S3 rclone problems
I'm trying to backup a datalake wit… h 100 million files at the root. They are mostly small files < 1mb.
rclone was simply not designed for this use case and will eat up all available memory and then crash. There was no machine instance that I could throw at it that would fix this issue. Even running locally in a docker instance would eat up all available memory and then crash.
All advice in the forums did nothing to help the situation. And a lot of people seem to be running into this. Therefore I wanted to post this here so that anyone searching for this problem can try our solution.
## Solution in a nutshell: PUT YOUR FILES INTO FOLDERS!!!!
What's interesting is the behavior: rclone would never start transferring files, it would always sit there saying 0 files transferred, 0 bytes transferred, eat up all available memory before crashing with an OOM.
I tried all the suggestions in the forums, reducing the buffering memory, reducing the number of checkers and transfers. Nothing worked.
## Cause & Fix
Without looking at the code or doing any profiling, my hypothesis was that rclone scans an all files in a"directory" into RAM before executing on it. This seems true whether not `--fast-scan` is used or not.
Obviously, having 100 million files at the root was causing our org a whole bunch of problems anyway and it's been something that I've wanted to fix for a while, so this problem gave me enough reason to go ahead and re-organize our entire datalakes.
Since each file is referenced in our database with a datestamp, I was able to write python scripts that would move these files from the root into folders by the service and year-month (for example name.html -> service/2023-04/name.html)
This worked extremely well and I was able to now run rclone and have it at least start transferring some files. However, there were still folders with 5+ million files, and eventually ran into the same out-of-memory error.
So again, I further re-organized the files in our datalake into service/yrmo/day. And now that seems to have done the trick. rclone now consistently runs under 2GB memory and I've been able to increase the number of transfers and checkers up to 100 each and have 3mb of buffer per transfer.
## Dead ends
All the advice about adjust memory buffers and number of transfers is mostly wrong. They will only cut your minimum memory usage by a constant factor, but will do very little to prevent the absolute unbounded memory that rclone uses for extremely large "directories".
If you have this same problem, no amount of setting tweaking will work... you MUST re-organize your data into folders or rclone will just run out of memory every single time. If you have too many files at the root, rclone will simply never start transferring anything and just crash. If one of your subdirectories is too big, you'll get a memory pattern that looks like this:
## Recommendations to the Devs of rclone
Please serialize your directory scans to disk if you start exceeding a certain threshold of memory or files in the current directory. You can probably just get away with just always doing that since the disk is so much faster than network anyway. I'm currently doing an inventory scan of our datalakes and 50 million files entries is only taking up 12 gb of disk without any fancy compression. I know you are storing a lot more file information, like metadata, so it could easily be double or triple that.
But it is simply so much easier and cheaper to allocate disk space to a docker instance than it is to get a machine with much more ram.
An additional pain point about an out of memory issue crash is that when the rclone process gets a kill signal, it will **exit 0** making it look like it succeeded. According to this thread https://github.com/rclone/rclone/issues/7966 this is a feature of linux and you must get the exit code from the operating system instead of the return value of the exited rclone process.
This is super scary if you are relying on rclone to backup your datalake but in reality, it starts failing because one of your directories has millions of files in it. I know on Digital Ocean it's easy to see that a docker instance has failed, on Render.com however you'll get a "Run Succeeded" and it's not until you look at the run history that you'll see that in fact your instance ran out of memory. I'm not sure about the other hosting providers.
Anyway, I'm glad this huge task is finally over with, and we have started syncing up our data for redundancy and backup purposes. So far so good!
2 Likes
ncw
March 6, 2025, 1:26pm
3
How are the files distributed under the 3 prefixes/directorys? Is there more structure underneath?
If you have more that 1000000 files in a directory then you'll like need the beta @kapitainsky linked above. This is due for release in 1.70
If you don't mind spending the extra transactions, then removing --fast-list will probably speed things up and use less memory though I would reduce checkers down a bit to 500 say
2 Likes
I will definitely check out the beta version, looks awesome.
Update here: 4000 transfers, 2000 checkers, fast-list were able to move 10 million objects per hour, should be finished in 6.2 days for full migration. Sitting about 332Gb memory used on each VM after listing and remaining consistent without going OOM.
Folder structure is 3 Top level prefixes, then in each of those is a ton of nested->nested->nested folders with files spread everywhere.
If we hit any issues, will try without -fast-list and bump checkers down.
And another "bleeding edge" feature which might be handy here.
opened 10:46PM - 16 Feb 25 UTC
**Feature Request: Add `--worker` Flag for Partitioned, Hash-Based File Transfer… s**
I'm going to do this myself in my python API to increase throughput. I thought i'd write a feature request for completeness. Feel free to close this feature request if not applicable. I may be able to implement this feature myself in rclone if this is something you are interested in.
**Overview**
I'd like to request a new feature that allows rclone to transfer only a portion of a server's content. This feature would enable users to run multiple rclone instances concurrently, with each instance responsible for a distinct subset of files. The goal is to facilitate distributed transfers and avoid duplicate work when syncing or copying large datasets.
**Proposed Approach**
Introduce a new flag, `--worker`, where the argument is formatted as `worker_id:(n_workers-1)`. For example:
- `rclone copy ... --worker 0:1`
- `rclone copy ... --worker 1:1`
In the above example, two workers are deployed, and each will handle roughly 50% of the files.
**How It Works**
For each file to be transferred, rclone will calculate a hash based on the file's path (e.g., using MD5). Then, using the worker parameters, it determines if the current worker should process the file based on the following pseudocode:
```python
worker_id = [provided worker id]
n_workers = [total number of workers]
for each file in files_to_copy:
md5_hash = md5(file.path)
# The addition of worker_id helps in balancing the distribution
if (md5_hash + worker_id) % n_workers == 0:
transfer(file)
else:
skip(file)
```
--hash-filter - not in the master beta yet but binaries are already generated for hash-filter branch. No need to compile from source.
It would allow easily to split massive task into parallel threads without much worry how to make sure that processed things do not overlap.
Some raw details in this commit:
committed 10:21AM - 26 Feb 25 UTC
2 Likes