What is the problem you are having with rclone?
My requirement is to transfer a directory containing hundreds of millions of files from AWS S3 to Alibaba OSS. After reading Big syncs with millions of files , I realized that transferring everything in one go carries a risk of OOM, so I decided to follow the approach in that post and split the big directory into smaller chunks. Since my requirement is that each chunk should be as balanced as possible, I additionally need the size of every file so that I can aggregate them into balanced chunks.
In the end we chose `copy --dry-run` to produce the listing of the files to be transferred (both path and size). The reason we chose `copy --dry-run` instead of `lsf -R --format ps --files-only` is that we compared the two on the same directory and found `copy --dry-run` to be much faster. Specifically, we set up a source with 30,000 files and an empty destination folder. The test showed that `copy --dry-run` was almost one hundred times faster than `lsf -R`: `lsf -R` took 400s, while `copy --dry-run` took only 4s. Given the huge number of files we have to deal with, a difference of that magnitude made us settle on `copy --dry-run`. Based on the estimate mentioned in other posts that one million files take roughly 1GB of memory, we thought we had enough memory, so at the beginning we also added `--fast-list` to trade memory usage for fewer HTTP requests.
We eventually built a mechanism that takes a large directory and splits it into small, reasonably size-balanced chunks. This worked fine when the file count was between a few million and ten or twenty million. Our container memory limit was initially set to 32G, and `copy --dry-run` used about 20G, which was not a problem.
However, when the scale grew to tens of millions and even beyond one hundred million files, I prepared a container with almost 150G of memory (based on the earlier "1 million files ≈ 1G" estimate), and the container still got OOM-killed. At that point I suspected `--fast-list` was causing the OOM, so I removed that flag and re-ran the scan, but some directories still ended up OOM. One directory that succeeded (70 million files) only used about 5G of memory — the cost was a longer scan time, but overall that was acceptable. Yet another directory with around 200 million files still hit the 150G limit and the container was OOM-killed.
So I have two questions:
-
Why do two different directories (whose file counts are not that different in order of magnitude) show such a huge difference in memory usage under the same configuration?
-
The files are not evenly distributed across the second-level directories under the root, and I don't know in advance how many files live under each prefix. So I cannot simply split by prefix and then run `copy --dry-run` per second-level directory. Is there a better way to accomplish what I need — producing a file listing that includes file sizes?
Run the command 'rclone version' and share the full output of the command.
rclone v1.74.3
- os/version: centos 7.2 (64 bit)
- os/kernel: 6.12.92-122.166.amzn2023.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.26.5
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS S3 and Alibaba OSS
The command you were trying to run (eg rclone copy /tmp remote:tmp)
step1
The container memory limit was initially set to 32G.
With a few million up to ten million files, using the following command:
rclone copy src dst --dry-run --checkers 64 --fast-list --user-json-log --stats 0
Memory usage was around 20G, within a safe range.
step2
With 50 million to 200 million files, using the following command:
rclone copy src dst --dry-run --checkers 64 --fast-list --user-json-log --stats 0
Memory usage exceeded 32G and went over the limit.
step3
I realized the problem might be caused by `--fast-list`, so I removed `--fast-list` and at the same time raised the container memory to 150G.
With 50 million to 200 million files, using the following command:
rclone copy src dst --dry-run --checkers 64 --user-json-log --stats 0
One directory with 70 million files finished the whole listing using only 5GB of memory, while another directory with 200 million files still hit the 150G limit and was OOM-killed.
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[aws-usw2]
type = s3
provider = AWS
access_key_id = xxxxx
secret_access_key = xxxxxx
region = xxxx
location_constraint = xxxxx
[oss]
type = s3
provider = Alibaba
access_key_id = xxxxxxxxxxxx
secret_access_key = xxxxxxxxx
endpoint = xxxxxx
A log from the command that you were trying to run with the -vv flag
I am sorry that I have no log for this problem .I use --user-json-log to create the files list.