Issue Copy Veeam 365 Bucket to a new Bucket

Kevin_F · August 9, 2024, 5:33am

What is the problem you are having with rclone?

Hello,

I am currently trying to copy 3 buckets from our On Prem Scality Artesca environment to a new bucket based on a NetApp Ontap S3.
The buckets contain Veeam O365 backup data.

I was able to copy two buckets with approx. 5 TB each without any problems. The last and largest (approx. 10TB) is driving me to despair.
Only 10 files are ever copied although there are over 10 million files in the bucket. After the 10 files have been copied, the time continues to run but nothing else happens.

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0
- os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
- os/kernel: 10.0.17763.6054 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Scality Artesca & NetApp Ontap S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone.exe copy Exchange:exchange Exchange_New:exchange -P -vv --no-check-certificate

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[Exchange]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = s3.artesca.optadata-gruppe.local

[Exchange_New]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = 172.16.80.128

A log from the command that you were trying to run with the `-vv` flag

2024/08/09 07:29:09 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone.exe" "copy" "Exchange:exchange" "Exchange_New:exchange" "-P" "-vv" "--no-check-certificate"]
2024/08/09 07:29:09 DEBUG : Creating backend with remote "Exchange:exchange"
2024/08/09 07:29:09 DEBUG : Using config file from "C:\\Users\\k.freund\\Desktop\\rclone-v1.67.0-windows-amd64\\rclone.conf"
2024/08/09 07:29:09 DEBUG : Resolving service "s3" region "us-east-1"
2024/08/09 07:29:09 DEBUG : Creating backend with remote "Exchange_New:exchange"
2024/08/09 07:29:09 DEBUG : Resolving service "s3" region "us-east-1"
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/Encryption/BackupKeys.00000000: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/RepositoryLock/cf02e44c375b46d29877ff8e61951a53_fe91a858f27c4aa4860e7fd05e4a7e7a: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/Encryption/BackupKeys.00000000: Unchanged skipping
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CommonInfo/RepositoryConfig.0000001F: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CommonInfo/RepositoryConfig.0000001F: Unchanged skipping
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CommonInfo/Organizations: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CriticalDataBackup/CommonInfo/Organizations: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CriticalDataBackup/CommonInfo/RepositoryConfig.0000001F: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CommonInfo/RestorePoints/4bd72a2f4c0d4a4bb76b830f2e0ff38a: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/RepositoryLock/cf02e44c375b46d29877ff8e61951a53_fe91a858f27c4aa4860e7fd05e4a7e7a: Unchanged skipping
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CriticalDataBackup/Encryption/BackupKeys.00000000: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CommonInfo/Organizations: Unchanged skipping
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CriticalDataBackup/CommonInfo/Organizations: Unchanged skipping
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CriticalDataBackup/CommonInfo/RepositoryConfig.0000001F: Unchanged skipping
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CommonInfo/RestorePoints/4bd72a2f4c0d4a4bb76b830f2e0ff38a: Unchanged skipping
2024/08/09 07:29:10 DEBUG : Veeam/Backup365/Exchange/CriticalDataBackup/Encryption/BackupKeys.00000000: Unchanged skipping
2024/08/09 07:29:11 DEBUG : Veeam/Backup365/Exchange/Organizations/741976add45a45c189dacd48bfd0046d/AccountMailbox: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:11 DEBUG : Veeam/Backup365/Exchange/Organizations/741976add45a45c189dacd48bfd0046d/AccountMailbox: Unchanged skipping
2024/08/09 07:29:11 DEBUG : Veeam/Backup365/Exchange/Organizations/741976add45a45c189dacd48bfd0046d/RestorePointObjects/f76d458edd9c4be8a067d8ac10f3156a: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/08/09 07:29:11 DEBUG : Veeam/Backup365/Exchange/Organizations/741976add45a45c189dacd48bfd0046d/RestorePointObjects/f76d458edd9c4be8a067d8ac10f3156a: Unchanged skipping
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                10 / 10, 100%
Elapsed time:         7.3s

kapitainsky · August 9, 2024, 7:42am

Probably rclone is busy listing this 10 millions files... can take hours.

Run with -vv --dump headers to see in more details what is going on.

Please note that you need about 1GB of RAM per 1 million files. It might improve in the future, see this issue on github:

github.com/rclone/rclone

Excess memory use when syncing millions of files in one directory

opened 08:44AM - 25 Jul 24 UTC

zackees

bug

## Background - Amazon S3 rclone problems I'm trying to backup a datalake wit…h 100 million files at the root. They are mostly small files < 1mb. rclone was simply not designed for this use case and will eat up all available memory and then crash. There was no machine instance that I could throw at it that would fix this issue. Even running locally in a docker instance would eat up all available memory and then crash. All advice in the forums did nothing to help the situation. And a lot of people seem to be running into this. Therefore I wanted to post this here so that anyone searching for this problem can try our solution. ## Solution in a nutshell: PUT YOUR FILES INTO FOLDERS!!!! What's interesting is the behavior: rclone would never start transferring files, it would always sit there saying 0 files transferred, 0 bytes transferred, eat up all available memory before crashing with an OOM. I tried all the suggestions in the forums, reducing the buffering memory, reducing the number of checkers and transfers. Nothing worked. ## Cause & Fix Without looking at the code or doing any profiling, my hypothesis was that rclone scans an all files in a"directory" into RAM before executing on it. This seems true whether not `--fast-scan` is used or not. Obviously, having 100 million files at the root was causing our org a whole bunch of problems anyway and it's been something that I've wanted to fix for a while, so this problem gave me enough reason to go ahead and re-organize our entire datalakes. Since each file is referenced in our database with a datestamp, I was able to write python scripts that would move these files from the root into folders by the service and year-month (for example name.html -> service/2023-04/name.html) This worked extremely well and I was able to now run rclone and have it at least start transferring some files. However, there were still folders with 5+ million files, and eventually ran into the same out-of-memory error. So again, I further re-organized the files in our datalake into service/yrmo/day. And now that seems to have done the trick. rclone now consistently runs under 2GB memory and I've been able to increase the number of transfers and checkers up to 100 each and have 3mb of buffer per transfer. ## Dead ends All the advice about adjust memory buffers and number of transfers is mostly wrong. They will only cut your minimum memory usage by a constant factor, but will do very little to prevent the absolute unbounded memory that rclone uses for extremely large "directories". If you have this same problem, no amount of setting tweaking will work... you MUST re-organize your data into folders or rclone will just run out of memory every single time. If you have too many files at the root, rclone will simply never start transferring anything and just crash. If one of your subdirectories is too big, you'll get a memory pattern that looks like this: ![image](https://github.com/user-attachments/assets/cad7926e-48b7-46f7-be81-543edc896460) ## Recommendations to the Devs of rclone Please serialize your directory scans to disk if you start exceeding a certain threshold of memory or files in the current directory. You can probably just get away with just always doing that since the disk is so much faster than network anyway. I'm currently doing an inventory scan of our datalakes and 50 million files entries is only taking up 12 gb of disk without any fancy compression. I know you are storing a lot more file information, like metadata, so it could easily be double or triple that. But it is simply so much easier and cheaper to allocate disk space to a docker instance than it is to get a machine with much more ram. An additional pain point about an out of memory issue crash is that when the rclone process gets a kill signal, it will **exit 0** making it look like it succeeded. According to this thread https://github.com/rclone/rclone/issues/7966 this is a feature of linux and you must get the exit code from the operating system instead of the return value of the exited rclone process. This is super scary if you are relying on rclone to backup your datalake but in reality, it starts failing because one of your directories has millions of files in it. I know on Digital Ocean it's easy to see that a docker instance has failed, on Render.com however you'll get a "Run Succeeded" and it's not until you look at the run history that you'll see that in fact your instance ran out of memory. I'm not sure about the other hosting providers. Anyway, I'm glad this huge task is finally over with, and we have started syncing up our data for redundancy and backup purposes. So far so good! ![image](https://github.com/user-attachments/assets/5c2b7096-ade2-45d7-993e-4fd0b0f52196)

system · September 8, 2024, 7:42am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.