RClone + OCFS2 (OPCC)

Hi,

I need to migrate a filesystem with 20T in RHEL in GFS2 to Oracle Linux in OCFS2 (OPCC).

This filesystem host more than 18 millions litle files and more than 3 millions changed files per day.

I have a application outage less than 60 hours and I am use rsync to do this copy but differencial rsync takes more than 3 days to finish.

RClone support GFS2 to OCFS2 ?
Can RClone is faster than rsync ?

Thank you
Daniel

My first use case for rclone tends to be:

  1. Is the thing I want to connect to a cloud based storage solution that rclone supports?

If yes, I use rclone.

If no, I use any other tool that I've already got experience with. In this case, I'd use rsync.

Thank you for your answer

I am looking for one replace solution for rsync because as I said rsync is not fast enough that I need.

Regards
Daniel

perhaps tweak --checkers=x and --transfers=x
and change x to whatever values works best.

Can you mount these both as file systems? I think you must be able to if you are using rsync on them.

It can as it has a lot more parallelism - you can increase --checkers and --transfers to check many directories at once and transfer many files at once.

rclone doesn't do the differential sync that rsync does - it transfers whole files only, but I don't think you really need that with millions of small files.

Hi Nick,

The "checkers" function are identify the modified files on source (update/new/delete) and copy they directly to target or put they in a queue to be copied by the "transfers" ?

Do you know the percentage of resource consume for each checker and transfer about cpu, memory and IO ? (the question is to understand the impact on source)

Thank you
Daniel

That's right.

If rclone is checking checksums then it can use a lot of CPU depending on the transfer speed.

You'll probably saturate either your disk io or your network before you run out of CPU or RAM though.

Lots of checkers uses many iops which really hurts HDDs.

Lots of transfers will saturate either your disk transfer speed or your network.

You'll have to experiment.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.