Linux Read/Write Size

What is the problem you are having with rclone?

When running rclone to sync multi-GB data between high performance file systems in Linux (Lustre / GPFS) it seems to only use 32k read/write sizes (verified with strace). Can this be changed (either via parameter or source) to use something much larger (like 1MiB)? I had tried --buffer-size and it doesn't seem to effect the read/write size. I'm not sure this is really a primary use case for rclone, but the tool is generally desirable for what we are doing (users moving large data sets).

Run the command 'rclone version' and share the full output of the command.

Using the rclone rpm in RedHat 8.4.
rclone v1.55.1-DEV

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.14.12
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Local storage (GPFS / Lustre)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --buffer-size 256M  --multi-thread-streams 4 --multi-thread-cutoff 1M /src/path/ /dest/path/

The rclone config contents with secrets removed.

empty

A log from the command with the -vv flag

There are no errors.  It works fine.

That's an older version, but shouldn't matter.

      --max-read-ahead SizeSuffix              The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)

Rclone and linux readahead - Feature - rclone forum

Update MAX_KERNEL_WRITE · Issue #309 · hanwen/go-fuse (github.com)

Some older info here too:

POC: thoughts into code for readaead by sjpotter · Pull Request #3926 · rclone/rclone (github.com)

Can you recheck this with the latest version v1.59 as I think we may have fixed this already?

I built the latest version and it did help some. I actually didn't realize there was a post copy checksum and disabling that was a significant improvement. I also dropped multi-threaded transfers in favor of just increasing --transfers to have more file level parallelism. Thanks!

If you do that you should get speeds the same as cp. Do you?

The test / example data set is 51 directories, 3970 files, and 80GiB in size. The test system has about 1.2 GiB/s of throughput available that I want to keep full since we have several PB to move.

A simple "cp -rp" takes ~4m and an rsync is ~5.5m. Rclone is substantially faster due to parallelism at ~1.3m. I use these arguements:

rclone sync -vv --stats-one-line -P --ignore-checksum --modify-window 1s --transfers 8 --checkers 8 /src/path/ /dest/path/

Using 8 transfers / checkers and ignore-checksum (to prevent the post-copy verify, which for a local copy shouldn't be necessary I'm assuming) seem required to hit a stable 1.2BiG/s. A few samples seem to show the buffer around 64M or more helps a small amount.

We will need to be able to resync data sets as we do an initial sync and then periodically update until we have a window to switch to the new area so it's important that we don't unnecessarily recopy data. One quirk I have seen is that some files seem to trigger a checksum even though the file is unchanged:

2022-08-03 11:46:29 DEBUG : file1234: Modification times differ by -472.475ms: 2016-04-14 10:10:48.472475 -0400 EDT, 2016-04-14 10:10:48 -0400 EDT
2022-08-03 11:46:29 DEBUG : file1234: md5 = aa249ca4f96c22846004a5943c920cfb OK
2022-08-03 11:46:29 DEBUG : file1234: Unchanged skipping

I'm not sure why this occurs. It may have something to do with differing storage technologies on the source and destination. Even if I add "--modify-window 1s" it still sometimes does the seemingly unnecessary read / checksum (not sure if it's doing that to the source or destination).

My thought is I can use rclone to do the bulk transfers and then use a final rsync to sync up permissions, acls, etc.

Great!

Are you using Glusterfs? That is renown for slightly variable modification times.

I don't think it should be doing that. I couldn't repro it myself (using touch to change age of file) can you come up with a simple repro I can try?

Sounds perfect.

Note that with rclone v1.59 you can use the -M flag which will sync permissions but it doesn't know anything about ACLs so rsync might be your best choice there.

PS you can also use rclone check after the transfer to give a nice sense of security that all the data is copied without corruption.