Best flags for local HDD (rotational) cloning?

Help and Support
1

What is the problem you are having with rclone?

I want to know the best settings for copying from one rotational HDD to another rotational HDD without fragmenting the files, or the directories.

Backstory

I reorganized my local Projects folder (terabytes of code (cough node_modules cough), video, and other assets) and now I'm trying to get it back in sync on the backup as efficiently as possible. I've mirrored some of the changes by hand, but the bulk of the work is sadly a matter of deleting existing files and writing new copies.

At first I was using rsync, but it was copying large files at 20/mbps on a drive that gets ~150/mbps reads and writes. That seemed odd, so I tested with cp and saw much faster transfers. rclone seems to also copy large files closer to the expected speed.

HOWEVER

rclone appears to be excessively fragmenting the disk, switching between various files and folders, rather than writing like files at the same time.

2021-02-02 16:01:16 INFO  : Sites/webinstall.dev/home/aj/..../zcallback_windows_arm.s: Copied (new)
2021-02-02 16:01:16 INFO  : Sites/telebit.io/var/.../blkio.throttle.write_bps_device: Copied (new)
2021-02-02 16:01:16 INFO  : Sites/coolaj86.com/root/.../cpu.a: Copied (new)
2021-02-02 16:01:16 INFO  : github.com/coolaj86/.../comp.h: Copied (new)

The copy order appears to be random despite using --order-by name and --transfers 1.

What is your rclone version (output from rclone version)

rclone version
rclone v1.53.4
- os/arch: darwin/amd64
- go version: go1.15.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS Catalina 64-bit

Which cloud storage system are you using? (eg Google Drive)

N/A

HFS+

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -vP \
    --transfers=1 --order-by name \
    --exclude 'node_modules/**' --exclude '.Spotlight-*/**' --exclude '.cache*/**' \
    ~/Projects/ /Volumes/Backup/Projects/

The rclone config contents with secrets removed.

N/A

A log from the command with the -vv flag

N/A
2

My two cents is I would not use rclone as I'd use rsync.

I tend to use rclone for any cloud work as it's a great tool for that.

For your use case, I think rsnapshot (which leveragse rsync) might be something to look as it should be available via brew.

3

rsync is not behaving properly ^^

4

You are comparing an apple to an orange somewhat though.

Rsync does checksums and other things to ensure you are getting a good copy.
A "cp" command does not.

felix@gemini:/tmp$ fallocate -l 1G test.img
felix@gemini:/tmp$ time cp test.img test1.img

real    0m0.606s
user    0m0.000s
sys     0m0.584s


felix@gemini:/tmp$ time rsync test.img test3.img -P
test.img
  1,073,741,824 100%  382.51MB/s    0:00:02 (xfr#1, to-chk=0/1)

real    0m2.680s
user    0m3.046s
sys     0m0.857s

Just a small sample test.

5

That looks like it should work to me to minimise fragmentation (the --transfers 1 is the important bit).

If you want perfect ordering with --order-by then add the --check-first flag - rclone will build the entire transfer list in memory first, then transfer it in order. This actually will improve the performance on HDD quite a bit if you are updating a transfer.

Personally I use rclone for copying all my local files about. It is fast, -P is way better than what the unix tools normally show and I know it checks the checksums.

I may be biassed though :wink:

Note that rclone doesn't (yet) preserve permissions or ownership for local copies so if you want that, you'll need a different tool.

1 Like
6

--check-first is what I was missing.

"fragmentation" probably isn't exactly the right word. If a project's files, which are likely to be used together, are located all over the disk (i.e. .git or a video project), then there's still a penalty to accessing things for that project... maybe... I think. :upside_down_face:

7

Yes --check-first is great for HDD :slight_smile:

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.