Rclone doesn't preallocate files when restoring

What is the problem you are having with rclone?

According to the documentation rclone is supposed to preallocate files:

Rclone preallocates the file (using fallocate(FALLOC_FL_KEEP_SIZE) on unix or NTSetInformationFile on Windows both of which takes no time) then each thread writes directly into the file at the correct place. This means that rclone won't create fragmented or sparse files and there won't be any assembly time at the end of the transfer.

But this is currently not happening, I've just restored a remote to an empty drive and every file that is somewhat "big" that has been restored is fragmented, 20508 files have been restored, from there 19191 files are =<2MB and only 299 files take 4.85TB. There are 681 fragmented files that equal 5.1TB in the hard drive. Shouldn't there be zero fragmented files if rclone preallocates the space?

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1
- os/version: Microsoft Windows 10 Pro for Workstations 22H2 (64 bit)
- os/kernel: 10.0.19045.2364 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Luckily I saved the entire command.

rclone copy "CRYPT-REMOTE" R:\ --transfers 8 --checkers 14 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --rc --rc-addr=:9966 --rc-user=admin --rc-pass=xxxxx --rc-web-gui --rc-web-gui-update --rc-serve --rc-web-gui-no-open-browser --create-empty-src-dirs --links --drive-chunk-size 512M

The rclone config contents with secrets removed.

Does this matter?

A log from the command with the -vv flag

I didn't save it, but I remember not getting any errors. I made the restore days ago.

that is confusing, not sure what it exactly means or needs an edit

as i would assume that the windows os, would not ensure a 2TiB sparse file is composed of contiguous sectors.

1 Like

rclone will not respond with an error, just a debug message.
i get this every time i use termux
DEBUG : preAllocate: got error on fallocate

and not all file systems support sparse files.

1 Like

Aah I've just read in the docs it says that on Windows the files will be sparsed unless --local-no-sparse is used, shouldn't the documentation say that on Windows there is no preallocation by default? I mean the default values disable preallocation unless explicitly set so is confusing that first it says that rclone preallocates then it says it wont preallocate unless a command is used.
Just tried a copy and I got this:

2022/12/29 15:57:36 DEBUG : drive.zip: Need to transfer - File not found at Destination
2022/12/29 15:57:36 INFO  : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable

Also may I propose making --local-no-sparse the default? The delay at the start of the download is that long for this to not be considered the default or is there something else?
I would think nobody would want to trade a little bit of speed for much lower performance in the long run, right?

sure, but that is not going to happen. very difficult to change defautls.
and for most rcloners, including myself, the default is correct.

the delay to create an empty sparse file should be very quick.
which takes no time

as i understand it, windows will pre-allocate and your debug log snippet shows that behaviour.

well, good that rclone has the flag and now you know how to use it with your scripts.

Yes, Windows/NTFS does supports it, but rclone doesn't preallocate by default on Windows unless the --local-no-sparse flag is used because multi-thread is enabled by default.

It also says that it "may cause long delays at the start of downloads".

if the underlying os supports sparse, then, by default, rclone will use sparse file.
--local-no-sparse is not default setting so by default rclone will use sparse file.
--multi-thread-streams is enabled by default, so rclone will use a sparse file.

as i understand it, only two cases where rclone will not use sparse file.

  1. rclone tries to use sparse and the underlying os does not support it, as with termux on android.
  2. the user intentionally changes rclone default behavior by adding --local-no-sparse

the docs can be confusing.
the potential long delays, that applies when adding --local-no-sparse

disable sparse files (which may cause long delays at the start of downloads)

1 Like

Note that preallocating files doesn't stop fragmentation, it just reserves the disk space from the os so we can be sure the file will fit.

Multi thread downloads cause fragmentation, so turn that off if you don't want fragmentation.

2 Likes

Interesting. I have just read the issue and understand a little better, so right now besides disabling using sparse we have to disable multi-thread too if we want to avoid fragmentation.
Does using --local-no-sparse disables multi-thread downloads?

No it doesn't.

It will mean that the first time you download a file, the OS will write the whole file as zeroes before allowing rclone to write bytes. This will avoid fragmentation at the cost of a delay at the start of each download.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.