According to the documentation rclone is supposed to preallocate files:
Rclone preallocates the file (using fallocate(FALLOC_FL_KEEP_SIZE) on unix or NTSetInformationFile on Windows both of which takes no time) then each thread writes directly into the file at the correct place. This means that rclone won't create fragmented or sparse files and there won't be any assembly time at the end of the transfer.
But this is currently not happening, I've just restored a remote to an empty drive and every file that is somewhat "big" that has been restored is fragmented, 20508 files have been restored, from there 19191 files are =<2MB and only 299 files take 4.85TB. There are 681 fragmented files that equal 5.1TB in the hard drive. Shouldn't there be zero fragmented files if rclone preallocates the space?
Run the command 'rclone version' and share the full output of the command.
rclone v1.61.1
- os/version: Microsoft Windows 10 Pro for Workstations 22H2 (64 bit)
- os/kernel: 10.0.19045.2364 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Aah I've just read in the docs it says that on Windows the files will be sparsed unless --local-no-sparse is used, shouldn't the documentation say that on Windows there is no preallocation by default? I mean the default values disable preallocation unless explicitly set so is confusing that first it says that rclone preallocates then it says it wont preallocate unless a command is used.
Just tried a copy and I got this:
2022/12/29 15:57:36 DEBUG : drive.zip: Need to transfer - File not found at Destination
2022/12/29 15:57:36 INFO : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable
Also may I propose making --local-no-sparse the default? The delay at the start of the download is that long for this to not be considered the default or is there something else?
I would think nobody would want to trade a little bit of speed for much lower performance in the long run, right?
Yes, Windows/NTFS does supports it, but rclone doesn't preallocate by default on Windows unless the --local-no-sparse flag is used because multi-thread is enabled by default.
It also says that it "may cause long delays at the start of downloads".
if the underlying os supports sparse, then, by default, rclone will use sparse file. --local-no-sparse is not default setting so by default rclone will use sparse file. --multi-thread-streams is enabled by default, so rclone will use a sparse file.
as i understand it, only two cases where rclone will not use sparse file.
rclone tries to use sparse and the underlying os does not support it, as with termux on android.
the user intentionally changes rclone default behavior by adding --local-no-sparse
the docs can be confusing.
the potential long delays, that applies when adding --local-no-sparse
disable sparse files (which may cause long delays at the start of downloads)
Interesting. I have just read the issue and understand a little better, so right now besides disabling using sparse we have to disable multi-thread too if we want to avoid fragmentation.
Does using --local-no-sparse disables multi-thread downloads?
It will mean that the first time you download a file, the OS will write the whole file as zeroes before allowing rclone to write bytes. This will avoid fragmentation at the cost of a delay at the start of each download.