Hello, I noticed that my files need "--sparse" when I upload them to Dropbox due to size differences. I'm not sure if rclone supports this feature. If I create a .tar file with --sparse before uploading the folder to Dropbox, it's okay. But if I upload files without this workaround, the uploaded data becomes three times or more larger.
Does rclone support this feature?
I tried with "rclone mount" and the classic Linux "cp --sparse" command, but it looks like this doesn't solve my problem.
Not sure what your issue is. Does Dropbox support sparse files? nope... equally not any other object orientated storage.
rclone will utilise local sparse file capability when you download file to local file system supporting this feature when multi threading downloads are possible. That's pretty much all about rclone and sparse files
What exactly you want to achieve with your --sparse flag? What do you mean by rclone to support it?
Here old boring wikipedia re what sparse file is, to make sure we talk about the same thing:
So local file system stored file which is 1GB in size can occupy 100MB of real space (sparse capability). It will occupy 1GB when stored by Dropbox (at least from your billing perspective). If you want rclone to support it then you have to talk to Dropbox to support it.... Then come back with a news that they enabled their new feature supported by API:)
I'm encountering issues with very specific encrypted files on server (multiple 2GB packages). I'm attempting to transfer these files to Dropbox, but when I use the regular 'rclone move' command, the single 2GB file balloons to 11GB.
I've observed the same phenomenon when using the classic 'tar' command or classic FTP transfer; the file simply expands to 11GB.
However, I've found that using 'tar --sparse' maintains the original size of my files (2GB).
Which makes perfect sense. Tar understands how to store empty spaces.
Problem is that cloud storage does not support sparse files. It is not rclone limitation.
Only solution to store sparse files efficiently in such case is to use some sort of container (like tar).
The same storage space savings can be achieved using any compression format - even on lowest compression settings as empty space (this is what sparse file system format cleverly "hides") always compresses very well. So if your specific workflow requires sparse files storage you can leverage compress overlay maybe. What would help here would be to add different compression formats to this remote, e.g. zstd to support extremely efficient on modern many cores CPUs format and maybe lz4 - the latter is extremely fast and would be ideal for sparse files compression.
But until destination storage does not support sparse file explicitly you will always have to use some sort of intermediary format (tar, zip, etc.)
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.