Problems with S3 sparse files

When I uploaded it to AWS S3, AWS S3 display took up 2GB. When I copied it to the local computer using RClone, the local display took up 2GB of real space. How can I restore this 2GB file to a sparse file? My native language is not English, please forgive me.

hello and welcome to the forum,

if the file in s3 is 2GB, when you download it, the file in local will also be 2GB.
not sure how sparse file will change that?

to increase download speed, rclone can use sparse files.
https://rclone.org/docs/#multi-thread-cutoff-size

First of all, my environment is Linux Ubuntu 20.0.4. The actual size of the file in the local hard disk is only over 100 MB, but when I uploaded it to S3, it was 2GB. When I downloaded it from another computer through RClone, the size of the file was 2GB, and then I could not use the file normally. By what kind of command, or method, shall I restore him to his former form? Due to a lot of similar files, through compression packaging, upload the way, in the download back, this is too troublesome

Rclone currently doesn't sparse or unsparse files and you can't store sparse files on S3.

However you can use cp to re sparse the file

cp --sparse=always src dest

hi,

just curious if this could be a solution?

rclone mount remote: /path/to/mount
cp --sparse=always /path/to/mount/file.txt /path/to/dest

Yes that would work too.

rclone should really implement a --sparse flag though like rsync does

    -S, --sparse                turn sequences of nulls into sparse blocks
1 Like

what is the problem, lack of free space on local storage?

this might be a workaround.

  1. use rclone mount to have the s3 remote appear as local storage.
  2. use command line cp --sparse=always which will download the file as a sparse file of size 100MB.

I haven't tried it yet. I'll reply when I do.

I have stored them in S3 files with 8T files and tens of thousands of files. The local space is 12TB. I wonder whether your method can be used.

There are tens of thousands of S3 files with a total of 8T. Using CP in this way will not work.

By mounting, sparse files can be copied using CP, but the speed is too slow, only 100MB per second, far worse than using RClone copy directly from S3 to local, or even up to 1GB per second. Is there any way to improve the copy speed of CP? By what order?

when you first posted, there was a template of questions to be answered.
without that info, not possible to answer your question

please post:

  • the results of speedtest, perhaps https://speedtest.net
  • the output of rclone version
  • the config file, redact id/secret/token/etc...
  • the rclone mount command.
  • the debug log of that rclone mount command.

The easiest thing to do would be do lots of cp in parallel. Are you copying just one file or lots?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.