Use a tar archive as a backend

wrouesnel · May 5, 2025, 4:06am

I'd like to be able to use a tar archive as a backend for running 1-way sync to the cloud - i.e. an uncompressed tar being used as a filesystem like store to do a sync up to a cloud drive.

My use case is large data imports which come as archives but which should be restored to an S3-like bucket service as individual files - and I'd like to intelligently skip already present files (i.e. based on size/date/modification etc.)

asdffdsa · May 5, 2025, 3:11pm

welcome to the forum,

imo, that is not a realistic, given how most cloud providers work.
in addition, rclone copy entire files, does not support delta copy.
what other cloud copy tools and cloud providers offer tar backend?

tho, i could be wrong.
rclone is open-source, you are welcome to add a tar backend or https://rclone.org/sponsor/

kapitainsky · May 5, 2025, 3:15pm

There was already attempt to implemented something similar:

github.com/rclone/rclone

Add libarchive support to tar/untar remote sites

opened 12:06AM - 30 Dec 24 UTC

fawzibr

enhancement

#### The associated forum post URL from `https://forum.rclone.org` https://fo…rum.rclone.org/t/add-libarchive-support/49191/2 #### What is your current rclone version (output from `rclone version`)? New feature, so latest #### What problem are you are trying to solve? Being able to tar/untar a remote to a local file or to another remote #### How do you think rclone should be changed to solve that? I asked in the forums but nobody answered. I worked with go/rclone before but never worked with libarchive or using C code in go but still decided to give it a try. After tinkering with things I added a very simple 'tar' based on the 'cat' code command to rclone (only tested with local files and writing to stdout as a proof of concept). So right now all I can do is: `rclone -q tar --format tar.gz directory > test.tgz` The format can be tar, tar.xz, tar.bz2, tar.lrzip, tar.lz4, tar.zst, tar.lzma, tar.lzop, tar.Z, tar.gz, 7z and zip. I could add iso as well just because I can. Creating a mountable cd could be cool. I looked in the rclone code to see how I could do it. Tried with the fs.ListFn() like 'cat' first but decided to use the Walk() function instead. After looking around for a way to get the files gid/uid/permissions/dates found the GetMetadata() function with returns a map[string]string with those values but still missing actual user name and group name. Is there a way to get that information? Do all backends get the same metadata? I don't know if the functions I'm using are the right ones (or if they are better ones) and I haven't tested an actual remote filesystem or writing to an actual file or remote file but still wanted to see if it actually works. I might change the command to something more generic like 'rclone archive/unarchive' instead of 'tar/untar' since it can create zip/7z files. Would anyone actually care for this feature? #### How to use GitHub * Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue. * Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue. * Subscribe to receive notifications on status change and new comments.

But it is still waiting for somebody to make it happen. It can be you @wrouesnel if it is something you need. I suspect it is extremely niche feature very few people would use.

asdffdsa · May 5, 2025, 3:25pm

yeah, but i think it lacks requested feature of sync up to a cloud drive.

might use rclone mount and whatever program(s) that works with tar files.

wrouesnel · May 5, 2025, 3:27pm

I saw that work and I think it's targeting a different problem? I have a separate afero based extension of treating a tar file as a filesystem since an uncompressed tar file can be relatively efficiently iterated by skipping through header offsets. This works well enough to read through using fuse.

Sounds like a patch would be accepted here, do I'll see what I can do.

EDIT: the use case is "files in tar" become "a bunch of files on a cloud drive" without needing my users to do more then run "rclone sync tar:somefile s3:some bucket"

sweh · May 5, 2025, 5:51pm

How sophisticated are the users? Potentially they could mount the tar file to the filesystem using something like ratarmount and then rclone sync the visible files.