Basically I want to roll the entire contents of my gdrive in to a tarball and then use the fact that gdrive allows the current download or upload operation occurring when a quota is reached to complete to bypass the 750GB limit. Once uploaded to gdrive, if I use rclone mount, is the rclone chunker able to address, chunk, and operate on a tarball without downloading the entire tarball?
This is what I want to do:
Upload an 8TB tarball.
(This will be allowed, despite the 750GB limit, because gdrive will not terminate an active upload even when the quota surpasses 750GB)
Mount my gdrive with rclone mount
Perform any future operations as extractions, additions, or deletions on the tarball itself. This will let the rclone chunker handle upload and download operations such that I can stay within bandwidth limits with reasonable reliability. I can adjust adjust the chunk size in an effort to make this more efficient, rather than use lots of service accounts.
Bonus: If I ever need to pull the tarball off of gdrive, I can do so without worrying about the 750GB cap due to that download being a single operation (just as the upload).
Hopefully that was clear… let me know if I can restate that in a better way.
Thanks!
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Is it possible to do this with just the vfs layer or is there an architectural limitation somewhere?
My understanding is that when skipping around in, say, video files, rclone is not required to pull down the whole file, but is able to jump to the proper block and stream the file starting at a given point, yes?
To add/delete, you have to block level change stuff which won't work as cloud backends that I'm aware of, do not support that. To replace part of the file without rewriting the whole file, that's usually done at the block level.
If you have to seek around a file, yes, that's correct as that's 1 out of 3 scenarios you've described. I believe a tar file though reads from start to finish to find something though.
Chunker remote isn't this use though to be clear.
Best thing I did was moving away from Google. No upload or download limits.
Okay thanks for clarifying/docs. Chunker is definitely not the solution.
However, and it looks like modifying the file is not possible. However, what about mounting as read-only and extracting individual files as necessary? This may require a technical understanding of tar that I don’t have because (I think) if the entire file must be read to create an index of its contents, then that will require a full download?
The limitations of gdrive are cumbersome but the price per GB really can not be beat. Glacier isn’t really an option for me…
I wonder if a deduped backup (e.g. Borg) would make this possible layered on top of the rclone vfs layer?
This is what I want to do:
Upload an 8TB tarball.
(This will be allowed, despite the 750GB limit, because gdrive will not terminate an active upload even when the quota surpasses 750GB)
Yes, rclone can stream only the needed parts of a zip file even if it is parts from multiple files. You can also use any other file archiver that creates files with indexes this includes 7z, rar, zip but not tar because it does not have indexing and for that reason the entire tar will need to be downloaded to build the index. You can even have a container vhd, vhdx, iso or any file, but once uploaded it can not be changed so you cannot easilly delete files from the archive for example.
Once you create a split zip files and move them to a remote with move/copy/sync you can mount the remote, point a archiver program to the files and extract the files you want, without needing to download the whole zip(s). There is still a big limitation of not being able to modify the zip files without a full reupload
You can experiment yourself if you want
So, Can you extract files from zip archives without a full download Yes
Can you remove files from a zip archive without a full upload No
Can you mount a remote and then create the zip on it No (zip file is not written sequentially and remotes need to be written sequentially)
Ahh, I should’ve been more clear; I was wondering if the zip backend would concatenate a split archive.
This still seems like the answer, I’m just trying to learn what to expect since the upload is substantial I want to be pretty sure what I’m getting in to here… haha. Thanks!