Too much bandwidth wasted with vps if I unrar or unzip with the local machine's command.
Unpacking at the local machine and uploading it back. I want to do it over cloud itself.
What is your rclone version (output from rclone version)
1.56.0
Which OS you are using and how many bits (eg Windows 7, 64 bit)
ubuntu 20.04
Which cloud storage system are you using? (eg Google Drive)
Google drives
The command you were trying to run (eg rclone copy /tmp remote:tmp)
oh let me clarify, so once rar parts are uploaded to the drives, I don't want to unrar them from the local side anymore. I want there to be a way for me to unpack them directly on google. So there won't be any write / upload being involved.
I just registered the f1 class free machine. Weird config with the free defined by hours used. Don't really know if that means something like: running 3 instances but total power on time does not exceed 720.
You could use rar2fs on top of rclone. this will let you stream files directly without the need to unrar them before. It does add an additional layer and you need to prewarm the cache so it works decently with rclone. But once it's set it works pretty good.
I only set my cache to be smth around read ahead 2GB. So even if warm it up, every time I start from there again there would be some delay. Which I don't mind for streaming.
Was gonna be smth I was going to ask, thanks for letting me know about it here!
i find that with a simple mount, without cache, such as rclone mount remote: b:\mount\rclone\remote
i can instantly access the folder/file structure of a .7z and download an individual file, not the entire .7z was in this case is 4GB
what advantage does rar2fs have?
what need is there for a vfs cache and to prewarm it?
Problem with that is, I can't seem to write into the directory correctly with a simple mount. The file would be copied to local mounted storage dir. Which is weird but vfs cache solves it.
rar2fs is for like split media files, more like you would know what it is if you happen to come across those types of media.
warming the cache is like warming the dram cache in your hdd, faster read and writes.
yes, to write, the odds are you do need the vfs cache.
but my post was about reading from a .7z in the cloud.
about prewarm/priming/ pre-caching the mount, using vfs/refresh.
that is not for faster read/writes as no data from files are downloaded.
the prewarm is only for folder/file structure, for quicker navigation
or to speed up a re-scan from media server, to find new or updated files from a rclone mount.
I was talking about warming rar2fs cache. As it will be slow first time you try to open a folder with rar files as it has to analyze the rar file to know the contents.
Pretty much it will show files inside the rar files without the need to open them. For example you have one big rar file (or a multiple parts rar file) in general you would need to unrar it to access the file inside. rar2fs does it on the fly, so you do not need to unrar before reading the contents. It is smart enough it allows you to seek etc. on the files inside the rar.
I was talking about warming the rar2fs cache not the rclone vfs. Basically what it will do is scan all rar files in the mounted directory and read the inside contents on the background therefore not adding IO Wait during directory seeking. This is huge when using it with rclone as we get enough delay when asking for a file, so in case of rar2fs it would lock fuse for a bit while it scans the rar file to see what content is inside.
yeah now it makes more sense, thanks for clarifying. Tho so far my experience using rar2fs on arch has been great with hardly any delay even when streaming over samba.
yeah rar2fs is great and does not use too much overhead. Be sure to try and use background cache
warmup option -owarmup with rar2fs, helps a lot when using rclone as it will scan all files in the background when you first mount everything.