Storing large (or split) archives on rclone mount

Help and Support
1

rclone v1.62.2

  • os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
  • os/kernel: 10.0.22621.1485 Build 22621.1485.1485 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: cmount

hello all, was doing some experiments and curious if people have had any experience in this regard

rclone chunk -> rclone crypt -> box - works like a champ, on a 5 gb max account

hopefully this makes sense.....

I had some large archives of very seldom used stuff - I created a 2.2 tb .7z file - copied it (which rclone - chunked fine) - and with rclone mount it certainly appears that it was able to read the header very quickly in 7zip (it certainly wouldn't be in local cache) and extracting random files from within the archive was quite fast.

however when I try this with (say) a 100 gig file that is in 50 2gb .7z chunks - it seems like it needs to scan (aka download to the cache) all the pieces.

I guess this is more of a deep .7zip question but wondering if anybody has played around with any archiving solution + rclone mount and what their experience might be.

really interested in being able to put split archives up bypassing rclone chunking if possible.

thanks

2

in my testing, after opening a multi-part .7z.
for each part, 7z read 17MB, so rclone downloaded 17MB into the cache.

image
image991×309 14.8 KB

3

Interesting I never saw partial items in cache , it sure looked to me like it was downloading all the objects into cache - which backend did you test on?

Edit: I see it’s on wasabi - I wonder if that makes a difference

4

that does not make a difference.

what is the file system that the cache is stored on?
ntfs, refs, exfat or what?

what you can do is:

  1. run the mount command with -vv for debug info.
  2. open that 2.2 tb .7z file in 7zip.
  3. kill the mount
  4. post the full debug log.
5

on NTFS

just tested again, opened a 966 piece 7z (100 gigs total)

image
image1630×1068 56.6 KB

(and I saw 7zip scan each file)

2.1 TB .7z opened in 15 seconds

image
image1642×1118 46 KB

6

what the cache empty before you did that test?

7

yes in both cases the cache was empty (for these files)

8

will nuke all metadata and test again

9

well, 2.09TB of data, 7z reads 226MB.
image

10

yep the 2tb .7z works as expected, the 900 pieces 100 gig one seems to insist on copying / downloading all 900 pieces to read them all in 7z

11

just tested the 900 piece file with a nuked cache, same thing - downloading all the pieces individually (still going in the b background as I write this) full copies of each file - very odd - sort of seems like a 7z thing I wonder what's different

for reference, mount command:

rclone.exe mount boxcrypt:/ c:\boxCRYPT --transfers 30 -v --vfs-cache-mode full --no-checksum --vfs-fast-fingerprint --vfs-read-chunk-size=512M --vfs-read-chunk-size-limit=off --vfs-read-ahead=512M --checkers=8 -cache-dir e:\cache\boxcrypt -

12

try something like this
rclone.exe mount boxcrypt: c:\boxCRYPT --vfs-cache-mode full --buffer-size 0 --cache-dir e:\cache\boxcrypt -vv

make sure the cache is empty, run the mount command, open the .7z, kill the mount and post the full debug log.

fwiw, --checkers=8 does nothing on a mount.

13

https://cryptosmasher.box.com/shared/static/0pj3xer3bxvbpru0bisvebswy5b4foi8.log

just to take one thing out of the mix I did this again the crypt backend not the chunk -> crypt (despite this being below the chunk limit)
thanks

14

you did not mention that, did you did post your config.
makes it hard to help, when you drip out the important info like that.

and from that log, you did not run the simplified command i asked for?
also, post screen snippets of the cache

15

apologies, I meant I took the chunk of the mix for that last run - re running with simplified command now

edit: with the simplified command it's only reading a subset of the files - I'm going to let it finish to make sure

image
image734×510 12.2 KB

16

went back to the docs, taking a stab at this - I think --vfs-read-chunk-size=512M is the problem here, since in my test case the files are 100 meg it's just going to read the whole thing right?

17

right, that is why i asked you run the simplified command.

18

thanks for your patience, I'm going to try this with 4 gig archive chunks and without chunk size I bet this will work as expected, thanks

19

You'll have better luck with .zip files. These have a central directory stored at the end so the unzipper should only have to read the last chunk. I've tested this is the past and it worked very well.

1 Like
20

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.