Cryptsetup and mounts

p789 · October 23, 2019, 4:15pm

Hi,

I have a file "enc_file" that contains a luks-encrytped filesystem that I copied to my google drive.

I now mount my gdrive like this:

rclone mount gdrive: mnt --allow-other --vfs-cache-mode full

And the md5sum between my local copy of this file and the file sitting on my gdrive (in my mounted directory) are the same.

But when I try to open the file on gdrive like this:

sudo cryptsetup luksOpen enc_file hubba

I get an error:

Device enc_file is too small. Need at least 16777216 bytes.
Device enc_file is not a valid LUKS device.

When I try the same operation against my local copy I am prompted for a password.

What could be the problem here?

Many thanks!

thestigma · October 23, 2019, 4:46pm

For a start I would just try not using cache mode full, because this is a mode that is generally not recommended for most uses. It will force all files to be downloaded completely before access - thus disabling any capacity to stream or quickly access things. It may be appropriate to use in very spesific setups, but you should probably prefer to use --vfs-cache-mode writes

Secondly, while it should work in theory I suspect that using a single-file encrypted container like that will be quite slow in many regards. rclone will need to do a lot of seeking and jumping around in that file - and each such operation takes some time to do. Using rclones native crypt support is almost certainly going to be much faster - so I would at least consider doing that. Let us know if you need help regarding that or want to know more about it (and you can read up on the documentation as well of course).

There are some limitations on what kind of file-access is possible to do in practice through a remote. write caching should normally fill in the gaps though.

So try cache mode writes and see if that works first of all, then report back and we'll continue from there.

p789 · October 23, 2019, 4:50pm

I think you are quite right in assuming that it will be quite slow, but I have now tried

rclone mount gdrive: mnt --allow-other --vfs-cache-mode writes

and the result is the same.

I simply would like to understand what is happening....

thestigma · October 23, 2019, 6:35pm

I have no experience with luks, but that error indicates it is measuring the size of the file and finding a problem. I guess it is mis-measuring for some reason and maybe is effectively polling size of the rclone file-segment or something instead pf the actual because of the method it uses to ask. That is my best hypothesis based on the little information we have to go on here.

Maybe @ncw can give you a suggestion for what settings it might be worth experimenting with.

As a shot in the dark, I'd try maybe setting

-vfs-read-chunk-size 1G

or a size that is at least the size of the file...
I don't know if you can completely disable read-chunking (I have never tried) but setting it to 0 would probably do that.

Animosity022 · October 23, 2019, 7:04pm

What rclone version are you running?

What's the ls -al of the file before you move it to your GD?

What's rclone ls show of the file once moved over?

If you run the mount with -vv, can you share the debug log when you run the action to create the error?

p789 · October 23, 2019, 8:51pm

rclone --version
rclone v1.45

os/arch: linux/amd64
go version: go1.11.6

ls -la enc_file
-rw-r--r-- 1 mh mh 134217728 Oct 23 17:33 enc_file

rclone ls gdrive:enc_file
134217728 enc_file

2019/10/23 22:50:34 DEBUG : enc_file: >Open: fd=enc_file (rw), err=
2019/10/23 22:50:34 DEBUG : enc_file: >Open: fh=&{enc_file (rw)}, err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Read: len=4096, offset=0
2019/10/23 22:50:34 DEBUG : enc_file: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/10/23 22:50:34 DEBUG : enc_file: Unchanged skipping
2019/10/23 22:50:34 DEBUG : enc_file(0xc000158ea0): Opened existing cached copy with flags=O_RDONLY|0x4000
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Read: read=4096, err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Read: len=12288, offset=4096
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Read: read=12288, err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Flush:
2019/10/23 22:50:34 DEBUG : enc_file(0xc000158ea0): RWFileHandle.Flush ignoring flush on unwritten handle
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Flush: err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Release:
2019/10/23 22:50:34 DEBUG : enc_file(0xc000158ea0): RWFileHandle.Release closing
2019/10/23 22:50:34 DEBUG : enc_file(0xc000158ea0): close:
2019/10/23 22:50:34 DEBUG : enc_file(0xc000158ea0): >close: err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Release: err=
2019/10/23 22:50:34 DEBUG : enc_file: Open: flags=OpenReadOnly+0x4000
2019/10/23 22:50:34 DEBUG : enc_file: Open: flags=O_RDONLY|0x4000
2019/10/23 22:50:34 DEBUG : enc_file: >Open: fd=enc_file (rw), err=
2019/10/23 22:50:34 DEBUG : enc_file: >Open: fh=&{enc_file (rw)}, err=
2019/10/23 22:50:34 DEBUG : enc_file: Attr:
2019/10/23 22:50:34 DEBUG : enc_file: >Attr: a=valid=1s ino=0 size=0 mode=-rw-r--r--, err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Flush:
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Flush: err=
2019/10/23 22:50:34 DEBUG : enc_file: Open: flags=OpenReadOnly
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Release:
2019/10/23 22:50:34 DEBUG : enc_file: Open: flags=O_RDONLY
2019/10/23 22:50:34 DEBUG : enc_file: >Open: fd=enc_file (rw), err=
2019/10/23 22:50:34 DEBUG : enc_file(0xc0006f83c0): RWFileHandle.Release closing
2019/10/23 22:50:34 DEBUG : enc_file: >Open: fh=&{enc_file (rw)}, err=
2019/10/23 22:50:34 DEBUG : enc_file(0xc0006f83c0): close:
2019/10/23 22:50:34 DEBUG : enc_file(0xc0006f83c0): >close: err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Release: err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Flush:
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Flush: err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Release:
2019/10/23 22:50:34 DEBUG : enc_file(0xc0001593e0): RWFileHandle.Release closing
2019/10/23 22:50:34 DEBUG : enc_file(0xc0001593e0): close:
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Flush:
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Flush: err=
2019/10/23 22:50:34 DEBUG : enc_file(0xc0001593e0): >close: err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: >Release: err=
2019/10/23 22:50:34 DEBUG : &{enc_file (rw)}: Release:
2019/10/23 22:50:34 DEBUG : enc_file(0xc0006f82a0): RWFileHandle.Release closing
2019/10/23 22:50:34 DEBUG : enc_file(0xc0006f82a0): close:
2019/10/23 22:50:34 DEBUG : enc_file(0xc0006f82a0): not modified so not transferring
2019/10/23 22:50:34 DEBUG : enc_file(0xc0006f82a0): >close: err=

Animosity022 · October 23, 2019, 9:53pm

That seems to be just a snippet of the log rather than the whole log so I'm missing all the mount commands it started with.

That's also a semi old version, you may want to grab the latest and repost a full log with that.

system · January 21, 2020, 9:53pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.