HOW TO: Switch from encfs to rclone crypt

Help and Support
42

And you’re using this on both of your servers?

Edit: well i think I’m doing something wrong, keep on getting mounting errors after a reboot:

plex@scw-855bb7:~/acd$ cd acd2017/01/29 19:18:17 mount helper error: fusermount: failed to access mountpoint /home/plex/acdcrypt: Permission denied
2017/01/29 19:18:17 Fatal error: failed to mount FUSE fs: fusermount: exit status 1
2017/01/29 19:19:23 mount helper error: fusermount: failed to access mountpoint /home/plex/acdcrypt: Permission denied
2017/01/29 19:19:23 Fatal error: failed to mount FUSE fs: fusermount: exit status 1

Too much of a sunday night to get bothered why though. Probably doing something stupid while mounting.

43

maybe you run it multiple times it may get messy check with findmnt

44

Weird… I rebooted the vps again, mounted - it seemed to work.
Then when I did a tree --du -h /home/plex/acd to check the size (to compare them to my acdcrypt) my whole putty session quit on me with some kind of MAC error as well.

Tried to reproduce that error, but couldn’t. Figured I could do a sync to continue the copy, but as soon as I started that, I got the same errors again (Failed to copy: Post https://content-eu.drive.amazonaws.com/cdproxy/nodes?suppress=deduplication: remote error: bad record MAC). Checked one of the files mentioned to see if it was corrupted, but this wasn’t the case. Played just fine.

Something weird is going on, not quite sure how to fix this.

45

Try to copy that file manually ( not play )
Set it as source/dest in script bellow and check if it will finish and with what speed

46 
plex@scw-855bb7:~/scripts$ ./copyspeedtest
COPY SPEED TEST START
|   PROGRESSs   |   RATE   | AVG. RATE |
2.18GiB 0:01:56 [19.2MiB/s] [19.2MiB/s] [============================================================================================================>] 100%

Seems to be different files every time I’m trying a sync again… This must be encfs related.

47

So you are copying from your ENCFS unencrypted to your cryptremote ?

This is what I used at the end
/usr/bin/rclone copy /storage/acd/ acdcrypt: --no-traverse --no-update-modtime --transfers=30 --checkers=30 --min-age 180m --log-file=/var/log/acd2crypt.log

/storage/acd/ - rclone mount was on /storage/.acd/ that was layered with encfs
acdcrypt - rclone crypt mount

I removed -c that I had at beginning since crypt does not support checksum, -no modtime to keep original ones so plex dont think its updated file, the min-age 180m was there since i was also uploading new files to encfs drive during copy and did not want to risk with copying some partial files.

48

I think I know what the problem was… I increased block size in my encfs.xml file to 4096. Just set it back to 1024 and now transfers seem to be going ok. I think I might as well rm -Rf everything that’s in my crypt folder now and start over with your suggested command. Minus --min-age, since I’m using local storage @ Hetzner for new stuff anyway.

Tnx (again)

49

No need to rm -rf whatever is copied so far should be, and whatever is not rclone check will show you.

50

Won’t a rclone copy overwrite files that were already transferred anyway? Shouldn’t I do a sync?
Btw, spoke to soon - checking the log now and the errors are back.

51

No rclone copy will skip files that have same size/date.

Can you check how much ram you have during this eg free

52

2017/01/29 23:43:36 Qfw,hA8Al,AZyo3an9knLaFq/EcUVDRCq,xlJk9wIYgjpdcRFWznf3CDrgjf0bCMjI3A6z1/5b7UTJiypkEHIMX7eArtY5aXxRMc8HdVZ3MwNWsJHYjuXEoJGY31uug0QqmgQPZSWnUp4znmmzWixh42caMA,J93: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF

These are new =)

total used free shared buff/cache available
Mem: 2051100 359440 101152 16424 1590508 1486652
Swap: 0 0 0

53

Ok, so I’ve let it run again last night. Lot’s and lot’s of errors again. Might be better to actually copy my entire library (it’s still <3TB) locally, decrypt it, and upload it with crypt.

Seems like a pain, but this isn’t going to work anyway.

54

@Ajki

I’ve quit the transfer. Somehow it just kept on going (i guess it had lot’s of errors or retries). Was uploading a library @ ~1.1TB and finally quit the upload which had transferred about 4TB.

No idea why this is, but for now I’ll stick with encfs.

55

How did you transfer ?

56

rclone copy /storage/acd/ acdcrypt: --no-traverse --no-update-modtime --transfers=50 --checkers=50 --log-file=/var/log/acd2crypt.log

57

If you had low amount of ram i suggest you lower the transfers …try to do with --transfers=1 --checker=1 to see if you are getting any corruption.

p.s. Just two days ago I finished transferring my 40TB library from acdDE to acdUK. ( and i had around 10% of overhead )

58

I am running a FUSE directory setup for encryption with encfs (local - unencrypted, .local - encrypted, media - plex mount point)

I have a dedicated plex server with 200GB local storage and I have a 14TB encfs encrypted acd library attached that I want to convert to rclone crypt and then push back up to acd as well as gdrive, now as far as I can see I have three options:

  1. Download 100GB chunks from acd library to .local folder, encrypt the decrpyted local folder with crypt, push back up to acd and gdrive

  2. Same as above but get a 2nd dedicated storage server with 12TB of local storage and download the 14TB of data in the respective libraries (Movies, TV, etc) and then re-encrypt and push back to acd and gdrive

  3. Start over, keep the current acd drive live. but re-download my whole library and encrypt it with crypt and then push to gdrive and either the same acd or a new one.

Ideally i’d like to give option 1 a try so my question is this, is it possible to use rclone to do what is listed in option 1 or would one of the other options be better.

59

@Geof_Grouch none of above, you don’t need any local storage since data will be downloaded/encrypted/uploaded in memory. Just copy from encfs mount to crypt drive.

60

@Ajki Can you elaborate as to what I have to do, is there a script to run? Cheers

61

Check the very first post in this topic.