HOW TO: Switch from encfs to rclone crypt

pushnoi · January 29, 2017, 6:44pm

And you're using this on both of your servers?

Edit: well i think I'm doing something wrong, keep on getting mounting errors after a reboot:

plex@scw-855bb7:~/acd$ cd acd2017/01/29 19:18:17 mount helper error: fusermount: failed to access mountpoint /home/plex/acdcrypt: Permission denied
2017/01/29 19:18:17 Fatal error: failed to mount FUSE fs: fusermount: exit status 1
2017/01/29 19:19:23 mount helper error: fusermount: failed to access mountpoint /home/plex/acdcrypt: Permission denied
2017/01/29 19:19:23 Fatal error: failed to mount FUSE fs: fusermount: exit status 1

Too much of a sunday night to get bothered why though. Probably doing something stupid while mounting.

Ajki · January 29, 2017, 7:38pm

maybe you run it multiple times it may get messy check with findmnt

pushnoi · January 29, 2017, 10:58pm

Weird… I rebooted the vps again, mounted - it seemed to work.
Then when I did a tree --du -h /home/plex/acd to check the size (to compare them to my acdcrypt) my whole putty session quit on me with some kind of MAC error as well.

Tried to reproduce that error, but couldn’t. Figured I could do a sync to continue the copy, but as soon as I started that, I got the same errors again (Failed to copy: Post https://content-eu.drive.amazonaws.com/cdproxy/nodes?suppress=deduplication: remote error: bad record MAC). Checked one of the files mentioned to see if it was corrupted, but this wasn’t the case. Played just fine.

Something weird is going on, not quite sure how to fix this.

Ajki · January 29, 2017, 11:04pm

Try to copy that file manually ( not play )
Set it as source/dest in script bellow and check if it will finish and with what speed

github.com

ajkis/scripts/blob/master/other/copyspeedtest

#!/bin/bash
## Requirements pv tool
## sudo apt install pv
## Create test file and upload it to your cloud destination
## fallocate -l 100M testfile

## More scripts at: https://github.com/ajkis/scripts
## If you find script useful feel free to buy me a beer at https://paypal.me/ajki
if pidof -o %PPID -x "copyspeedtest"; then
   echo "Already running, exiting."
   exit 1
fi

SOURCE="/mnt/unionfs/testfile" # SET PATH TO FILE
DESTINATION="/tmp/copytestfile"

if [[ -f "$DESTINATION" ]]; then
    rm "$DESTINATION"
fi
start=$(date +'%s')

This file has been truncated. show original

pushnoi · January 29, 2017, 11:18pm

plex@scw-855bb7:~/scripts$ ./copyspeedtest
COPY SPEED TEST START
|   PROGRESSs   |   RATE   | AVG. RATE |
2.18GiB 0:01:56 [19.2MiB/s] [19.2MiB/s] [============================================================================================================>] 100%

Seems to be different files every time I'm trying a sync again... This must be encfs related.

Ajki · January 29, 2017, 11:26pm

So you are copying from your ENCFS unencrypted to your cryptremote ?

This is what I used at the end
/usr/bin/rclone copy /storage/acd/ acdcrypt: --no-traverse --no-update-modtime --transfers=30 --checkers=30 --min-age 180m --log-file=/var/log/acd2crypt.log

/storage/acd/ - rclone mount was on /storage/.acd/ that was layered with encfs
acdcrypt - rclone crypt mount

I removed -c that I had at beginning since crypt does not support checksum, -no modtime to keep original ones so plex dont think its updated file, the min-age 180m was there since i was also uploading new files to encfs drive during copy and did not want to risk with copying some partial files.

pushnoi · January 29, 2017, 11:31pm

I think I know what the problem was... I increased block size in my encfs.xml file to 4096. Just set it back to 1024 and now transfers seem to be going ok. I think I might as well rm -Rf everything that's in my crypt folder now and start over with your suggested command. Minus --min-age, since I'm using local storage @ Hetzner for new stuff anyway.

Tnx (again)

Ajki · January 29, 2017, 11:34pm

No need to rm -rf whatever is copied so far should be, and whatever is not rclone check will show you.

pushnoi · January 29, 2017, 11:44pm

Won't a rclone copy overwrite files that were already transferred anyway? Shouldn't I do a sync?
Btw, spoke to soon - checking the log now and the errors are back.

Ajki · January 29, 2017, 11:46pm

No rclone copy will skip files that have same size/date.

Can you check how much ram you have during this eg free

pushnoi · January 29, 2017, 11:49pm

2017/01/29 23:43:36 Qfw,hA8Al,AZyo3an9knLaFq/EcUVDRCq,xlJk9wIYgjpdcRFWznf3CDrgjf0bCMjI3A6z1/5b7UTJiypkEHIMX7eArtY5aXxRMc8HdVZ3MwNWsJHYjuXEoJGY31uug0QqmgQPZSWnUp4znmmzWixh42caMA,J93: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF

These are new =)

total used free shared buff/cache available
Mem: 2051100 359440 101152 16424 1590508 1486652
Swap: 0 0 0

pushnoi · January 30, 2017, 9:04pm

Ok, so I’ve let it run again last night. Lot’s and lot’s of errors again. Might be better to actually copy my entire library (it’s still <3TB) locally, decrypt it, and upload it with crypt.

Seems like a pain, but this isn’t going to work anyway.

pushnoi · February 15, 2017, 1:52pm

@Ajki

I’ve quit the transfer. Somehow it just kept on going (i guess it had lot’s of errors or retries). Was uploading a library @ ~1.1TB and finally quit the upload which had transferred about 4TB.

No idea why this is, but for now I’ll stick with encfs.

Ajki · February 15, 2017, 2:12pm

How did you transfer ?

pushnoi · February 15, 2017, 2:14pm

rclone copy /storage/acd/ acdcrypt: --no-traverse --no-update-modtime --transfers=50 --checkers=50 --log-file=/var/log/acd2crypt.log

Ajki · February 15, 2017, 2:16pm

If you had low amount of ram i suggest you lower the transfers …try to do with --transfers=1 --checker=1 to see if you are getting any corruption.

p.s. Just two days ago I finished transferring my 40TB library from acdDE to acdUK. ( and i had around 10% of overhead )

Geof_Grouch · May 9, 2017, 2:50pm

I am running a FUSE directory setup for encryption with encfs (local - unencrypted, .local - encrypted, media - plex mount point)

I have a dedicated plex server with 200GB local storage and I have a 14TB encfs encrypted acd library attached that I want to convert to rclone crypt and then push back up to acd as well as gdrive, now as far as I can see I have three options:

Download 100GB chunks from acd library to .local folder, encrypt the decrpyted local folder with crypt, push back up to acd and gdrive
Same as above but get a 2nd dedicated storage server with 12TB of local storage and download the 14TB of data in the respective libraries (Movies, TV, etc) and then re-encrypt and push back to acd and gdrive
Start over, keep the current acd drive live. but re-download my whole library and encrypt it with crypt and then push to gdrive and either the same acd or a new one.

Ideally i’d like to give option 1 a try so my question is this, is it possible to use rclone to do what is listed in option 1 or would one of the other options be better.

Ajki · May 10, 2017, 3:26am

@Geof_Grouch none of above, you don’t need any local storage since data will be downloaded/encrypted/uploaded in memory. Just copy from encfs mount to crypt drive.

Geof_Grouch · May 10, 2017, 7:29am

@Ajki Can you elaborate as to what I have to do, is there a script to run? Cheers

Ajki · May 10, 2017, 9:34am

Check the very first post in this topic.