My streams seem much more stable on my new server, but I’ve got 2 questions;
Using your ACD Speedtest, (with Crypt) - I seem to get about a 40s delay before Rclone will begin to copy the file, once it starts its consistently 30MiB/s+, but it takes way too long to begin. It seems to be the same with Plex too
Have you noticed that Rclone seems to be noticeably slower when copying through the unionfs mount then when copying directy from the rclone mount point? do you have that issue?
Iam also checking it with direct_io it seems it improves performance on file access.
Sometimes it can take 10 to 30 seconds before file is actually being copied ( copyspeedtest ) and since I was trying with direct_io i never encounter it … could be a fluke as it does not make sense.
This option disables the use of page cache (file content cache)
in the kernel for this filesystem. This has several affects:
1. Each read(2) or write(2) system call will initiate one or more
read or write operations, data will not be cached in the kernel.
2. The return value of the read() and write() system calls will
correspond to the return values of the read and write
operations. This is useful for example if the file size is not
known in advance (before reading it).
acdcrypt: $MPOINT &
Did not notice any difference with or without -v and logging the advantage with this info log is that you can tail with what speed if file getting transferred. eg tail -f /home/plex/logs/mountacdcrypt.log | grep ETA
Iam pointing it to fuse and it seems direct_io solved the flakiness with unionfs, the main problem is ACD and not really the fuse. I did several tests with local fuse and while read is almost on pair the write speed is a bit slower but still you will hit bandwith limit way before you hit that one.
This is read speed of local files unionfs compared to local copy
If 1GB is copied it over fuse in 0,75 sec, instead of 0.66 is no big deal.
As for speed from ACD with or without fuse its more about amazon none consistent speeds then fuse it self.
few minutes later same test ( rclone remounted so cache is cleared )
Install Ubuntu 16.04 , wget latest rclone beta and copy your rclone config
make simple bash script acdus2acdde.sh
and put it incrontab -e
This will make sure that script would be run every minute, but dont worry it wont allow to instances at same time. IMPORTANT. Sync remote drives directly and not secret/crypt ones.
Once all your files are synced ( 3 days top’s for 8TB ) just check the log, remove it from crontab and change the sync part with /usr/bin/rclone check acdus: acdde: -v --log-file=/path/to log file/
Run it once and when its finished check the logs if all files are ok.
Now when you have everything synced open rclone.conf find entry with type = crypt and change remote = acdde:/myencryptedfiles ( just the remote part if it was acd now change it to the new registered remote )
And thats it you don’t need to change anything else.
A little off topic but has anyone looked into using https://github.com/trapexit/mergerfs instead of unionfs ? Seems like it has some pretty cool features. Probably helpful to those who have a primary SSD drive and a secondary SATA drive.