I’ve uploaded several terabytes to Dropbox without knowledge of the rclone crypt functionality, and need to encrypt the existing files within a single directory. I’ve managed to follow the documentation and set up a secret mount, but moving the files from the plaintext remote to the secret remote seems to take forever. Is there a way to encrypt these files in-place? I’m on Linux and just using the mv shell command. Should I perhaps be using rclone copy?
Also, a question on how the process goes. As I’m moving them from the plaintext to the secret, are the files actually downloaded? What are the mechanics on how that works? I’ve got a gigabit link but trying a small 70 gigabyte folder seems to take forever. I assume it might be because it’s encrypting it locally, which would imply download then upload, but I just wanted to verify this. Thank you.
You will not be able to encrypt them in place as rclone will need to download them to encrypt them.
Copy would be the better option since it’s the simpler operation. You’re right that it will take a long time since you’ll be downloading, encrypting, and then uploading again. If you have enough storage locally for all of the files and a decent download speed, you will probably find it faster to download all 3 Tb first (and use something like --transfers 12 --checkers 100) and then copy it all back up to the crypt.
2017/06/20 14:52:19 NOTICE: Show/poster.jpg: Removing partially written file on error: close /mnt/PCrypt/TV/Show/poster.jpg: input/output error 2017/06/20 14:52:19 ERROR : Show/poster.jpg: Failed to remove partially written file: remove /mnt/PCrypt/TV/Show/poster.jpg: no such file or directory 2017/06/20 14:52:19 ERROR : Show/poster.jpg: Failed to copy: close /mnt/PCrypt/TV/Show/poster.jpg: input/output error
Not much info there. It doesn’t seem to like it, and it was a basic formatted command of rclone copy /Plex/TV /PCrypt/TV
It doesn’t download it all and then upload it all. If you copy from a decrypted remote to a encrypted remote, it will download it in memory based on the buffer and upload it. You don’t need 5TB.
So I imagine a high-memory system would provide faster results. Is this dynamic, would having 16GB as opposed to 1GB automatically produce quicker file operations in this scenario or would the mounts have to be mounted in a different way?
I’m also considering rsync as opposed to cp -R -u -v source dest
Consider that disk space is not a problem locally or in Dropbox I don’t see a problem with duplicating the entire directory structure for a verification of file integrity via diff, but as it is I am having problems.
I’ve set up servers in NJ, CA, TX and GA, each one is taking a major folder (one is actually taking 2 smaller folders). On all 4 servers though the rclone mount connection is failing. Perhaps I need to use the rclone copy and I’m just guessing that means I don’t need the mounts at all. Who knows. I’m gonna try booting a vultr server with Windows and try this graphically as well maybe… Linux is the best but my mileage right now sucks, and time is money and I’m running out of time.
The following catch my attention about rclone move - Should I ditch the mounts and use some form of these?
–b2-chunk-size int Upload chunk size. Must fit in memory. (default 96M)
–buffer-size int Buffer size when copying files. (default 16M)
–checkers int Number of checkers to run in parallel. (default 8)
–dropbox-chunk-size int Upload chunk size. Max 150M. (default 128M)
–retries int Retry operations this many times if they fail (default 3)
–size-only Skip based on size only, not mod-time or checksum
–transfers int Number of file transfers to run in parallel. (default 4)
and of course, I’d probably want to run this massive operation in verbose. -v
So, how would one construct this? Can you brainstorm with me in laymen? I’m thinking something like this but group opinions would make me feel much more confident.
rclone move --b2-chunk-size [90% of ram] --checkers 16 --dropbox-chunk-size 149M --retries 5 --size-only --transfers [4 x #GB of RAM] sourceremote:folder secret:
As I build the command I wonder about the necessity of buffer size, Does this look good? Not forgetting about the warning on the page though. Important: Since this can cause data loss, test first with the --dry-run flag.
Just to restate - the goal here is to move as much as possible as fast as possible, and to consume the entire vps while doing it. No resources wasted. The rclone mount methodology and using system native gnu commands for file manipulation is slow and faulty at best.
I’ve tried a few combinations, wound up stripping out all the flags except a near-max Dropbox chunk size. I can’t figure out how to make this thing use all the ram and theoretically speed up the ability to transfer multiple concurrent transfers. I guess I’ll just have to wait it out.
