I have about 50 TB of datat stored on Jottacloud and would like to use the file/folder obfuscation that rclone has.
With this amount of data it’s just not reasonable to re-upload anything.
Since the obfuscation basically is “just” a rename (moveto), how would I go about actually doing this?
I found @calisro saying in another thread that this could be scripted but I couldn’t really figure out the steps.
Anyone care to help?
It’s not just a rename. It encrypts the files. It’ll download/encrypt/upload in order to achieve that.
Hmmm, is there any way to just rename the files to obfuscated filenames in rclone? File encryption is not interesting for me really.
I’ve not tested this. But in theory I think you can list the files / directories using something like
rclone cryptdecode --reverse targetremote: file
and then create a move script. That sounds like a lot of scripting though. If it works, it would only for obfuscated ones.
EDIT: This was incorrect.
The problem is that you can only supply ten filenames with cryptdecode. If it had a recursive option and list like lsl/lsd/ls, then I could make it work.
And then you’d also have to do this every time you upload anything new.
Agree with recursive. But once you converted everything, you would just point you remote directly and do it through crypt obfuscate.
But this would then encrypt any new files for real, not just filenames, and that’s not an option with files that are bigger than my amount of RAM. It would crash the machine as I’ve seen in other threads about crypt.
What do you mean crash your machine and your RAM?
I’ve seen bug reports about that. While encrypting the files, rclone puts them in memory. If the file is bigger than your amount of RAM, the machine will end up swapping and then crash when the swap also runs out.
Seems like fiction.
I encrypt rather large files and rclone never uses any more than a few MB
Crypt is a slight CPU hit for the most part and really negligible from a memory perspective.
And you can use a obfuscate remote. That doesn’t encrypt. But crypt won’t crash your machine. I run it on a vps with 2GB ram and 2 vcpus
I’m not quite sure what you mean? There is no remote with only file and foldername obfuscation and you can’t create one either unless I’m severely missing something while doing the config.
This is a simple “rotate” of the filename, with each file having a rot distance based on the filename. We store the distance at the beginning of the filename. So a file called “hello” may become “53.jgnnq”
This is not a strong encryption of filenames, but it may stop automated scanning tools from picking up on filename patterns. As such it’s an intermediate between “off” and “standard”. The advantage is that it allows for longer path segment names.
There is a possibility with some unicode based filenames that the obfuscation is weak and may map lower case characters to upper case equivalents. You can not rely on this for strong protection.
- file names very lightly obfuscated
- file names can be longer than standard encryption
- can use sub paths and copy single files
- directory structure visible
- identical files names will have identical uploaded names
Cloud storage systems have various limits on file name length and total path length which you are more likely to hit using “Standard” file name encryption. If you keep your file names to below 156 characters in length then you should be OK on all providers.
There may be an even more secure file name encryption mode in the future which will address the long file name problem.
Never used this but thought that is what this was. Doesn’t seem to work the way I thought. Still encrypts. That only affects the file name.
Sorry for the misinformation. That being said, if you are worried about crypt crashing your machine, that is fiction. I use it everywhere.
The wording in configuring the crypt remote is indeed pretty… cryptic (no pun intended).
It all says “filename encryption” meaning that only the names are encrypted but it’s actually encrypting the files too. Made a feature request for a new remote just for obfuscating names on Github.
Even if encryption didn’t kill my machine, there’s just no way I’m re-uploading my 50TB with a 50 Mbit upload speed.
I agree that is tough. But. what you can do is spin up a FREE micro google compute which has free use and free ingress/egress data to/from google servers and do a sync with a band-width limit of 8.5 to keep under the 750 daily limit and move it. Then at the end, you do one final sync before switching over.
Yes, it will take over many days but you can run it and forget it for a while. I’ve done this many times with around 15TBs each time. Just keep the checkers/transfers at default and buffer so it is low resourced.
I created a Google Compute account and I can’t see any limits regarding bandwidth. I set up rclone and am doing a moveto between a jottacloud remote and a jottacloud crypted remote. Speed isn’t all that great though, about 9 MB/sec so around 70 Mbit. Would take two to three months to encrypt everything.
That is a single stream. You can get very fast speeds on them. but drive limits to 750/Gbit a day so you’d need to stay within that constraint to avoid a 24 hour ban. Thats why I said bw-limit.
unless jottacloud limits the speed of course. But you’ll pay for jotta cloud data… But usually you can sign up for a free trial account with a $300 credit of which you may use $10.
The data doesn’t seem to get stored on the VM though, looks like it just passes through for the encryption and then goes back. Disk space on the VM doesn’t change at all.
Jottacloud doesn’t have any limits on pretty much anything so I wouldn’t be paying extra. There’s not supposed to be a limit on their speed either but I guess that also depends on how much traffic they can handle.
you pay for ingress/egress data. Disk space isn’t an issue.
Any idea where I can check my usage so far? I looked under quotas but there’s nothing there that seems to refer to ingress/egress.