How would you recommend encrypting a large 70tb library that has already been uploaded to gdrive?
Sign up for Google Compute Engine and use your $300 free compute credits, choose a low spec VM (2 cores, 2gb RAM should suffice). Create the encrypted folder and then do a copy or sync to the encrypted folder using rclone. Something like rclone sync gdrive:/media crypt:. I'm not sure if the upload limitation of 750GB / day applies here, but if it does, you want to set either --max-transfer or --bwlimit.
Is creating a Vm on google still better for doing transefer between gdrives?
To encrypt you will need to download and upload again all your content. So this will take at least 140TB of data (70TB down, 70TB up) and you will face the 750GB per day upload limit.
That is true but as long as you dont activate a public IP address on the GCP you wont pay a dime for that network data. It will be free.
im seeing a lot about data and rates, but no mention of storage. this free tier google compute is unlimited storage? they dont mind 70tbs?
so 70tbs on free tier is fine?
you already have 70TB in gdrive, so you would be moving files from one bucket to another.
Correct, that's why KaMoS recommendation to use Google Compute Engine is right on point, he would only pay for cpu and ram used.
No, he would still be using140TB of TRANSFERRED data. He needs to download the 70TB, encrypt and reupload. I never spoke about storage capacity needed.
No, and you don't want to store it in Google Compute Engine. You would basically download and upload all data back to the same unlimited GDrive account you currently use. If you don't want to increase your usage to 140TB, using move instead of copy or sync would keep your storage use at 70TB. Yet, for all the time rclone would be syncing you would need to seek for data in both places as part will be encrypted and parts will not.
Either way it's something that is going to take a long time. (at least 96 days of nonstop syncing based purely on the 750GB upload limit, and nothing else being uploaded)
yes, good point
see what i dont understand dis how do you download something without storing it?
are you saying that i should copy from one gdrive folder to another?
because i can do that without cgoogle compute. i can do it from my dedicated server
if im downloading 70tb to google compute im storing it to a hard drive them moving or copying it back to gdrive
Yes basically you would be moving from remote location to remote location so no need to store it locally.
If you have a dedicated server you could use it to do the same. Yet speed wise the google compute would be faster as your dedicated server might not experience the same local access to Google servers as GCP. Downloading the content to any non unlimited platform will be pretty expensive, but you could do that as well, but personally I would not and take advantage of doing everything remotely and just move the data instead of a whole download and upload.
can you explain what this process is like?
Dont make it too complicated. rclone will simply stream down the unencrypted files and upload the encrypted ones on the fly. It will use network traffic to do this and wont store anything locally. While it streams the data down, it will encrypt it and stream it back up. It will use memory to do it. It will take a few months given the size of your data.
Something like this where remote is your unencrypted drive and remote-enc is your encrypted one:
rclone sync remote: remote-enc:
(or you can use move but then you'll have data in both spots until the entire process is complete)
70 TB cost about $ 2,000 only on hard drive. Maybe Google will close umlimited, as Amazon drive.
PD: only have 6 TB
1 Like
It's not needed at all if you aren't changing the files (like with encryption).
If not you can just use the server-side functionality available to Google systems via rclone. This will transfer files straight from one server to the other via Google and you don't have to use your own bandwith - which is obviously much much faster.
Unless you move the files however, it still counts towards the usual 750GB/day quota.
Just to be clear - this is not a solution for the problem posted by the OP however - since he plans to encrypt (change) the files. Therefore a server-side transfer is not possible (the servers can not do processing for you).
No you have very limited HD space on GCP (unless you pay for it).
That is not an issue however as the files won't need to be locally saved at all. It just passes through the RAM. (you should obviously not use mounts with write-caching enabled to avoid any local caching)
The only HDD space you need on the VM is enough to install a basic Linux distro + rclone, and a free-tier mini-instance is plenty for this job (although you do have a 1-year trial with 300USD credits you could use too - but it just won't help much here). The bandwidth on a mini-instance is very plentiful for the job, and since no data needs to be exported outside of the Google network in this case it is free. You only pay for export to outside networks.
In either case (doing it via GCP or at home) this operation does not need any local storage. The data is just piped in from the network and spit right out again after passing through the CPU and RAM to encrypt the file. Assuming that you don't use any caching systems the data is never saved locally. (for this reason, I recommend doing the operation via commandline rather than via mounts), plus commandline will be more robust anyway.
The main benefit to using a GCP mini-instance for the job is:
- You don't have to use your own bandwith
- You can prety easily automate the process. You might for example just use --bwlimit 8.5M to make it never hit the daily quota - and then just let it run until done. Or you could use --max-transfer 725G and have a simple script that runs once pr day. Basically, you don't need to have a machine of your own to run it, or have to remember to transfer your daily quota manually once pr. day - which would get annoying to do for 94 days straight...
If you have a home-server or other always-on system + plenty of bandwidth then you can just as well do it at home though. If you have no need to bypass these two limitations then this works just as well.
The operation would be simple:
- Set up a new remote for encrypted files
- Run an rclone move, copy or sync job (as you feel is most appropriate) from the old drive to the new. Using --bwlimit as described above to prevent hitting the limit and stalling the operation may be a good idea. The command would look something like this:
rclone copy UnencryptedDrive: EncryptedDrive: -P --bwlimit 8.5M --drive-chunk-size 64M
-P shows progress
--bwlimit I explained already
--drive-chunk-size helps with using bandwidth more efficiency when uploading compared to the default 8M chunk-size. Don't use a higher number than your system can easily fit in RAM. For example the default 4 transfers * 64M will need 256MB. If you use a mini-instance you have limited RAM (1GB), so make sure that's not going to be an issue if you do it that way, but I suspect 64M would probably be fine on a small linux setup without the need for a desktop environment.