My Google Enterprise plan provider is already kicking people out of our shared plan, as Google is now limiting storage at 5Tb per user. I need a fast and reliable way to migrate my 22Tb of data from G Drive to Dropbox Advance.
Run the command 'rclone version' and share the full output of the command.
rclone v1.52.2
os/arch: linux/amd64
go version: go1.14.4
(yes, I'll update it ASAP)
Which cloud storage system are you using? (eg Google Drive)
Google Drive
Dropbox Advance (unlimited storage for 3 users / month)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
hi, @Animosity022 might have some suggestions about migrating from gdrive to dropbox.
much depends on your internet speeds when transferring large amount of data up and down simultaneously.
once you have maxed that hard limit, tweaking rclone will not help.
i had a recent need to move a few TB from wasabi to aws s3 deep glacier.
i rented a cheap cloud vm, ran rclone on that.
as for your command, might test by increasing --transfers and --checkers
A few things things you should note when migrating to Dropbox:
If you want to transfer multiple files at once, it is recommended to take advantage of Dropbox's batch mode uploads (read up on it here)
10 TB is indeed the maximum threshold download from Google and you should never try to circumvent it (e.g. by using SA's) as that will get your API project banned (egress is expensive)
Assuming you want to push only 9 TB daily (to have 1 TB for other stuff like on-going streams etc.), these rclone params should get you going fairly quickly:
By limiting bw to 50 MB/s you're effectively not going over 9 TB daily. You can play around with this value, e. g. to give you a bit more room for other operations that eat up your download quota.
BTW: For the time of migrating, if you want to make sure that the rest of quota (1 TB) will be enough for any traffic, you could set up a fairly large VFS cache and allocate something like 200-300 GB.
true, that will yield potentially higher transfer speeds,
tho, at the major loss of full data integrity
as noted by rclone can't check the file got uploaded properly using this mode.
tho, after the initial sync, can run rclone check to find all the mismatches and feed that to rclone sync --files-from
Generally, my approach to things is simple as possible so I went that route. I made 2 other user accounts and used those for migration as each user gets a limit. I ran with 12 TPS / 8 transfers on the 2 users and just let it go. My process took a few weeks with Google quotas and for my case, I used rclone move.
I used mergerfs between all my remotes so locally, nothing seemed different.
What are the bandwidth limits on my links? Can I purchase more?
Dropbox doesn’t offer a way to buy more bandwidth allowance for your account.
To prevent abuse, Dropbox accounts have the following limits:
Basic accounts and accounts on a trial of Dropbox Business: 20 GB of bandwidth and 100,000 downloads per day
Plus, Family, and Professional accounts: 400 GB and unlimited downloads per day
Dropbox Business Standard accounts: 1 TB and unlimited downloads per day
Dropbox Business Advanced and Enterprise accounts: 4 TB and unlimited downloads per day
Both downloads work fine from my G Suite and Dropbox, I seem to have triggered the 4TB / day limit.
Anyone can confirm? @Animosity022?
Thanks in advance.
L
You're right (as always?), I'm above their 3TB first threshold. I missed that part, as I was looking for storage from the admin console, not the main account view.
After checking their forum, the only way to lift it is to contact support and ask for it, which I did.
I'll keep you guys posted.
L
Thanks for the information since it is perfect for me to transfer my content from drive to dropbox,
I have been using rclone for some time and now it occurs to me as the author of the post that I want to transfer my account to dropbox, but I have been doing tests and I would like to know if you encrypt the content or not, since I have done it in the standard mode and cases where there is a folder and long names I exceed the maximum number of characters that dropbox allows (over 160 in length.), therefore I have started to use obuscate because the standard is not valid in dropbox or someone explain me if I'm wrong.
Thanks
You're welcome.
As for the encryption, yes, I have a rclone crypt, standard too. I made some research here, " Use less than 260 characters in file or folder paths."
I guess I'll be far beyond the 260 characters if folder paths are included... yet they seem to be added into my storage space pretty fine.
What could be a solution here if we want to reduce the files' length?
Thanks in advance.
@Animosity022 , I've checked your rclone config on your github, you seem to have maintained standard encryption for your Dropbox remotes. No issues on your side for file paths above 260 characters? (if they even do get that high)
Okay thanks but in my case I'm not encrypting my files directly into Dropbox storage space.
Ideally I will for future contents, but right now I'm just copying encrypted files from GDrive to Dropbox.
Does that I mean that I should stop, change crypt settings (e.g. with base64 setting as you suggested) set for GDrive and only then copying encrypted files into Dropbox?
Just want to make sure that I am not transfering encrypted files the wrong way.
Any help welcome.
Thanks
The truth is that I cannot assure you with certainty since in my case it is the same, but I am doing tests and in cases of files with long names I still have some problem I am testing base64 in dropbox from encrypted drive, and passing it to my network to upload it in base64 And as you say, I'm not sure what the character limit is "Use less than 260 characters in file or folder paths".
I think the best thing will be to rectify some long file names and leave it in base 32 in both drive and dropbox to have the same in drive and dropbox.
Is there an expert who can clarify this question for us?
@deejayexe I seem to have transfered all contents from my Gdrive to Dropbox storage space without changing any encryption settings, although I'm pretty sure some of my folder paths are above the threshold set by Dropbox in their conditions.