We have more than 100TB of data in Dropbox and planning to migrate data to Google Drive. The data contains mix of everything like users folders, shared folders..etc. I have read the old posts and to the continuation of that have few questions:
We have 1000 of users So, to copy each user data do we need to run "rclone config" to impersonate them? --dropbox-impersonate
Does rclone create the directory structure in Google drive and take care of metadata like user permissions?
Are there any limitations with rclone in terms of this migration?
Run the command 'rclone version' and share the full output of the command.
rclone v1.64.2
Yes
Which cloud storage system are you using? (eg Google Drive)
short answer is yes, so I worked on something comparable and ran into a specific issue - depending on your dropbox setup it is not always easy to see user folders properly - what I mean is - if you impersonate a user you will see multiple types of files/folders:
1 - stuff that's only in their account
2 - stuff they are sharing with other users (if you allow this)
3 - teams folders they are a part of
rclone does have capabilities to filter some of this - but I ended up duplicating a great deal of content I realized - as it wasn't possible with just rclone to differentiate some of these folder types.
if you have your user folders locked down you can just create a master filter list of known teams folders so they are excluded as a starting poiint
I am trying to find my notes, but what I started working on is using this tool:
to generate activity reports (which takes a LONG time) and doing fancy grep/awk parsing to find folders users were not sharing with anyone
as one example of an issue that came to mind - let's say you ave a top level team folder call FOLDER1 - well, that's easy enough to filter for - the problem is some users were given access to different depths within this folder structure - and there is no direct way to see this from the user impersonation perspective.
also as a reminder, google only allows 750 GB of uploads per 24 hour period.
Thanks for the details. I have never used toolbox we will definitely explore this.
Could you share more details about your migration? Like how did you deal with different kinds of data?
How long does it take to complete your migration?
Dropbox didn't publish download API limits anywhere so I was thinking of something like copy entire dropbox content to Google cloud storage and then copy from there to Google drive. With this approach, not sure if user permissions and other metadata will be preserved.
Unfortunately the project was abandoned around the time I ran into these issues - the direction I was going was dumping activity logs via tbx and parsing for folder lists - there are API limits but they are quite high I remember testing large transfers and it barely made a dent in the Dropbox developer portal for the “app” I created for rclone