Accelerating migration GDrive to Dropbox

What is the problem you are having with rclone?

My Google Enterprise plan provider is already kicking people out of our shared plan, as Google is now limiting storage at 5Tb per user. I need a fast and reliable way to migrate my 22Tb of data from G Drive to Dropbox Advance.

Run the command 'rclone version' and share the full output of the command.

rclone v1.52.2

  • os/arch: linux/amd64
  • go version: go1.14.4
    (yes, I'll update it ASAP)

Which cloud storage system are you using? (eg Google Drive)

Google Drive
Dropbox Advance (unlimited storage for 3 users / month)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync plex-gdrive: plex-dropbox: -P --checkers 3 --log-file /home/user/rclone_migration.log -v --tpslimit 3 --transfers 2 --drive-chunk-size 32M --max-transfer 10000G --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"

N.B:

  • Why 10000G? It's seems to be the maximum threshold download from Google.
  • Why 'sync' and not 'copy'? I was tempted to do 'copy' but I feel I'll have to run this script several times, I'm depending on my seedbox provider.

A log from the command with the -vv flag

45.237G / 16.831 TBytes, 0%, 19.284 MBytes/s, ETA 1w3d13h33m12s

Any suggestions / advices that would help me beat the clock would truly be appreciated.
Thanks in advance.

2 Likes

hi,
@Animosity022 might have some suggestions about migrating from gdrive to dropbox.

much depends on your internet speeds when transferring large amount of data up and down simultaneously.
once you have maxed that hard limit, tweaking rclone will not help.

i had a recent need to move a few TB from wasabi to aws s3 deep glacier.
i rented a cheap cloud vm, ran rclone on that.

as for your command, might test by increasing --transfers and --checkers

and this topic might be a good read.
https://forum.rclone.org/t/sync-not-removing-deleted-files/30865

Hi LLuwen,

A few things things you should note when migrating to Dropbox:

  • If you want to transfer multiple files at once, it is recommended to take advantage of Dropbox's batch mode uploads (read up on it here)
  • 10 TB is indeed the maximum threshold download from Google and you should never try to circumvent it (e.g. by using SA's) as that will get your API project banned (egress is expensive)

Assuming you want to push only 9 TB daily (to have 1 TB for other stuff like on-going streams etc.), these rclone params should get you going fairly quickly:

rclone sync plex-gdrive: plex-dropbox: --transfers 12 --checkers 12 --tpslimit 12 --dropbox-batch-mode async --fast-list --checksum --dropbox-batch-size 1000 --dropbox-batch-timeout 10s --bwlimit 50M:60M --progress

By limiting bw to 50 MB/s you're effectively not going over 9 TB daily. You can play around with this value, e. g. to give you a bit more room for other operations that eat up your download quota.

BTW: For the time of migrating, if you want to make sure that the rest of quota (1 TB) will be enough for any traffic, you could set up a fairly large VFS cache and allocate something like 200-300 GB.

hello and welcome to the fourm,

true, that will yield potentially higher transfer speeds,
tho, at the major loss of full data integrity
as noted by
rclone can't check the file got uploaded properly using this mode.
tho, after the initial sync, can run rclone check to find all the mismatches and feed that to rclone sync --files-from

1 Like

Generally, my approach to things is simple as possible so I went that route. I made 2 other user accounts and used those for migration as each user gets a limit. I ran with 12 TPS / 8 transfers on the 2 users and just let it go. My process took a few weeks with Google quotas and for my case, I used rclone move.

I used mergerfs between all my remotes so locally, nothing seemed different.

Thanks guys, much appreciated.

So far, I tweaked the "safest" command in my case below:

rclone sync plex-gdrive: plex-dropbox: -P --checkers 12 --log-file /home/user/rclone_gdropbox.log -v --tpslimit 12 --transfers 8 --max-transfer 9000G --drive-chunk-size 32M --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"

I managed to transfer 4Tb in around 10-12 hours but got hit by the following error messages:

2022/05/21 11:19:10 INFO  : Starting HTTP transaction limiter: max 12 transactions/s with burst 1
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/9f2gqat2ukb01el18amrblo0ks/dgtt3gfvvvu84qhao7b45of8q04fora3l1eh2cdiclbn5g5ng1v8m4toc2v03r5l5cb053ggfqvjbukg8kunc0j12d7mcslvknshf60/0egsh2vrqa93b23u3l8kvcnupc: Failed to copy: upload failed: path/insufficient_space/.
2022/05/21 11:19:22 ERROR : Cancelling sync due to fatal error: upload failed: path/insufficient_space/.
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/9f2gqat2ukb01el18amrblo0ks/e1mqkqcddqbd8ccojgjnnek73inn906ggio22g5vorcebtisj74hebeq3v2fn3egr7mj5dc38bt3c/ccnl0iej50ano877irhk2cu01v5ijm9lpe97ju34elesrrujnsp9vv80fht9jndu81b5f2718nth1nothq6pa3g9rjdam2ufdot22f8: Failed to copy: failed to open source object: open file failed: Get "https://www.googleapis.com/drive/v3/files/1MZNGkv6mGMWnxiZUsssIuY5HpoQb-EnE?alt=media": context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/jbiju68kj8pagrop4qpf3ef7j4/dgd7675bgoq1786gfoesftlcbg/ce1qjurl7qft0thuq6bkot58elgi65k5n8erkvqqft6qkgo9ukt3baf3ides7iuhcdu6fs0gr6p9m/n4hkosoqib38282941gab6et2190522nvr8abf1rh3kjfmkfb8tcvit4etqq0807brh2ueomkp4e9jebqe4jrejcngk1jqt7bq7jc9o: Failed to copy: failed to open source object: open file failed: Get "https://www.googleapis.com/drive/v3/files/16zSzQRqvwvva_tmer_riyVYhqPAEFFnz?alt=media": context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/qfsartgdcn5s18f7h2v93m1t98/14d2j10t9ks7uhplapg7dr3kb870q613lrd2orso9r9243ine310: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/9f2gqat2ukb01el18amrblo0ks/ebf2478ikelg2jhgs1po0k86fecuc2nkkfumn438jrfhechd3gt7kt7jeqkv9pdmt1k7splnr3aqo: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/i45vbc953tduoidmrken5tfv8g/6q3kbn4roohp5fct5enu9q70v0: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/jbiju68kj8pagrop4qpf3ef7j4/dgd7675bgoq1786gfoesftlcbg/dm6u1t2ash0hjgfn0h2f34lm6ss0i72428mhhui60do29376rv28t70796qt7vq4ltdiake2ih3eq: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/9f2gqat2ukb01el18amrblo0ks/dsjdk1gfohr5jth6trkk984s88p5pjm4upqalpkk9295apimj30va3h6ram7n0rvf8diojbs5m1pa/5l938nurp84947pv0knirn29fbsmn4nnbjp61j6kip0lnqqvdco3cj38bdu5l1a1ih8f0tpkd22as06gqupj66g8q5hvqbps1oip8p8: Failed to copy: failed to open source object: open file failed: Get "https://www.googleapis.com/drive/v3/files/16LwR4ruUDvF3kOCbh6kY4bsgo5UVrbTW?alt=media": context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/qfsartgdcn5s18f7h2v93m1t98/17p8ehsutmuondu9pf9291ubtf9sihs1tp5blnkktcmd5qcjuvg0: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/9f2gqat2ukb01el18amrblo0ks/dgtt3gfvvvu84qhao7b45of8q04fora3l1eh2cdiclbn5g5ng1v8m4toc2v03r5l5cb053ggfqvjbukg8kunc0j12d7mcslvknshf60/s35h80lokg8sac7nkjvn9dh0p40u9obje5pprvp2b9ncb1srj52jugt4peiurkoqqf1q92q7f693vs3t3nkrhk2l7lf3ib5mpt5i46nob1oooupn4a18qrr0ibddr696mdqhdlllh14b3384tgma9fclt4: Failed to copy: upload failed: Post "https://content.dropboxapi.com/2/files/upload_session/start": context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/9f2gqat2ukb01el18amrblo0ks/dgtt3gfvvvu84qhao7b45of8q04fora3l1eh2cdiclbn5g5ng1v8m4toc2v03r5l5cb053ggfqvjbukg8kunc0j12d7mcslvknshf60/4719nobd6tp3ff0q6q47a23clo: Failed to copy: upload failed: Post "https://content.dropboxapi.com/2/files/upload": context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/qtfuehu869ssf1veql4f6hr5b0/3e2lvmfrlevfgple6jido92sv7u06lpv0o1na9oqqqsf2himkjgg: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/euauvnb9bmog8gi73af0ki3jds/feam6tge6av8kanigjvf3q69f4h7af9kq4fg1qeqn520i12og3b0: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:22 ERROR : fi0sk7froa06jqlnli6fngp27g/9f2gqat2ukb01el18amrblo0ks/dgtt3gfvvvu84qhao7b45of8q04fora3l1eh2cdiclbn5g5ng1v8m4toc2v03r5l5cb053ggfqvjbukg8kunc0j12d7mcslvknshf60/cehuko9ba271c4deivtde978qk: Failed to copy: upload failed: Post "https://content.dropboxapi.com/2/files/upload": context canceled
2022/05/21 11:19:23 ERROR : fi0sk7froa06jqlnli6fngp27g/p1mldnomjgpirbj12ddhek7c14/ettqdgoeceod3oj88t9eeprl8i9p4miberndjdilact6e267lc8edk21dakfg0i9j8rgbemakggmm5h36ksomi6jta0r69img99kemrcbukri1kcqf8ghm2d2e0eliqu/6qbm0hidp10ojuuohhpthm2hpc: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:23 ERROR : fi0sk7froa06jqlnli6fngp27g/jbiju68kj8pagrop4qpf3ef7j4/mo2ejgcutqg5pmr8tm6r1bqujnmr05qd7uvn65ndk3h8gui7c5e0/vrt5o3cpl4so11oj45oab89e4c: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:23 ERROR : fi0sk7froa06jqlnli6fngp27g/9f2gqat2ukb01el18amrblo0ks/dohqdnmv79qsd83h9bdt8sqib4m94esjhcs06jqugsmbus6v504trh5tjt7nk6827j557gcb5lj74cabdv5mmk801s4p9t6m9ifddnn8p1svj1ame85vmsb4ou7i4g3ecnes0pv1umb8rdal4gaavkghg8/soop62e6s3f1kifahb3tue9316kj5pmjo17031hcs2rmua0rhafbcp1evb82nvht0ejvlam8gi6v7iqda82vb8pf9g56cc0ed3oee8n1qre7vuokbbdgki7t1iads0ti0qjv2i7c7saa9ir9rja95480e2lcnqci1gsojpqn6ni4dv03ecd0: Failed to copy: upload failed: Post "https://content.dropboxapi.com/2/files/upload_session/start": context canceled
2022/05/21 11:19:23 ERROR : fi0sk7froa06jqlnli6fngp27g/jbiju68kj8pagrop4qpf3ef7j4/2q1schbslrh9qu1c3dnvshdjrg/8c4092920g8i5ucpoqt9b6p2ckod9kin0tk6akj5rmf299592bvs6cbi8961jfhko5bo60j1ng8lcdbc5epsot10ruvka7ltjr1nj18/0b59fm9k87nkng4lcr6lesg3ll5ap6a38nv7hoet0hleq3aiqd10: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:23 ERROR : fi0sk7froa06jqlnli6fngp27g/qtfuehu869ssf1veql4f6hr5b0/0o9rk1o2696r9nvtnka1ajkrheuhjc9k7oh8kbacmbjm1vt6v490/65q2ud3745q4rscbn5avsf6gu8: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:24 ERROR : fi0sk7froa06jqlnli6fngp27g/qtfuehu869ssf1veql4f6hr5b0/3oh7cdc0r07gf45k27lmebjr5jce1668qf53vlgp5cu6gkp3vllg: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:24 ERROR : fi0sk7froa06jqlnli6fngp27g/p1mldnomjgpirbj12ddhek7c14/ik167q1mo4o95t0k2625gdmifbgjlrb6m489ebnde5eaffov82f7egqtsgs0451pufbj0m2cio6f2vv9qurbrg5n56fq45ahgpo3pf0/6qbm0hidp10ojuuohhpthm2hpc: error reading source directory: couldn't list directory: context canceled
2022/05/21 11:19:24 ERROR : Dropbox root '': not deleting files as there were IO errors
2022/05/21 11:19:24 ERROR : Dropbox root '': not deleting directories as there were IO errors
2022/05/21 11:19:24 INFO  : There was nothing to transfer
2022/05/21 11:19:24 ERROR : Fatal error received - not attempting retries
2022/05/21 11:19:24 INFO  : 
Transferred:   	   39.215M / 39.215 MBytes, 100%, 7.987 MBytes/s, ETA 0s
Errors:                21 (fatal error encountered)
Checks:               364 / 364, 100%
Elapsed time:         4.9s

None of the dropbox links in the log works (404 Error).

After checking Dropbox's T&Cs, I found this:

What are the bandwidth limits on my links? Can I purchase more?

Dropbox doesn’t offer a way to buy more bandwidth allowance for your account.

To prevent abuse, Dropbox accounts have the following limits:

    Basic accounts and accounts on a trial of Dropbox Business: 20 GB of bandwidth and 100,000 downloads per day
    Plus, Family, and Professional accounts: 400 GB and unlimited downloads per day
    Dropbox Business Standard accounts: 1 TB and unlimited downloads per day 
    Dropbox Business Advanced and Enterprise accounts: 4 TB and unlimited downloads per day 

Both downloads work fine from my G Suite and Dropbox, I seem to have triggered the 4TB / day limit.
Anyone can confirm? @Animosity022?
Thanks in advance.
L

Looks like you ran out of space.

The T&C you linked are for links and has nothing to do with the space error message.

Thanks but how is it possible? How can I fix it please?

I have a Business Advanced account with 3 licences, so logically unlimited storage available. I’m still in trial version though.

Plus I’ve added my admin account as “exception” for storage limit.

Check your storage. If you are out, ask support for more.

You're right (as always?), I'm above their 3TB first threshold. I missed that part, as I was looking for storage from the admin console, not the main account view.

After checking their forum, the only way to lift it is to contact support and ask for it, which I did.
I'll keep you guys posted.
L

I got a bulk add done when I first signed up in my trial period to handle my entire migration before I signed up.

Since then, once I get close every few months, I ask for more. I did it all via chat support without any issue ever.

The minor adds are done in seconds via support.

Hello.

Thanks for the information since it is perfect for me to transfer my content from drive to dropbox,
I have been using rclone for some time and now it occurs to me as the author of the post that I want to transfer my account to dropbox, but I have been doing tests and I would like to know if you encrypt the content or not, since I have done it in the standard mode and cases where there is a folder and long names I exceed the maximum number of characters that dropbox allows (over 160 in length.), therefore I have started to use obuscate because the standard is not valid in dropbox or someone explain me if I'm wrong.
Thanks

Hi,

You're welcome.
As for the encryption, yes, I have a rclone crypt, standard too. I made some research here, " Use less than 260 characters in file or folder paths."
I guess I'll be far beyond the 260 characters if folder paths are included... yet they seem to be added into my storage space pretty fine.

What could be a solution here if we want to reduce the files' length?
Thanks in advance.

@Animosity022 , I've checked your rclone config on your github, you seem to have maintained standard encryption for your Dropbox remotes. No issues on your side for file paths above 260 characters? (if they even do get that high)

I see with this flag in last version of rclone help for short characters -> --crypt-filename-encoding in "base64"

Encode using base64. Suitable for case sensitive remote.

I testing it and updated my rclone setup and i think it is better than default base32

Okay thanks but in my case I'm not encrypting my files directly into Dropbox storage space.
Ideally I will for future contents, but right now I'm just copying encrypted files from GDrive to Dropbox.

Does that I mean that I should stop, change crypt settings (e.g. with base64 setting as you suggested) set for GDrive and only then copying encrypted files into Dropbox?

Just want to make sure that I am not transfering encrypted files the wrong way.
Any help welcome.
Thanks

The truth is that I cannot assure you with certainty since in my case it is the same, but I am doing tests and in cases of files with long names I still have some problem I am testing base64 in dropbox from encrypted drive, and passing it to my network to upload it in base64 And as you say, I'm not sure what the character limit is "Use less than 260 characters in file or folder paths".
I think the best thing will be to rectify some long file names and leave it in base 32 in both drive and dropbox to have the same in drive and dropbox.
Is there an expert who can clarify this question for us?

@deejayexe I seem to have transfered all contents from my Gdrive to Dropbox storage space without changing any encryption settings, although I'm pretty sure some of my folder paths are above the threshold set by Dropbox in their conditions.

How's it going on your end? Any need for help?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.