Performance expectations rcloning from dropbox to google drive

What is the problem you are having with rclone?

Performance when copying from dropbox to Google Drive.

2023-10-27 12:12:26 INFO  : Agora/fader-web-master/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/generated/Node.js: Copied (new)
2023-10-27 12:12:27 INFO  : Agora/fader-web-master/node_modules/jest-validate/build/validate.js: Copied (new)
2023-10-27 12:12:28 INFO  : Agora/sweetsummer/SSCL-passive-backup/node_modules/gulp/node_modules/gulp-cli/lib/versioned/^4.0.0-alpha.2/index.js: Copied (new)
2023-10-27 12:12:29 INFO  : prismarine/prismarine-web-client-github/node_modules/proxy-addr/package.json: Copied (new)
2023-10-27 12:12:29 INFO  : monoceros/survey-react/node_modules/object.fromentries/implementation.js: Copied (new)
2023-10-27 12:12:30 INFO  : Agora/fader-web-master/node_modules/jest-runtime/node_modules/debug/CHANGELOG.md: Copied (new)
2023-10-27 12:12:31 INFO  : Agora/fader-web-master/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/generated/NodeIterator.js: Copied (new)
2023-10-27 12:12:31 INFO  : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/public/textures/1.17.1/particle/sga_b.png: Copied (new)
2023-10-27 12:12:31 INFO  : Agora/fader-web-master/node_modules/jest-validate/build/validate_cli_options.js: Copied (new)
2023-10-27 12:12:33 INFO  : Agora/old/agora-app-builder 3/SweetSummerWP/node_modules/gulp/node_modules/yargs/lib/apply-extends.js: Copied (new)
2023-10-27 12:12:34 INFO  : Agora/old/agora-app-builder 2/TotalVU_Watchparty/node_modules/history/umd/history.js: Copied (new)
2023-10-27 12:12:35 INFO  : monoceros/survey-react/node_modules/object.fromentries/index.js: Copied (new)
2023-10-27 12:12:35 INFO  : watchparty/liveswitch-webrtc/watchparty/node_modules/fsevents/LICENSE: Copied (new)
2023-10-27 12:12:36 INFO  : Agora/fader-web-master/node_modules/jest-runtime/node_modules/debug/LICENSE: Copied (new)
2023-10-27 12:12:36 INFO  : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/public/textures/1.17.1/particle/sga_c.png: Copied (new)
2023-10-27 12:12:37 INFO  : Agora/fader-web-master/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/generated/NodeList.js: Copied (new)
2023-10-27 12:12:38 INFO  : Agora/fader-web-master/node_modules/jest-validate/build/warnings.js: Copied (new)
2023-10-27 12:12:38 INFO  : Agora/old/agora-app-builder 3/SweetSummerWP/node_modules/gulp/node_modules/yargs/lib/argsert.js: Copied (new)
2023-10-27 12:12:38 INFO  : monoceros/survey-react/node_modules/object.fromentries/package.json: Copied (new)
Transferred:   	   66.553 GiB / 66.703 GiB, 100%, 6.316 KiB/s, ETA 6h56m18s
Errors:              1310 (retrying may help)
Checks:            150515 / 150515, 100%
Transferred:      1120487 / 1130499, 99%
Elapsed time:  1w22h19m17.1s
Transferring:
 * Agora/old/agora-app-bu…ory/umd/history.min.js:100% /9.874Ki, 0/s, -
 * watchparty/liveswitch-…les/fsevents/README.md:100% /2.889Ki, 0/s, -
 * Agora/fader-web-master…modules/debug/Makefile:100% /1.034Ki, 0/s, -
 * prismarine/prismarine-…7.1/particle/sga_d.png:  0% /97, 0/s, -

We are getting only a few Kb per sec throughput.
Insights:

  • Speedtest reliably shows 300up and down
  • A full sync of this dropbox to the local machine takes around 24 hrs.
  • There doesn't seem to be any API limits being hit according to the Google Drive API dashboard on the Google Project
  • There are over 2 million files in the Dropbox, we are copying the largest folder which probably has 1 Million or more files.
  • I'm hoping to switch from dropbox to Google drive for this large folder which has many many git repos that are actively being used.
  • The host is a Mac Studio M2 with 64 GB ram on Ventura 13.5.2
  • The hard drive that is the backing store for the Google Drive and the dropbox is the same hard drive, an external USB drive. It is mounted like this:
    disk4 - WD easystore 264D 14.00 TB
    External USB 480 Mbit/s USB
    disk4s1 - EFI (MS-DOS FAT32) [EFI] 210 MB
    disk4s2 [APFS Container] 14.00 TB
    disk5 [APFS Virtual drive] 14.00 TB (Shared by 1 volumes)
    disk5s1 - 1********c (APFS) (8.04 TB used)
    Here is the ATTO benchmark on the drive:

Run the command 'rclone version' and share the full output of the command.

Are you on the latest version of rclone?

No the latest is 1.64.2, I'm on 1.64.0
rclone version
rclone v1.64.0
- os/version: darwin 13.5.2 (64 bit)
- os/kernel: 22.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.21.1
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Both Dropbox and Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --progress -v Dropbox:/Michael\ Williams/BITBUCKET-REPOS GDrive-TotalVU:/BITBUCKET-REPOS

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

MGW$ rclone config redacted
[Dropbox]
type = dropbox
chunk_size = 128Mi
batch_size = 100
token = XXX

[GDrive-TotalVU]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = 
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2023-10-27 12:20:58 INFO  : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/public/textures/1.18.1/particle/big_smoke_2.png: Copied (new)
2023-10-27 12:21:00 INFO  : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/node_modules/three/examples/jsm/nodes/core/NodeBuilder.js: Copied (new)
2023-10-27 12:21:00 INFO  : Agora/totalvu-ag-pass/node_modules/hermes-profile-transformer/dist/types/Phases.d.ts: Copied (new)
2023-10-27 12:21:00 INFO  : Agora/totalvu-ag-pass/node_modules/history/es/createMemoryHistory.js: Copied (new)
2023-10-27 12:21:03 INFO  : Agora/sweetsummer/SSPL-passive-b4-changes/node_modules/gulp/node_modules/wrap-ansi/package.json: Copied (new)
2023-10-27 12:21:03 INFO  : Agora/totalvu-ag/node_modules/hash-base/node_modules/readable-stream/lib/_stream_writable.js: Copied (new)
2023-10-27 12:21:03 INFO  : dekabeza/cms-new/node_modules/har-validator/node_modules/ajv/lib/dotjs/_limitProperties.js: Copied (new)
2023-10-27 12:21:03 INFO  : dekabeza/cms-new-bad/node_modules/har-validator/node_modules/ajv/lib/compile/resolve.js: Copied (new)
2023-10-27 12:21:05 INFO  : Agora/totalvu-ag-pass/node_modules/history/es/createTransitionManager.js: Copied (new)
2023-10-27 12:21:05 INFO  : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/node_modules/three/examples/jsm/nodes/core/NodeFrame.js: Copied (new)
2023-10-27 12:21:05 INFO  : Agora/totalvu-ag-pass/node_modules/hermes-profile-transformer/dist/types/SourceMap.d.ts: Copied (new)
2023-10-27 12:21:05 INFO  : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/public/textures/1.18.1/particle/big_smoke_3.png: Copied (new)
2023-10-27 12:21:07 INFO  : Agora/sweetsummer/SSPL-passive-b4-changes/node_modules/gulp/node_modules/wrap-ansi/readme.md: Copied (new)
2023-10-27 12:21:08 INFO  : dekabeza/cms-new/node_modules/har-validator/node_modules/ajv/lib/dotjs/allOf.js: Copied (new)
2023-10-27 12:21:09 INFO  : dekabeza/cms-new-bad/node_modules/handlebars/dist/cjs/handlebars/helpers/block-helper-missing.js: Copied (new)
2023-10-27 12:21:09 INFO  : dekabeza/cms-new-bad/node_modules/har-validator/node_modules/ajv/lib/compile/rules.js: Copied (new)
Transferred:   	   66.561 GiB / 66.710 GiB, 100%, 6.180 KiB/s, ETA 7h1m33s
Errors:              1310 (retrying may help)
Checks:            150515 / 150515, 100%
Transferred:      1121177 / 1131189, 99%
Elapsed time:  1w22h27m47.2s
Transferring:
 * dekabeza/cms-new-bad/n…v/lib/compile/rules.js:100% /1.743Ki, 892/s, 0s
 * prismarine/prismarine-…rticle/big_smoke_4.png:100% /177, 176/s, 0s
 * prismarine/prismarine-…/nodes/core/NodeLib.js:100% /870, 0/s, -
 * prismarine/prismarine-…/chainmail_layer_1.png: transferring

Google Drive sucks at creating files so you can only create about 2 per second so with a lot of small files, that is expected and will be your slow point.

@Animosity022 Thanks. But, how can anyone migrate to GDrive at that speed?
Is there a work around?

I'm not aware of any work around with rclone as I believe it's a limit of the API.

OK I see. Is there an alternative app then?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.