What is the problem you are having with rclone?
Performance when copying from dropbox to Google Drive.
2023-10-27 12:12:26 INFO : Agora/fader-web-master/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/generated/Node.js: Copied (new)
2023-10-27 12:12:27 INFO : Agora/fader-web-master/node_modules/jest-validate/build/validate.js: Copied (new)
2023-10-27 12:12:28 INFO : Agora/sweetsummer/SSCL-passive-backup/node_modules/gulp/node_modules/gulp-cli/lib/versioned/^4.0.0-alpha.2/index.js: Copied (new)
2023-10-27 12:12:29 INFO : prismarine/prismarine-web-client-github/node_modules/proxy-addr/package.json: Copied (new)
2023-10-27 12:12:29 INFO : monoceros/survey-react/node_modules/object.fromentries/implementation.js: Copied (new)
2023-10-27 12:12:30 INFO : Agora/fader-web-master/node_modules/jest-runtime/node_modules/debug/CHANGELOG.md: Copied (new)
2023-10-27 12:12:31 INFO : Agora/fader-web-master/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/generated/NodeIterator.js: Copied (new)
2023-10-27 12:12:31 INFO : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/public/textures/1.17.1/particle/sga_b.png: Copied (new)
2023-10-27 12:12:31 INFO : Agora/fader-web-master/node_modules/jest-validate/build/validate_cli_options.js: Copied (new)
2023-10-27 12:12:33 INFO : Agora/old/agora-app-builder 3/SweetSummerWP/node_modules/gulp/node_modules/yargs/lib/apply-extends.js: Copied (new)
2023-10-27 12:12:34 INFO : Agora/old/agora-app-builder 2/TotalVU_Watchparty/node_modules/history/umd/history.js: Copied (new)
2023-10-27 12:12:35 INFO : monoceros/survey-react/node_modules/object.fromentries/index.js: Copied (new)
2023-10-27 12:12:35 INFO : watchparty/liveswitch-webrtc/watchparty/node_modules/fsevents/LICENSE: Copied (new)
2023-10-27 12:12:36 INFO : Agora/fader-web-master/node_modules/jest-runtime/node_modules/debug/LICENSE: Copied (new)
2023-10-27 12:12:36 INFO : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/public/textures/1.17.1/particle/sga_c.png: Copied (new)
2023-10-27 12:12:37 INFO : Agora/fader-web-master/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/generated/NodeList.js: Copied (new)
2023-10-27 12:12:38 INFO : Agora/fader-web-master/node_modules/jest-validate/build/warnings.js: Copied (new)
2023-10-27 12:12:38 INFO : Agora/old/agora-app-builder 3/SweetSummerWP/node_modules/gulp/node_modules/yargs/lib/argsert.js: Copied (new)
2023-10-27 12:12:38 INFO : monoceros/survey-react/node_modules/object.fromentries/package.json: Copied (new)
Transferred: 66.553 GiB / 66.703 GiB, 100%, 6.316 KiB/s, ETA 6h56m18s
Errors: 1310 (retrying may help)
Checks: 150515 / 150515, 100%
Transferred: 1120487 / 1130499, 99%
Elapsed time: 1w22h19m17.1s
Transferring:
* Agora/old/agora-app-bu…ory/umd/history.min.js:100% /9.874Ki, 0/s, -
* watchparty/liveswitch-…les/fsevents/README.md:100% /2.889Ki, 0/s, -
* Agora/fader-web-master…modules/debug/Makefile:100% /1.034Ki, 0/s, -
* prismarine/prismarine-…7.1/particle/sga_d.png: 0% /97, 0/s, -
We are getting only a few Kb per sec throughput.
Insights:
- Speedtest reliably shows 300up and down
- A full sync of this dropbox to the local machine takes around 24 hrs.
- There doesn't seem to be any API limits being hit according to the Google Drive API dashboard on the Google Project
- There are over 2 million files in the Dropbox, we are copying the largest folder which probably has 1 Million or more files.
- I'm hoping to switch from dropbox to Google drive for this large folder which has many many git repos that are actively being used.
- The host is a Mac Studio M2 with 64 GB ram on Ventura 13.5.2
- The hard drive that is the backing store for the Google Drive and the dropbox is the same hard drive, an external USB drive. It is mounted like this:
disk4 - WD easystore 264D 14.00 TB
External USB 480 Mbit/s USB
disk4s1 - EFI (MS-DOS FAT32) [EFI] 210 MB
disk4s2 [APFS Container] 14.00 TB
disk5 [APFS Virtual drive] 14.00 TB (Shared by 1 volumes)
disk5s1 - 1********c (APFS) (8.04 TB used)
Here is the ATTO benchmark on the drive:
Run the command 'rclone version' and share the full output of the command.
Are you on the latest version of rclone?
No the latest is 1.64.2, I'm on 1.64.0
rclone version
rclone v1.64.0
- os/version: darwin 13.5.2 (64 bit)
- os/kernel: 22.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.21.1
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Both Dropbox and Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy --progress -v Dropbox:/Michael\ Williams/BITBUCKET-REPOS GDrive-TotalVU:/BITBUCKET-REPOS
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
MGW$ rclone config redacted
[Dropbox]
type = dropbox
chunk_size = 128Mi
batch_size = 100
token = XXX
[GDrive-TotalVU]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive =
### Double check the config for sensitive info before posting publicly
A log from the command that you were trying to run with the -vv flag
2023-10-27 12:20:58 INFO : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/public/textures/1.18.1/particle/big_smoke_2.png: Copied (new)
2023-10-27 12:21:00 INFO : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/node_modules/three/examples/jsm/nodes/core/NodeBuilder.js: Copied (new)
2023-10-27 12:21:00 INFO : Agora/totalvu-ag-pass/node_modules/hermes-profile-transformer/dist/types/Phases.d.ts: Copied (new)
2023-10-27 12:21:00 INFO : Agora/totalvu-ag-pass/node_modules/history/es/createMemoryHistory.js: Copied (new)
2023-10-27 12:21:03 INFO : Agora/sweetsummer/SSPL-passive-b4-changes/node_modules/gulp/node_modules/wrap-ansi/package.json: Copied (new)
2023-10-27 12:21:03 INFO : Agora/totalvu-ag/node_modules/hash-base/node_modules/readable-stream/lib/_stream_writable.js: Copied (new)
2023-10-27 12:21:03 INFO : dekabeza/cms-new/node_modules/har-validator/node_modules/ajv/lib/dotjs/_limitProperties.js: Copied (new)
2023-10-27 12:21:03 INFO : dekabeza/cms-new-bad/node_modules/har-validator/node_modules/ajv/lib/compile/resolve.js: Copied (new)
2023-10-27 12:21:05 INFO : Agora/totalvu-ag-pass/node_modules/history/es/createTransitionManager.js: Copied (new)
2023-10-27 12:21:05 INFO : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/node_modules/three/examples/jsm/nodes/core/NodeFrame.js: Copied (new)
2023-10-27 12:21:05 INFO : Agora/totalvu-ag-pass/node_modules/hermes-profile-transformer/dist/types/SourceMap.d.ts: Copied (new)
2023-10-27 12:21:05 INFO : prismarine/prismarine-web-client-old/node_modules/prismarine-viewer/public/textures/1.18.1/particle/big_smoke_3.png: Copied (new)
2023-10-27 12:21:07 INFO : Agora/sweetsummer/SSPL-passive-b4-changes/node_modules/gulp/node_modules/wrap-ansi/readme.md: Copied (new)
2023-10-27 12:21:08 INFO : dekabeza/cms-new/node_modules/har-validator/node_modules/ajv/lib/dotjs/allOf.js: Copied (new)
2023-10-27 12:21:09 INFO : dekabeza/cms-new-bad/node_modules/handlebars/dist/cjs/handlebars/helpers/block-helper-missing.js: Copied (new)
2023-10-27 12:21:09 INFO : dekabeza/cms-new-bad/node_modules/har-validator/node_modules/ajv/lib/compile/rules.js: Copied (new)
Transferred: 66.561 GiB / 66.710 GiB, 100%, 6.180 KiB/s, ETA 7h1m33s
Errors: 1310 (retrying may help)
Checks: 150515 / 150515, 100%
Transferred: 1121177 / 1131189, 99%
Elapsed time: 1w22h27m47.2s
Transferring:
* dekabeza/cms-new-bad/n…v/lib/compile/rules.js:100% /1.743Ki, 892/s, 0s
* prismarine/prismarine-…rticle/big_smoke_4.png:100% /177, 176/s, 0s
* prismarine/prismarine-…/nodes/core/NodeLib.js:100% /870, 0/s, -
* prismarine/prismarine-…/chainmail_layer_1.png: transferring