Hi - I'm wondering if rclone makes sense for my use case - I have over 20M files taking ~200Gb to backup from a server , ideally to dropbox since I'm already paying for several Tb there anyway. dropbox can't seem to handle this number of files, i know rsync would prb be fine - so can rclone bridge the gap?
that is super slow, is that a DSL connection?
what is the result of a speedtest?
rclone is well known for uploading as fast as a provider and internet connection can handle.
i have no issue doing that with my 1Gbps fiber optic internet connection.
Thanks for reply. I imagine the script does support batch , but iiuc the rclone will do the driving on its own , i guess instead of the dropbox daemon ? anyway i should rather have asked if rclone will be limited by dropbox's extremely slow indexing (or whatever it is that's slow) - i should have decent network speeds when needed, its a digitalocean server so i blv i just pay for whatever bandwidth i use up to some large hardware limitation
The dropbox client had a soft limit at around 500k files before it starts to crumble. Rclone or dropbox as a service does not have any such limits.
Your scenario should be fine with rclone.
Edit: given the huge amount of small files, and the fact that you are saying it is a backup, some local tar:ing before uploading with rclone could drastically reduce your upload time here.
When uploading lots of small files, check out the batch mode options. async is the one you want with batch size set to 1000.
--dropbox-batch-mode sync
In this mode rclone will batch up uploads to the size specified by --dropbox-batch-size and commit them together.
Using this mode means you can use a much higher --transfers parameter (32 or 64 works fine) without receiving too_many_requests errors.
This mode ensures full data integrity.
Note that there may be a pause when quitting rclone while rclone finishes up the last batch using this mode.
--dropbox-batch-mode async
In this mode rclone will batch up uploads to the size specified by --dropbox-batch-size and commit them together.
However it will not wait for the status of the batch to be returned to the caller. This means rclone can use a much bigger batch size (much bigger than --transfers), at the cost of not being able to check the status of the upload.
This provides the maximum possible upload speed especially with lots of small files, however rclone can't check the file got uploaded properly using this mode.
If you are using this mode then using "rclone check" after the transfer completes is recommended. Or you could do an initial transfer with --dropbox-batch-mode async then do a final transfer with --dropbox-batch-mode sync (the default).
Note that there may be a pause when quitting rclone while rclone finishes up the last batch using this mode.
Ok I'll try the batch - in the meantime I've hit some trouble doing the config which seems to be different in my case than the docs .
> root@master:~/Dropbox# rclone config
> 2022/02/10 02:08:01 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
> No remotes found - make a new one
> n) New remote
> s) Set configuration password
> q) Quit config
> n/s/q> n
> name> dropbox_remote
> Option Storage.
> Type of storage to configure.
> Enter a string value. Press Enter for the default ("").
> Choose a number from below, or type in your own value.
**[already ambiguous - am i supposed to choose a number or type a string, or either is ok ? ]**
...
> 11 / Dropbox
> \ "dropbox"
> Storage> dropbox
> Option client_id.
> OAuth Client Id.
> Leave blank normally.
> Enter a string value. Press Enter for the default ("").
> client_id>
> Option client_secret.
> OAuth Client Secret.
> Leave blank normally.
> Enter a string value. Press Enter for the default ("").
> client_secret>
> Edit advanced config?
> y) Yes
> n) No (default)
righto again, cheers. This is a great tool and just wht the doctor ordered in my case.
Just being able to see what's happening is already a vast improvement to the dropbox daemon.
I still seem to have slower uploads than what dropbox and my connection would allow, after setting async and batchsize 1000 but possibly skipping over a lot of files (I requested not changing modification times in interactive mode) is costing me in speed.
root@master:~# speedtest-cli
Retrieving speedtest.net configuration...
Testing from Digital Ocean (a.b.c.d)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Webe Digital Sdn Bhd (Petaling Jaya) [299.90 km]: 7.269 ms
Testing download speed................................................................................
Download: 811.00 Mbit/s
Testing upload speed......................................................................................................
Upload: 701.21 Mbit/s
root@master:~# wget www.dropbox.com
URL transformed to HTTPS due to an HSTS policy
--2022-02-10 12:48:21-- https://www.dropbox.com/
Resolving www.dropbox.com (www.dropbox.com)... 2620:100:6031:18::a27d:5112, 162.125.81.18
Connecting to www.dropbox.com (www.dropbox.com)|2620:100:6031:18::a27d:5112|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’
index.html [ <=> ] 525.90K --.-KB/s in 0.05s
2022-02-10 12:48:22 (10.1 MB/s) - ‘index.html’ saved [538517]
If you post a snippet, I have no idea what you ran so it's really impossible to figure out how to answer your question.
For any question, the template really helps out as that collects all the right information and helps to keep a post for a question as it just gets confusing.
