Tried the new version, this is the output:
At that point it printed:
Transferred: 92.542M / 335.137 MBytes, 28%, 3.573 MBytes/s, ETA 1m7s
Transferred: 22 / 202, 11%
But nothing actually appeared on my Dropbox
This batch approach is really interesting for me in one of my teams current activities, as we are struggling with uploading many smaller files to Dropbox where we seem to be hitting the pacer no matter what limits we try. Would you say this beta is "stable" enough now to use for a production sync of mostly stale data?
Or would it give value if I were to chip in and help with some specific testing of it?
/Magnus
I found a bug in the code which wasn't commiting the last batch when using async mode.
Try this
v1.55.0-beta.5334.ac7acb9f0.fix-dropbox-batch-sync on branch fix-dropbox-batch-sync (uploaded in 15-30 mins)
Note that othing will appear on your dropbox until the batch is comitted which will happen
- at the end of the sync
- when there are more than --dropbox-batch-size items in the batch
- when nothing has been written to the batch for longer than --dropbox-batch-timeout
If you want the batch to complete quicker you can fiddle with those parameters.
I'm hoping to launch it in 1.56 so it should be stable!
Its failure mode is to not commit items, so if it fails there will be items missing. This is easily detectable with rclone check or by running the sync again.
Yes please more testing would be great!
It is very much faster, and if you use --dropbox-batch-mode async even faster (with less guarantees). See this comment on the github issue for performance testing and pointers to the docs.
This is the output with this new version:
Only 156 out of 202 files were actually copied to my Dropbox
This is the problem
2021/04/09 18:30:41 ERROR : Dropbox root '': async batch commit: failed to commit batch length 46: batch had 46 errors: last error: too_many_write_operations
Which agrees with your test results. I don't understand why the batch commit failed like that though.
I presume your files aren't all really called img.jpg?
You've got 202 files totalling about 330M. Are they all about the same size? Also are they all in one directory? I'll try to recreate your problem here with some similar files.
Are you doing any other uploads to Dropbox at the same time? You are only allowed one batch per namespace.
I presume your files aren't all really called img.jpg?
No, they all have different names
Are they all about the same size? Also are they all in one directory?
Yes and yes
Are you doing any other uploads to Dropbox at the same time?
Yes, I do 4 concurrent uploads to Dropbox of videos I scrape. They all are in different processes but they're using stable rclone 1.55.0
That will be the problem. The dropbox upload system really hates people doing simultaneous uploads. Can you pause or stop the other uploads and try your test without them running?
In particular you can only upload to one namespace at once without getting lots of this error: DBX Team Files Guide - Dropbox
I paused the 4 concurrent uploads I had and tried again. This is the output with the latest beta version:
It froze for quite a few seconds and was very slow to upload, it printed "waiting for 10 seconds" or something but this time all the 202 files were uploaded to my Dropbox
2021-04-11 01:11:22 INFO : img.jpg: Copied (new)
2021-04-11 01:11:35 DEBUG : Dropbox root '': Batch idle for 10s so committing
2021-04-11 01:12:00 DEBUG : Dropbox root '': Committing async batch length 70 starting with: /img.jpg
2021-04-11 01:12:19 DEBUG : Dropbox root '': Adding "/img.jpg" to batch
What was rclone doing in this period? It looks like it was uploading some larger files or your computer was very busy. I don't know why there was 25 seconds between the message Dropbox root '': Batch idle for 10s so committing and Dropbox root '': Committing async batch length 70 - that should have been instant. Is your computer swapping here? Or your uplink maxed out?
That is a success anyway! From looking at your log it seemed to work properly.
You can now try increasing --transfers - the advantage of batch mode is that dropbox doesn't care how many simultaneous transfers you do.
What was rclone doing in this period? It looks like it was uploading some larger files or your computer was very busy. I don't know why there was 25 seconds between the message
I'm not sure, it just froze there and updated the screen every like 5 seconds.
So, did Dropbox start adding limitations recently? I never had problems doing multiple uploads (like I do with different processes) until now
That sounds like the computer was swapping.
I know they've been tweaking the limits so m maybe that has affected you.
I've been testing it on my Ubuntu VPS too and it seems like it makes it slower, the screen froze too for several seconds a few times before updating the output. I'm using --transfers 6 --dropbox-batch-mode async
By the way, sorry if this is not directly related but is the about command not working?
I get
2021/04/13 07:06:59 Failed to about: About call failed: about failed: missing_scope/
1 Like
Looks like that is a casualty of dropbox scope changes that we missed.
Try this.
You'll need to rclone config reconnect your remote to pick up the new scope.
v1.56.0-beta.5398.a7333844e.fix-dropbox-about on branch fix-dropbox-about (uploaded in 15-30 mins)
1 Like
Can you check to see if the VPS has enough memory? It sounds like it is swapping?
It has 1GB of RAM, this is just before running rclone with free --mega
total used free shared buff/cache available
Mem: 1028 310 71 0 646 577
Swap: 245 0 245
Does asyc need/consume more memory?
It shouldn't do but it may have a bug.
Can you do a test to see how much ram it does use?
Can you do a test to see how much ram it does use?
How do I do that, sorry? Just run rclone in the background and check memory usage with the process ID?
Also, I've been doing more tests with my own app secret and app id
In this one for example, it froze when it said it had transferred 29 files out of 183 (no other things were being uploaded at the same time), it froze for like a minute before updating the screen, and after that it consequently froze for a few seconds before updating the screen again, but even so it didn't seem like files were being transferred. I get similar freezes with sync too but it's slower. I also tried uploading just 29 files with the same commands and they were all uploaded without problems with just a couple of freezes of one or two seconds but it was faster.
It's very annoying these limits Dropbox has implemented since with the app or web interface it only lets you upload one file at once which is really slow
i can confirm this problem with "move". at first it runs quite well after starting, but at some point it starts sporadically.
the speed goes down every second ( switch --process ) and the files are at 0% or 100% or higher, but nothing happens anymore. if i then restart rclone move it continues normally again. until it happens sporadically again at some point. this can be after 30gb or only after 1000gb.
ransferred: 379.912G / 1.003 TBytes, 37%, 6.868 MBytes/s, ETA 1d2h48m16s |
Errors: 2183 (no need to retry) |
Checks: 26861 / 26877, 100% |
Deleted: 12339 (files), 0 (dirs) |
Renamed: 12336 |
Transferred: 12336 / 22364, 55% |
Elapsed time: 15h44m12.8s |
Checking: |
|
Transferring: |
- filename:100% /14.305M, 0/s, - |
- filename:100% /14.305M, 0/s, - |
- filename:105% /840, 0/s, - |
- filename:103% /1.197k, 0/s, - |
- filename: 0% /0, 0/s, - |
- filename:100% /47.684M, 0/s, - |
- filename:100% /19.073M, 0/s, - |
- filename:100% /47.684M, 0/s, - |
- filename:100% /11.249M, 0/s, - |
- filename:100% /47.684M, 0/s, - |
- filename:100% /33.543M, 0/s, - |
- filename:100% /14.305M, 0/s, - |
- filename: 0% /0, 0/s, - |
- filename:100% /47.684M, 0/s, - |
- filename:100% /47.684M, 0/s, - |
- filename:100% /143.051M, 0/s,