Rescan during upload (add new files to que )

Feature
1

Any chance for rclone to rescan for new files during upload.
Example.
I set 50 transfers, when I run rclone in crontab in my upload folder I have only 1x file that is 10GB and it will take couple of hours to upload.

During upload in that folder additional 50 files were moved that wont be uploaded until the first file is finished.

The average speed for one file is around 1MB, however for 50 files Iam hitting between 30-50MB

It would be great if rclone could check every minute if there are any additional files for upload and add them to current taks, assuming you have “free transfers” left.

This is what i get with one file

051e3058da.png860×116 4.2 KB

Compared with multiple files at once.

dc12410690.png874×774 53.4 KB

2

An interesting idea… It would be quite difficult to implement though as rclone streams things from in to out.

Make a issue at github if you like with the above in.

I wonder why you get that 1 file vs many files? I think you are using acd, and I don’t get that with ACD. I can transfer at 20-40 MByte/s for a single file normally. Is your ISP or your router limiting the upload speed of individual uploads?

3

No and the server is in datacenter on unrestricted 1GB optics.
It only happens with ACD, everything else i can download/upload full speed.

4

Where is the server and which ACD endpoint are you connecting to?

5

It seems Amazon Drive US is really throttling single file speed from EU.
So far i have no problems with Amazon Drive Germany.

6

Correction, same weird upload speed to Amazon Drive DE.
If iam uploading multiple files I get speed from 20 to 40MB, however if iam uploading one file its between 200K and 1MB, not really sure how to fix this.

Using 200+ transfers now jut to get normal speed,
Transferred: 29.507 GBytes (33.529 MBytes/s)
Errors: 0
Checks: 35
Transferred: 10
Elapsed time: 15m1.1s
Transferring:
* …,2sGLqD6lI2nADA8eKt1nvs09KlsVn2qM0Ss-qiPV-: 7% done. avg: 308.2, cur: 343.5 kByte/s. ETA: 2h34m11s
* …,FBbyguASpauABITu1C1iQ48z2j,qxcOhJhkbuMfP-: 3% done. avg: 308.2, cur: 328.0 kByte/s. ETA: 5h27m56s
* …,UT-Q-KvA7Zsr02pqVZfBVhlh17jeUJBK4ckd6Auh8: 7% done. avg: 308.5, cur: 351.9 kByte/s. ETA: 2h30m16s
* …,W22WodOnhABscRFLgy0Z5rXdP1npDqJBR4TV0JOS1: 15% done. avg: 310.8, cur: 277.5 kByte/s. ETA: 1h24m41s
* …-2g,HTo8dH0NCK-RMu170nPbs4b,ecINbbTJG3F,F1: 8% done. avg: 302.2, cur: 288.6 kByte/s. ETA: 2h53m20s
* …-CsIUqXgBnw0KiJ7ATZEhDaupJ,TosniuXBovk0QD,: 7% done. avg: 317.1, cur: 334.4 kByte/s. ETA: 2h37m52s
* …-Qnoyx8Gq6aJNYOVbg1ietxAAYOOLtBI2o4ja93R0,: 7% done. avg: 315.2, cur: 320.6 kByte/s. ETA: 2h44m51s
* …-SVgdueJ0rvwAJ3EaiBbbKY6bT9xWbhT2O-1YnCVa-: 4% done. avg: 297.8, cur: 254.2 kByte/s. ETA: 5h26m45s
* …-XwWwKMy9PdLK3mkAr-BrLXDw4ki6lKR0IQlh0wci,: 7% done. avg: 292.6, cur: 294.0 kByte/s. ETA: 3h0m50s
* …-pATqZIs5wo3IMQNvohcJzKoPXEx2eb8sajlTMnpg1: 7% done. avg: 298.0, cur: 240.1 kByte/s. ETA: 3h41m2s
* …0Dd9lltF,4aSYR5Zr7aSML23iHzLSqD1fuydQSGQI0: 15% done. avg: 311.6, cur: 308.0 kByte/s. ETA: 1h16m17s
* …3jcqBwfC9dKeCLXZUFScA3YFeA8Y0aMLrQgAWVMX-D: 7% done. avg: 294.9, cur: 272.8 kByte/s. ETA: 3h14m36s
* …3nzVY6HEmMHAy6zpMoGC0P-p2STC8ZLNe0ulddNZy1: 7% done. avg: 320.8, cur: 373.6 kByte/s. ETA: 2h21m13s
* …3t1BnxcTTDTVGb8UQXhhG2XW4mJ0BVyoIkBRUIXs63: 7% done. avg: 309.1, cur: 335.6 kByte/s. ETA: 2h37m38s
* …522FcJJA,o,j-,awQx2S23TuMin6nmvSOtajKSbTr8: 7% done. avg: 301.2, cur: 268.0 kByte/s. ETA: 3h17m52s
* …56TD0T3bDv3SsETyy,5-vjbo8jo6GyMT4ZMJciKLE,: 7% done. avg: 306.6, cur: 330.3 kByte/s. ETA: 2h40m19s
* …5AtvHopEOallK4mUeMHRaZX5Y8A14iC-UKXzwYoKu0: 7% done. avg: 304.5, cur: 302.1 kByte/s. ETA: 2h55m21s
* …5FEgbaOoglf,uGG0y/bnvSgBznXhcGpNAmij1PUn6h: 2% done. avg: 348.3, cur: 373.7 kByte/s. ETA: 8h39m29s
* …5v2HW0Z4dMaZAnx6HjEOJmIuQB0BDidqhyRj-W28I-: 3% done. avg: 322.1, cur: 289.4 kByte/s. ETA: 6h20m26s
* …68grTxOf4Cdv6DGhlOnLLhYA7GJ9dVQffiQPfnNHE8: 7% done. avg: 288.9, cur: 318.7 kByte/s. ETA: 2h46m59s
* …6ZizDUzre-Ia1LADxI2ko9JiEAPkaPdvErF-ZMIl10: 14% done. avg: 303.7, cur: 316.4 kByte/s. ETA: 1h20m15s
* …6aFkZ4cDZHd1bRP2QVh7Pj2JtlsvJuLGbfGJ4Pp5M-: 7% done. avg: 308.1, cur: 305.3 kByte/s. ETA: 2h53m25s
* …6akWYf6rwqsR–SCqMp9HCskk2V5uQePdNBY4lwLP1: 13% done. avg: 316.5, cur: 276.6 kByte/s. ETA: 1h49m3s
* …6l2yHW3a4mqb3dsQnjpgTOXNoXiXtzHfsr7Yz50Jh,: 15% done. avg: 302.7, cur: 295.9 kByte/s. ETA: 1h16m12s
* …77ZM,ee18derldcK4oanyYeQbzzlTwwZM3F8lWmCg3: 7% done. avg: 315.1, cur: 346.8 kByte/s. ETA: 2h32m17s
* …79XZZsESKpSTnGveHH1gWh7hj0xVU8BlMs2FqQJTf5: 7% done. avg: 299.9, cur: 280.2 kByte/s. ETA: 3h9m7s
* …7wWkK5JOBUALihVj7ZfWFF0j5nonvAGVHj3b-TIjd9: 7% done. avg: 304.5, cur: 303.0 kByte/s. ETA: 2h54m43s
* …8UVNNFQLTqgmITvzDOH7Aq,v0cIgblJQB4WSMasT64: 7% done. avg: 313.4, cur: 299.8 kByte/s. ETA: 2h55m59s
* …9,MGWwcn,0jkpynaLQ4Bg7sS2pi2sB8ZEocYxdmEw0: 7% done. avg: 323.0, cur: 345.2 kByte/s. ETA: 2h32m26s
* …B5iwB96akV1kqGaBHkd9Xnpf7BsXkc5LJ5GHKKs0d0: 15% done. avg: 319.9, cur: 331.1 kByte/s. ETA: 1h10m46s
* …BPp6iBUazjeKuSqzn5IPvjYByupYanBoUGgrJACxO,: 7% done. avg: 301.0, cur: 281.7 kByte/s. ETA: 3h8m6s
* …C2zW8M388yc8RgBRVzgEOvpBljaj7uFWRmVo0AIa3,: 16% done. avg: 306.7, cur: 274.4 kByte/s. ETA: 1h21m40s
* …CDoPi0ED7z37ZsBOI9E-B12J8z4utihPeBpgnHpGC1: 7% done. avg: 300.8, cur: 311.5 kByte/s. ETA: 2h50m3s
* …CG5es8lhh-SN43iL7KE8iN5jbYZTOL8vrVPDA72QvA: 7% done. avg: 318.0, cur: 349.5 kByte/s. ETA: 2h30m58s
* …Cd-vBs4UvtV7AT7g5ThiDGD,r4v1euMEJNa,hPPVl1: 9% done. avg: 340.5, cur: 276.5 kByte/s. ETA: 2h48m39s
* …D-R52dpD4-rW1r,oRFCKaqiNFP3HzTGvQQftdKCAzB: 6% done. avg: 281.0, cur: 287.5 kByte/s. ETA: 3h5m19s
* …DHMPWi7kIy7KXPjCsvFZYM1fiuulfuqpuHznpOpzk-: 7% done. avg: 297.1, cur: 370.4 kByte/s. ETA: 2h23m25s
* …DPS8kUhSZzUWDdov13d3utIa3gf2oXpglhUKgidlA,: 7% done. avg: 305.4, cur: 273.3 kByte/s. ETA: 3h13m30s
* …EQ,PISxwnEaZfdGpJ17-4gFlYbvk01XGhmnQW8QKy0: 6% done. avg: 285.2, cur: 276.9 kByte/s. ETA: 3h12m22s
* …ETEcD8GOifzX6yBWDMIvcv0bfuEPOY5QupQanOU0p-: 6% done. avg: 322.1, cur: 296.0 kByte/s. ETA: 3h31m53s
* …EhZTC4bKglXheCDZ36gVESX-O7DWtnCg-rhb23XbP-: 16% done. avg: 326.7, cur: 308.6 kByte/s. ETA: 1h17m25s
* …FrDwJDHp9Wqh-wYhkEDuN7JXRiTfjyOXclnZw4X,H2: 7% done. avg: 302.8, cur: 240.2 kByte/s. ETA: 3h40m13s
* …GxN0FnrxKfFdA3Cb8GCPPOCMXujV15Eg6MUOPE,Bq0: 7% done. avg: 323.6, cur: 296.4 kByte/s. ETA: 3h4m19s
* …HZE4gXm4JtRt4chWC9Gxpe1vORuKP,FtPS99Z26haA: 7% done. avg: 299.0, cur: 292.6 kByte/s. ETA: 3h1m18s
* …JERBOlsdf,0ZPBBUDyXAHcOGhpWi,sQSos-LybR-rA: 7% done. avg: 298.8, cur: 327.8 kByte/s. ETA: 2h41m50s
* …JNEaOcMJacUC,49VB2zf1ZIdMfal,96LcNI6K,mTd1: 7% done. avg: 316.1, cur: 343.2 kByte/s. ETA: 2h33m56s
* …LBzD8M3tYk4NqrQLp/BalGCvqumIt85R8q,UoZG4DD: 11% done. avg: 307.2, cur: 258.9 kByte/s. ETA: 2h10m46s
* …M0I0AKY82Ck5pVgIZPqqFOQgrmAhEUQmQ-mBqQecn0: 6% done. avg: 338.0, cur: 315.1 kByte/s. ETA: 3h33m49s
* …MeMbx1zXObs6e0hYCdVpERNLj5jMNgUFJ1EvvTvc,0: 7% done. avg: 317.8, cur: 355.2 kByte/s. ETA: 2h28m43s
* …MiLXog97arSXQNO9ywvEksa-F9i,EMfUZdf,YbxNl-: 3% done. avg: 326.3, cur: 334.2 kByte/s. ETA: 5h29m29s
* …N2j9,z35emPxk2sxmhYyd8OD1FSUKx5dPpXpMrucE3: 7% done. avg: 301.3, cur: 283.2 kByte/s. ETA: 3h7m6s
* …NmUmlOhSAW8X4MGsRBhtQW9dP6Azk4IikAsrXaIOi-: 4% done. avg: 325.1, cur: 363.6 kByte/s. ETA: 4h30m55s
* …NrLx6-XnIlbXa5U9CoIrqXE5HTNFIdv8X7HRLxxFR1: 21% done. avg: 296.3, cur: 250.8 kByte/s. ETA: 1h0m4s
* …Ns3nFa6p3386N9cboXjAV-tOiMj8Wn0Rm0HZoRHT69: 7% done. avg: 316.5, cur: 284.3 kByte/s. ETA: 3h5m36s
* …NuEo0RfrW4EbCs9YQYmRgttbiByEvQXfJYiOVxDWJ0: 7% done. avg: 326.0, cur: 305.6 kByte/s. ETA: 2h52m19s
* …PcgygQbZr7XQ0Z,yjoUt47xP8KH5NMOu6AuyAXb8J,: 7% done. avg: 306.4, cur: 289.5 kByte/s. ETA: 3h2m42s
* …Pk0hfxxI9w6YlGGtN-yapA7yh8T-mECUSWqGlLZPE7: 7% done. avg: 302.5, cur: 325.2 kByte/s. ETA: 2h42m47s
* …Q,g0crhYfjhOl7GZknxS1yqcgdV,-drd4yhHzwgES1: 6% done. avg: 313.1, cur: 244.1 kByte/s. ETA: 4h38m1s
* …Q8h-IBKxHoN8BcXCE3qPl9OdRXcfPc55J-,y6O1PT,: 3% done. avg: 304.0, cur: 231.9 kByte/s. ETA: 7h55m47s
* …QMYswXKGJrO29M9A8i0f9FA7rsY47uIzu8wLvTBJ30: 7% done. avg: 300.8, cur: 285.1 kByte/s. ETA: 3h6m0s
* …R4X4HcQ-S1TSD3huNUaetapavRrZhGf9XCu0fJ,gE2: 7% done. avg: 320.1, cur: 328.0 kByte/s. ETA: 2h40m39s
* …RmU0AdG1VUt1r3,mPGy2VsDmy7CqGz2W9cvwVid6c-: 8% done. avg: 327.1, cur: 351.6 kByte/s. ETA: 2h29m30s
* …S7i6a5l14eLH0VkNhR5sug5DLGN2mmatbp-J3vRuM7: 8% done. avg: 327.5, cur: 297.0 kByte/s. ETA: 2h56m43s
* …SjhrwyvCxuIiwZ6N8nNmI2cqJhIiahbiOM1NwAvyIB: 7% done. avg: 306.9, cur: 297.5 kByte/s. ETA: 2h57m54s
* …Sz7-igrrsQbcXn5pVUyIZ1SFxVypvXy5bzdf74aH0B: 7% done. avg: 322.7, cur: 321.6 kByte/s. ETA: 2h43m54s
* …U1L21U2snWpc2-oNQHExhOwvpi5901ZEXsMk4AU,X8: 7% done. avg: 299.0, cur: 329.4 kByte/s. ETA: 2h41m2s
* …U1pdm65Pa0K9xTdUykC4lJv9ns1WqT5Mra7iOlQqx1: 7% done. avg: 301.9, cur: 305.7 kByte/s. ETA: 2h53m34s
* …UPs-JI0bDrbNOM3DlLbzUYDEb4uv9pmdmWvGnjMAZ1: 7% done. avg: 295.4, cur: 282.6 kByte/s. ETA: 3h8m2s
* …Xf4mkjvWro9nPxx2XaiJzzLfy53,zSuF8Cpis2wHU6: 7% done. avg: 294.0, cur: 298.9 kByte/s. ETA: 2h57m37s
* …Y77mNFuxHjFk3ZM9m9RnD3FQD7dOlzvwi76NcGo5,0: 7% done. avg: 300.9, cur: 318.3 kByte/s. ETA: 2h46m37s
* …YBD,-Ps-in447nzPbCDSBIJ5cMFhUTh6corImf1iP1: 7% done. avg: 319.7, cur: 366.0 kByte/s. ETA: 2h24m5s
* …ZZ7tQ7yBTmhGU-J,JGCslwvmfz6EQQapquOPa8ktNB: 7% done. avg: 302.2, cur: 280.9 kByte/s. ETA: 3h8m34s
* …ZguGA9dAE8LC7SYtHlRK4tCgThszXZeY,M4MU-YSt1: 7% done. avg: 307.7, cur: 285.4 kByte/s. ETA: 3h5m37s
* …ZjPJ1Quf,-zKBU6luUMsX5mNWC46P7CyEdVmqZ5RA3: 7% done. avg: 308.4, cur: 305.4 kByte/s. ETA: 2h53m10s
* …aAuQtEajNqao6-DcL9aoRmpsMsekH9cZuEjpuVfvv,: 7% done. avg: 305.2, cur: 312.3 kByte/s. ETA: 2h49m36s
* …aT67dpBViLrRvKqb7v3QcqcYhOvVU3sM,1sagg,lC,: 8% done. avg: 327.5, cur: 303.0 kByte/s. ETA: 2h38m48s
* …b,iuvo53gQ9RLNtrWeEY,cInPAo2nBNxzzzbfv8Wm8: 7% done. avg: 325.6, cur: 289.7 kByte/s. ETA: 3h1m54s
* …bILSMz7Az-zPAwiQsQo1hvpHHNFoe5FbgrRvrLd514: 7% done. avg: 320.4, cur: 317.2 kByte/s. ETA: 2h46m13s
* …bQmW3fEh0tciXKHnOfAXTi2twvrW8-0ZdSMo8V-aB1: 16% done. avg: 321.6, cur: 286.9 kByte/s. ETA: 1h21m36s
* …cHXLqLQXwmrIX0kNrBGdPZofiRKDelTXkXp13151D1: 4% done. avg: 296.0, cur: 237.9 kByte/s. ETA: 6h24m7s
* …d5mRwgbiyn8grimGWkHGqV7-FOaWrLIr5NgOAVSNh0: 7% done. avg: 303.0, cur: 338.9 kByte/s. ETA: 2h36m27s
* …e5ymOdrXJQIgJqwwwjcxq3cllZyYxbZuDXkilQtr11: 7% done. avg: 309.3, cur: 301.0 kByte/s. ETA: 2h55m54s
* …fEE9EbOW8kA1V6H8NmNg8VydE,PyLLOhuJcNMeI197: 7% done. avg: 311.6, cur: 291.2 kByte/s. ETA: 3h1m30s
* …finIvvjg-2YGu2GMKyFLt5GTtr4Oav,e8oYzHoB029: 7% done. avg: 306.5, cur: 271.7 kByte/s. ETA: 3h14m50s
* …fpOi9z,Cw4NUTWtcp6,pf2OotU0UqyOJg6F2I3RBN0: 5% done. avg: 314.2, cur: 296.5 kByte/s. ETA: 4h13m33s
* …gBCoejWshbAr,Hg3GJJ6jMnFAOUUh,aZ8M6dNj6Ne0: 7% done. avg: 293.9, cur: 278.9 kByte/s. ETA: 3h10m31s
* …gE5bYv3xl55I2bki367VuI85j9Vi0t0hQ1wVy7ahQ8: 7% done. avg: 317.3, cur: 355.2 kByte/s. ETA: 2h28m44s
* …gnk,onZ4G6wKM,bbbHPs3fgP55e,FpMPopUodfYxD1: 3% done. avg: 322.6, cur: 315.9 kByte/s. ETA: 5h48m42s
* …hLtMM7uY6XBrh0X9CZ8eIqpfUzy1ikWO1y38-gjhC1: 7% done. avg: 297.0, cur: 279.7 kByte/s. ETA: 3h9m55s
* …hmma9kUe-1yfXUcj1IesaKYitKhThepGTKCN,lFen,: 14% done. avg: 286.6, cur: 273.7 kByte/s. ETA: 1h23m49s
* …i7XooCCBHjDnSiRnD3BPVJGS5KThpYPS4rO006JDCD: 7% done. avg: 314.2, cur: 323.1 kByte/s. ETA: 2h43m32s
* …iOEs5gqrviHhLuHc0gJfkT,YcJaOJ9yj8R1uLNNf90: 7% done. avg: 300.5, cur: 296.3 kByte/s. ETA: 2h58m37s
* …jPaYjIo65Bck6AIGzOXUP0pq4s5ngTbla2wLKV,F6B: 7% done. avg: 293.3, cur: 269.0 kByte/s. ETA: 3h17m27s
* …jiyXwGAmPINHvGPkFoTEEovvuZI2BA1bygRFZgS-b-: 4% done. avg: 330.9, cur: 306.8 kByte/s. ETA: 5h58m42s
* …k4iBzOCP7WQ0ausKrd-H751ZwezX5g,UhL8S5k04P0: 7% done. avg: 303.5, cur: 302.6 kByte/s. ETA: 2h55m9s
* …kua7Q3rt4lxe94dXG6uXQsm3rFPjju94V-kjgK7sB1: 7% done. avg: 317.6, cur: 336.3 kByte/s. ETA: 2h36m42s
* …lphCdKXSSbGuX2DkMbc7K24b-grIHvfhUwnrAHdES5: 7% done. avg: 300.3, cur: 264.7 kByte/s. ETA: 3h20m18s
* …mF5YdfHFB8ZqHi2ZpNPXgGCTuJi5VHmkxfRuRReat7: 7% done. avg: 291.3, cur: 345.4 kByte/s. ETA: 2h33m49s
* …mN,ysjXW,wkrl61vD8zoyXloSFz5aGKOj5E,l1QD74: 7% done. avg: 318.5, cur: 337.5 kByte/s. ETA: 2h36m6s
* …mWIqhTOkamEF5VkKfQ,bnSIeCkd8QhP1duZiZRSnv1: 7% done. avg: 314.5, cur: 327.1 kByte/s. ETA: 2h41m33s
* …myOffy5TYM6DM2SuGpso6c5P5kvibA8q2CdMUOzRz-: 15% done. avg: 318.1, cur: 338.2 kByte/s. ETA: 1h11m9s
* …n-bzUUqBKA,YplSJUMTQyrpWNCWJmEnfuNeKiYwDt2: 7% done. avg: 311.3, cur: 304.7 kByte/s. ETA: 2h53m12s
* …n5DdA6X2zEcfJu6d9JO5BsL65VD71GREHx247BP9h,: 7% done. avg: 286.9, cur: 288.6 kByte/s. ETA: 3h4m23s
* …nac4zD9O9gl9x8VR3j7biDzOx8MtCxnu9pOQeuaDI6: 7% done. avg: 301.7, cur: 234.5 kByte/s. ETA: 3h45m58s
* …owk9dwAd5JKX3KgZMhPXCt,4phuXqAe47BIVvBRmz3: 8% done. avg: 328.8, cur: 313.3 kByte/s. ETA: 2h47m47s
* …p6nqLWJtNi8750MV1JZFfdtNAj-i2ajH3CmGiSvOb,: 7% done. avg: 298.0, cur: 261.4 kByte/s. ETA: 3h23m1s
* …pnG9GIKNCwASRGBosgAd8jX19T6ApwqAMqApXFhWY,: 14% done. avg: 313.6, cur: 310.8 kByte/s. ETA: 1h21m27s
* …qRYvEHWr3G4eKdYhxDkl97QHQ9wemm27Q4HIdlK5O1: 3% done. avg: 294.1, cur: 306.9 kByte/s. ETA: 6h0m11s
* …rPLZ-DZXQ2pZ8nvl,7EeC1bhJLFvDeSwl7QnEL9g65: 7% done. avg: 310.3, cur: 266.4 kByte/s. ETA: 3h18m31s
* …rQnYSnLIllAdClsfg4a02N9IKP05JZYrJD0mlKx-X0: 3% done. avg: 301.5, cur: 310.8 kByte/s. ETA: 5h55m9s
* …rVX-6P7NR,61Ul-DjWGm-ajXCn6oG3Gi-t2NGzltP-: 7% done. avg: 320.7, cur: 300.9 kByte/s. ETA: 2h55m12s
* …rnc2OH4CKJe5ePWCUJfRXdU,MA4pZbBbvmVFn4,RY0: 9% done. avg: 295.2, cur: 343.4 kByte/s. ETA: 1h51m55s
* …uwdm81kweBtofrPDLtf1SJbwBJJ8nZDI7Sbi3TtGb1: 16% done. avg: 339.7, cur: 313.1 kByte/s. ETA: 1h21m46s
* …viUajoE4wZNe0RbWVK5K2ywaoDX5J,4OuD-iam1Em0: 3% done. avg: 293.8, cur: 269.6 kByte/s. ETA: 6h49m55s
* …w6m1PtS4dLw3T21FGapyalQGDox3h9G89ZKg3nIlE,: 14% done. avg: 292.8, cur: 277.6 kByte/s. ETA: 1h26m39s
* …xIbsqKLnMemSTRwXHQjTmTHQh643xYmr3jxhz6L0w4: 7% done. avg: 310.4, cur: 337.1 kByte/s. ETA: 2h36m36s
* …xob6FZpvaNRdTZtd8Tqsg-AnQ,6z-uw9,Gu1K9cIq,: 14% done. avg: 287.8, cur: 274.5 kByte/s. ETA: 1h28m43s
* …yCwxKg0WImUyqiJ5m/IxSD2FkgKLgYK3f5dIXt2p5h: 2% done. avg: 296.5, cur: 302.9 kByte/s. ETA: 9h13m43s
* …zMKt93qpG-OlLGl,x0bLkAGUpDe,8mztR15D1vFPx-: 7% done. avg: 316.8, cur: 294.7 kByte/s. ETA: 2h59m4s

7

I’m not sure how either! I still suspect the ISP providing the service to your server, but it could equally well be amazon, or one of the transit providers.

Anyway that is why rclone has an adjustable transfers limit to work around limits like that.

8

Hopefully in the long run some type of rescan can be implemented.
I could also run multiple copies or rclone if I could set it not to touch files currently in progress eg if rclone somehow locked the file its transfering.

Daily I have between 500GB and 1TB of data that is being moved to upload folder randomly and sometimes is just pain to see 600GB left for upload and rclone (running it in crontab ) is transferring one 8GB file with 200KB.

In those cases I just kill rclone and wait for crontab to rerun it again.

p.s. The crontab rclone sh script is not being runned if is already running.

9

That sounds like a reasonable workaround - rclone will get there in the end!