Hello, When i built my rclone directories my root has 4 folders that with my script runs one folder, uploads, and then the next. I’ve been running tests and found i get way more speed doing 15 transfers at once. Problem is with only doing one folder at a time lets say folder A only has 5 files and folder B has 5 and folder C has five, it will still only transfer 5 at a time. Is there a way to look at all the folders at once so it can transfer 15 at a time in that scenario?
Thanks
I presume you are using a reasonably recent beta - hence the folder at a time comment.
It is an interesting concern but I think it is unfounded as rclone just does the folder comparison one at a time it should punt the transfers off to be done asynchronously.
I did a couple of checks just to double check and I believe it is working like that.
If you have evidence (logs) of it not working like that, can you post them and I’ll investigate. If it isn’t working properly I’d like to nail it down before the 1.36 release.
First check - 5 directories with 5 files, transfers 15. Snapshot after 2 files have transferred.
* A/1: 0% done, 216 Bytes/s, ETA: 1h20m18s
* A/2: 0% done, 177 Bytes/s, ETA: 1h38m1s
* A/3: 0% done, 0 Bytes/s, ETA: 406h36m42s
* B/1: 80% done, 7.454 kBytes/s, ETA: 26s
* B/3: 65% done, 5.743 kBytes/s, ETA: 1m1s
* B/4: 62% done, 6.639 kBytes/s, ETA: 58s
* B/5: 80% done, 9.751 kBytes/s, ETA: 20s
* C/2: 77% done, 6.326 kBytes/s, ETA: 36s
* C/3: 58% done, 3.378 kBytes/s, ETA: 2m4s
* C/4: 68% done, 6.716 kBytes/s, ETA: 48s
* E/1: 100% done, 12.739 kBytes/s, ETA: 0s
* E/2: 55% done, 2.161 kBytes/s, ETA: 3m29s
* E/3: 77% done, 12.511 kBytes/s, ETA: 18s
* E/4: 62% done, 4.732 kBytes/s, ETA: 1m21s
* E/5: 52% done, 7.730 kBytes/s, ETA: 1m2s
Second check 5 nested directoreis each with 5 files, transfers 15. snapshot after 1 file has transferred.
* A/1: 27% done, 5.546 kBytes/s, ETA: 2m13s
* A/2: 43% done, 5.936 kBytes/s, ETA: 1m37s
* A/4: 43% done, 4.708 kBytes/s, ETA: 2m3s
* A/5: 43% done, 7.314 kBytes/s, ETA: 1m19s
* A/B/1: 43% done, 4.844 kBytes/s, ETA: 1m59s
* A/B/2: 49% done, 13.500 kBytes/s, ETA: 38s
* A/B/3: 24% done, 2.706 kBytes/s, ETA: 4m45s
* A/B/4: 33% done, 4.827 kBytes/s, ETA: 2m20s
* A/B/5: 62% done, 11.425 kBytes/s, ETA: 33s
* A/B/C/1: 58% done, 5.686 kBytes/s, ETA: 1m13s
* A/B/C/2: 40% done, 6.558 kBytes/s, ETA: 1m33s
* A/B/C/3: 43% done, 6.827 kBytes/s, ETA: 1m24s
* A/B/C/4: 37% done, 8.564 kBytes/s, ETA: 1m15s
* A/B/C/5: 21% done, 3.720 kBytes/s, ETA: 3m36s
* A/B/C/D/1: 0% done, 264 Bytes/s, ETA: 1h5m41s
Thanks for the reply. I am not talking about nested directories. The root of my acd has 3 folders and the way my script works it does
rclone move source1 acd:home/source1
rclone move source2 acd:home/source2
rclone move source3 acd:home/source3
So basically I was hoping for something simple like
rclone move source1 acd:home/source1; source2 acd:home/source2; source3 acd:home/source3
In hindsight I should have done the acd structure like home/media/source1 so i could sync the media folder and then it would do it all but the guide i used did it like this and with 6tb loaded its too late to redo it because as far as i can tell an acd to acd move actually copies or loops the data back through me as opposed to an instant move.
Sorry I mis-understood your question.
Try the latest beta - you should find you can move the directories about without having to upload and download.
Correct me if im wrong but that is only on rclone encryption while i use encfs?
I’m not sure about encfs.
You could also use --track-renames in your sync (latest beta).
I’ll give it a shot. Thanks!