Hi!
Lets say:
rclone -v move --files-from inputfile.txt /some/arbitrary/path crypt:another/path
How can I make rclone process the paths in "inputfile.txt" sequentially / in the order ?
As it is now, it seems to process as it pleases
Hi!
Lets say:
rclone -v move --files-from inputfile.txt /some/arbitrary/path crypt:another/path
How can I make rclone process the paths in "inputfile.txt" sequentially / in the order ?
As it is now, it seems to process as it pleases
Rclone is a concurrent program so will process things in parallel.
You can limit this parallelism by using --checkers 1 --transfers 1
- that might get things processed in sequential order, I'm not 100% sure!
What's the reason you want it to go in sequential order?
You can limit this parallelism by using
--checkers 1 --transfers 1
- that might get things processed in sequential order, I'm not 100% sure!
Hehe, it doesn't seem to fix it unfortunately
What's the reason you want it to go in sequential order?
The inputfile is computed based on some criteria's such as mod dates and more. The dataset is huge (above limit), so I want it to go as far as it can with the daily limit, based on my priority. Basically I need some ordering functionality.
I think NCW once indicated to me that rclone will tend to process files a folder at a time internally.
So if your input order is all-files by some criteria (like modtime) then it's not likely to obey that I think.
Not exactly sure what the benefit of this strategy is though. Some kind of sorting would probably be useful to have. I think there is an active issue on this already.
hello,
perhaps change the way you compute that inputfile.
to calculate the sum total of file sizes of the files to upload and stop when you compute that you have reached the daily limit.
then feed that inputfile to rclone
loop it,
Could work as a workaround. It wouldn't be in-order file-by-file, but the sets would be in-order compared to the next and previous sets. You would need to script a parser to do this job though.
But I think, ultimately, it would make sense to look into some basic sorting.
I've suggested something along these lines for optimization before. A sorting that interleaves large and small files for example would for example have some pretty significant impacts on efficiency on any backend that has limits on how fast you can start new transfers. (preventing bottlenecking on either amount of transfers or wasted bandwidth as much as possible) given any mixed set of files.
Once you implement a sorting system it has a lot of flexibility in what it can be used for.
Funny you should mention that, I've been working on an --order-by
flag which can order transfers by size, name or modtime - would that help?
Nice!
Sounds like that would be exactly what OP is asking for.
Are you looking into that big/small sorting we've discussed earlier also, or are you trying to just keep it as simple as possible for now?
You must be my secret santa
Jup, that would be the workaround in the meantime
Here is a beta of --order-by
use it with --order-by size,ascending
or --order-by size,descending
You can say size
, modtime
, name
.
It may not do things in absolutely perfect order but it will be close! I haven't written the docs yet!
https://beta.rclone.org/branch/v1.50.2-073-g56c0953e-fix-1205-order-by-beta/ (uploaded in 15-30 mins)
I'll try it out as soon as I get some daily upload back.
Many, many thanks. It's really appreciated
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.