--files-from processing order

Hi!

Lets say:

rclone -v move --files-from inputfile.txt /some/arbitrary/path crypt:another/path

How can I make rclone process the paths in "inputfile.txt" sequentially / in the order ?

As it is now, it seems to process as it pleases

:slight_smile:

Rclone is a concurrent program so will process things in parallel.

You can limit this parallelism by using --checkers 1 --transfers 1 - that might get things processed in sequential order, I'm not 100% sure!

What's the reason you want it to go in sequential order?

You can limit this parallelism by using --checkers 1 --transfers 1 - that might get things processed in sequential order, I'm not 100% sure!

Hehe, it doesn't seem to fix it unfortunately :stuck_out_tongue:

What's the reason you want it to go in sequential order?

The inputfile is computed based on some criteria's such as mod dates and more. The dataset is huge (above limit), so I want it to go as far as it can with the daily limit, based on my priority. Basically I need some ordering functionality.

I think NCW once indicated to me that rclone will tend to process files a folder at a time internally.
So if your input order is all-files by some criteria (like modtime) then it's not likely to obey that I think.

Not exactly sure what the benefit of this strategy is though. Some kind of sorting would probably be useful to have. I think there is an active issue on this already.

hello,

perhaps change the way you compute that inputfile.
to calculate the sum total of file sizes of the files to upload and stop when you compute that you have reached the daily limit.
then feed that inputfile to rclone

loop it,

  1. create input file based on daily limit.
  2. feed input file to rclone to move files.
  3. goto 1

Could work as a workaround. It wouldn't be in-order file-by-file, but the sets would be in-order compared to the next and previous sets. You would need to script a parser to do this job though.

But I think, ultimately, it would make sense to look into some basic sorting.
I've suggested something along these lines for optimization before. A sorting that interleaves large and small files for example would for example have some pretty significant impacts on efficiency on any backend that has limits on how fast you can start new transfers. (preventing bottlenecking on either amount of transfers or wasted bandwidth as much as possible) given any mixed set of files.

Once you implement a sorting system it has a lot of flexibility in what it can be used for.

Funny you should mention that, I've been working on an --order-by flag which can order transfers by size, name or modtime - would that help?

1 Like

Nice!
Sounds like that would be exactly what OP is asking for.

Are you looking into that big/small sorting we've discussed earlier also, or are you trying to just keep it as simple as possible for now?

You must be my secret santa :slight_smile: :+1:

Jup, that would be the workaround in the meantime :slight_smile:

Here is a beta of --order-by use it with --order-by size,ascending or --order-by size,descending You can say size, modtime, name.

It may not do things in absolutely perfect order but it will be close! I haven't written the docs yet!

https://beta.rclone.org/branch/v1.50.2-073-g56c0953e-fix-1205-order-by-beta/ (uploaded in 15-30 mins)

:smiling_face_with_three_hearts: I'll try it out as soon as I get some daily upload back.

Many, many thanks. It's really appreciated :slight_smile:

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.