Filtering max number of files to transfer

Help and Support
1

Hello, guys.

Is there a way to filter the maximum number of files?
Something that behaves like --max-files, as in --max-age or --max-size?
Maybe some --filter trick?

Let me explain.
I'm trying to do specific filtering to copy some backup files to the cloud once a day.
When filtering, a lot of options are working well, like in:

rclone lsf "D:\bkps" --exclude "*.log" --max-age 2h --order-by modtime,desc

This give me a good list of files: logs excluded; only newer than 2 hours; newer files first.

These files are auto-generated by a 3rdy-party application, 2 or 3 times over the day.
When using --max-age 2h I'm getting the last needed backup files, well done.
But sometimes the 3rdy-party application fail or refuse to do the last-day backup (have no control over it), and the latest backup files was generated something like 7h ago. Or, worst case, 18h ago.

So, the best way seems to be use --max-age 24h, as rclone runs one time a day.
But if all 3rdy-party backup files are generated, I'll copy 3 full backups - more than my cloud quota allows.

Well, as I know the number of generated files, I'm searching for an option to filter only the last (newest) 2 backup files on the past 24h. Is there a way?

I've tried some regex on filtering, but rclone don't seems to implement go regexp grouping.
Any help will be appreciated.

Maybe --max-files (or max/min operations, objects) or something like that can be added as a feature, if it seems reasonable.

Sorry for the long post.
And thanks you, guys, for this amazing tool.
Really useful, versatile and well documented.
I'm really impressed.

rclone v1.51.0
Windows 7 Ultimate, 64 bits, pt-br
Microsoft OneDrive

2

hello and welcome to the forum,

long story short,
you want the newest two files in the last 24 hours?

what do you mean by newest?
newest based on creation date
or
newest based on modification date

3

Really great short story, thanks! :smiley:

Yes, I need filter down to the newest (by modification date) two files in the last 24 hours.

Anyway, by creation date will do de job, also.
But by modification it will match the --order-by modtime,desc parameter also.

4

we could write a script
i am a long time windows user but recently i have been working on my linux skills

if you have windows subsystem for linux, you could save this to a file called doit

find $source -type f -mtime -1 ! -name '*.log' -printf '%f\t%TY%Tm%Td%TH%TM\n' | sort -r -k 2 | head -2 | cut -d$'\t' -f1 | ./rclone copy $source $dest --include-from=-

and run bash doit

5

Well, great ideas. But WSL is not an option on Windows 7.
I'm using some old-fashioned bat scripts, I'm sure I can do something like that.

But I'm interested on some rclone native filter option to do this.
Rclone seems too powerful to do not have such option.
We've options like --max-delete=N and --max-transfer=SIZE, for example.

6

sure i understand you want rclone to do that,
but at this time rclone does not do that, perhaps never will.
tho ncw, the primary author of rclone, could read this post and add those options.

you have a problem now, of uploads failing, so if you need a solution now, you have one.
i used to use https://www.cygwin.com/, to run linux commands, which runs on W7.

as an aside, off-topic, you can upgrade from W7 to W10, free and legally via microsofts website.

7

I'd do the find bit with rclone lsf something like

rclone lsf -R --csv --files-only -F tp --max-age 1h

This will produce a csv file of all the recent files with the mod time at the start. This file needs to be sorted and the first two lines taken and the file names stripped out. That is something that one skilled in the arts of .bat or powershell could do I'm sure (not me :slight_smile:

1 Like
8

updated command, as per ncw

rclone lsf $source -R --csv --files-only --format=p --max-age=24h --exclude='*.log' | tail -n 2 | ./rclone copy $source $dest --include-from=- -vv

1 Like
9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.