Rclone copy files only based on filename

[jitvimol]

All of my files has structure like this "#####_picture.jpg" where # is number.

For example 22000_picture.jpg , 22001_picture.jpg

I could transfer with the below command but it stopped because there is a limit 150K number of files already transfer to onedrive]

rclone --verbose --cache-workers 64 --transfers 64 --retries 32 copy gcp:photo onedrive:photo

I want to continue the transfer with filename start from 22002 onward. I google for hours and don't know how to set glob to filter filename to be transferred. Can you please advice? Thank you very much.

[albertony]

Welcome to the forum!

I'm not sure I follow what your problem is, exactly:

because there is a limit 150K number of files already transfer to onedrive]

If you just want it to continue from where it stopped, you can run the same command again and it will skip existing files (based on size/timestamp/hash) and copy the missing ones. But maybe you already knew this?

[jitvimol]

Hi albertony,

Yes I did do that. it seems to skip the already copied files, but after it reach150k file, it shows error like this.

2021/03/08 21:57:45 ERROR : 24188_PhotoGrid_1585121032827.png: Failed to copy: notAllowed: childItemCountExceeded: Max limit on the number of child items was reached

Onedrive limit only 150,000 files per folder.

I will solve the problem by copying the remaining files to another folder.

[albertony]

Ok, then I understand the issue.

Using rclone's filtering option there are probably several alternatives, I'm not an expert, but perhaps something like this:

--include 222\d\d_picture.jpg

With input

22000_picture.jpg
22001_picture.jpg
22100_picture.jpg
22101_picture.jpg
22200_picture.jpg
22201_picture.jpg

it would match output

22200_picture.jpg
22201_picture.jpg

According to rclone filter demo

[jitvimol]

--include 222\d\d_picture.jpg

Million thank you. I have never used filtering command like this. Let me learn by trying your suggested code above.

Let me share back from last night that I was struggling for this solution.

--include "2????_*.*"

This filtering command works as well, filtering the copy of filename like 20001_pic. jpg It seems to be unix filtering style while your suggested way seems to be "glob" style. Not sure, I need to dig deeper

[albertony]

Sounds good.

My solution used character class \d from regular expression syntax, and will match numbers only, while yours with glob ? matches any character - depending on what other filenames you have in your case they may or may not give identical results.

The filtering functionality is really flexible, but then can also be hard to understand some times, so don't be afraid to ask in the forum if you have further questions.