Updating without creating new files if source has some?

Suspected Bug
1

Hello,

I would like to know if there is any way to update the content of a target folder without creating new files that could have been created on the source meanwhile ?

I found this topic Update only the existing files on a remote side? which led to this kind of workaround :

rclone lsf --files-only -R $dest > dest.txt
rclone copyto $source/$n $dest/$n -v --include-from=dest.txt

And that worked, for a simple test.
Once I tried to make it work for more complex folders with a lot of content in each, I met errors. And blocking errors which stopped the copyto (simulated) command from start.
And if I remove this file line in my reference list, I can't possibly know if I will have a new line to create an issue.
Also, I can't really remove a file line if this very file has to be updated.

Example of my two commands :

rclone lsf --files-only -R DRH: > lsf-DRH.txt
rclone copyto source:"DDRH" DRH:: --dry-run --include-from=lsf-DRH.txt -v --log-file onlyUpdate-DRH.txt

The main issue was regarding a regex. Here is an example of what I got :
(and there is no other content in the generated logfile)

2023/04/14 21:42:19 Failed to load filters: bad glob pattern "04-AF_closes/2016/DD04CA - 25 DAUPHINE/14 PLANS-CALCULS/BET/[BTP-Consultants]_FED_1_2015-06-03.pdf" (regexp "(^|/)04-AF_closes/2016/DD04CA - 25 DAUPHINE/14 PLANS-CALCULS/BET/[BTP-Consultants]_FED_1_2015-06-03\.pdf$"): error parsing regexp: invalid character class range: 'P-C'

To be noted, it was the character ` and not ' that was used in the error message, but it doesn't seem to be accepted by the forum :expressionless:

The thing is I don't get why the string 'P-C' would led to an error.

For other tests, it will trigger that very error for another folders with different strings, such as 'L-E', '6-1' and other similars. There is a clear pattern of two characters separated by a hyphen.

Otherly, I also got for several other tests folders, I had this one :

2023/04/14 22:11:16 Failed to load filters: mismatched ']' in glob "04-AF_closes/KK00W028 - CDEF - AULNAY/FT aulnay/FR] AVANCEMENT RAPPORT DE CLASSIFICATION -CALYSTO L EN2.jpg"

Here even if it seems weird, I am not here to judge why the users had not as many [ than ] in their filenames.
I got another example with three [ and no ] to close them, which led to the same error code.

I have no such issues with a copy --update (--dry-run), so I guess the --include tag works indeed with a maniacal regex.
Is there a way to get through this restrictive and nitpicking regex ? The files are named the same way in the source as they are in the target, even if it is badly.

Would --filtering-from do the trick ? I am not sure to understand how it works (and what it is supposed to do).

Thanks :slight_smile:

2

please post that file.

fwiw, the easiest way to test a filter is using rclone ls, as that does not modify the dest.
once rclone ls outputs the correct list of source files, then try rclone copy

3

Sadly I can't for three reasons :

  • DRH was one of the few repertories to have no issue at all (but was the first lines in my batch)
  • it contains titles of documents that are confidential
  • way too big to be posted !

But, what I can do is to provide another error and its line in the reference file that was created before :

2023/04/15 10:13:31 Failed to load filters: bad glob pattern "AO2019-ST41/AO Generali/BPM004649-2-2-9+Messine+-+Renovation+Works+-+Macro+Lot+Technique/[CCAP-CCAG] CCAP - CCAG -AB 2018 11 07 def.pdf" (regexp "(^|/)AO2019-ST41/AO Generali/BPM004649-2-2-9\+Messine\+-\+Renovation\+Works\+-\+Macro\+Lot\+Technique/[CCAP-CCAG] CCAP - CCAG -AB 2018 11 07 def\.pdf$"): error parsing regexp: invalid character class range: 'P-C'

And here is the lsf line (942/32673) :

AO2019-ST41/AO Generali/BPM004649-2-2-9+Messine+-+Renovation+Works+-+Macro+Lot+Technique/[CCAP-CCAG] CCAP - CCAG -AB 2018 11 07 def.pdf

I will give a try with a simple ls on the destination, then a rclone copy as a --dry-run and let you know

4

try https://rclone.org/docs/#dump-filters

5

So, I tried a ls, and here is what was written in the simulated copy logfile :slight_smile:

2023/04/16 16:11:31 ERROR : Using --filter is recommended instead of both --include and --exclude as the order they are parsed in is indeterminate
2023/04/16 16:11:32 INFO : There was nothing to transfer
2023/04/16 16:11:32 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 1.3s

2023/04/16 16:11:43 ERROR : Using --filter is recommended instead of both --include and --exclude as the order they are parsed in is indeterminate
2023/04/16 16:11:44 INFO : There was nothing to transfer
2023/04/16 16:11:44 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 1.2s

2023/04/16 16:11:59 ERROR : Using --filter is recommended instead of both --include and --exclude as the order they are parsed in is indeterminate
2023/04/16 16:12:00 INFO : There was nothing to transfer
2023/04/16 16:12:00 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 1.2s

2023/04/16 16:48:09 Failed to load filters: mismatched '[' and ']' in glob "1705705 Auger45/DOC TECHNIQUE COUVERTURE/BARDAGE/SILVALBP bardage bois/SIV01[FT[2002903[01.pdf"

Do you think I should go on something like a --filter-from [myList] --files-from-raw, if I understand correctly ? I am not sure other options could negate unusual characters

6

why not give it a try.....

7

¯_(ツ)_/¯

Anyway, I tried this one :

rclone copyto source:"G/01-ST/ST45.TODRIVE" ST45: --dry-run --filter-from=ls-ST45.txt --files-from-raw -v --log-file onlyUpdateFromLS-ST45-exclude11.txt

And here is the content of the logfile :

2023/04/16 18:57:11 Failed to load filters: malformed rule "63756 AO2021-ST45/REPERTOIRE MOTS DE PASSE DEMAT (Récupéré).pdf"

And that's all. No reported issue of else. I am quite puzzled on this one, to be honest :

  • I don't get the regex related error
  • I get a new one that is not very clear to me

I then tried :

rclone copyto source:"G/01-ST/ST45.TODRIVE" ST45: --dry-run --filter-from=ls-ST45.txt -v --log-file onlyUpdateFromLS-ST45-exclude11.txt
(the same minus the --files-from-raw options)

Same result.

If I go back to an --include-from=[myList] :
rclone copyto source:"G/01-ST/ST45.TODRIVE" ST45: --dry-run --include-from=ls-ST45.txt -v --log-file onlyUpdateFromLSFilter-ST45-exclude11.txt

I get :
2023/04/16 19:11:13 Failed to load filters: mismatched '[' and ']' in glob "1705705 Auger45/DOC TECHNIQUE COUVERTURE/BARDAGE/SILVALBP bardage bois/SIV01[FT[2002903[01.pdf"

And if I ultimately try the same include command with the --files-from-raw option, I get the same result.

At the end I tried a :

rclone copyto source:"G/01-ST/ST45.TODRIVE" ST45: --dry-run --files-from=ls-ST45.txt -v --log-file onlyUpdateFromLSIFROM-ST45-exclude11.txt

Aaaaaand :

2023/04/16 19:16:41 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 2.6s

Which is weird but not unlikely.

I will have to try other source directories to copy to check.

8

If you use filters and you have reserved characters listed here, you have to escape \ them in the filter file.

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.