As part of #3762 we realized that --files-from isn't working how it was originally intended.
Currently --files-from parses the input files in the same way that --include-from does, namely it
ignores comments starting with # and ;
strips whitespace from start and end of strings
This makes --files-from less useful that it might be and was never the original intention.
I propose to make the backwards incompatible change so that --files-from reads every line from the file ignoring comment characters and not stripping whitespace.
This will restore the expected behaviour that rclone lsf is good input to --files-from when files have leading spaces, ; or #.
The expected impact of this is minimal I think as most --files-from lists are auto generated.
backwards incompatible compatibility changes are scary, very scary.
like a time traveler re-writing history in the same time line and thus changing the current future.
perhaps create a new flag for the new behavior and thereby create a new separate time line?
Well.. that is actually quite reasonable...
I mean, if we have a --files-fromv2 instead and keep the old then that is a really small inconvenience in return for full backwards compatibility while not having to compromise on the new function.
It can be made the default choice in an eventual rclone 2.0 maybe if you feel it is important to keep old "trash" commands from acumulating.
(Jojo knows well... you don't mess with the time-police... )
This would definitely better, however I think including comment in the search file might not be necessary for few reasons:
--files-from suggests that the search file is a list of file names, nothing more should be added.
there might be some file names and, for some crazy reason, it has a leading of # or ;.
backwards incompatible compatibility changes are scary, very scary.
That is right only for big change, however this change might not affect even remotely close to that.
Remember, the purpose of --files-from is to limited the looking files in the directory. The change is right with the purpose, more intuitive, easier to understand and use.
remember, @ncw, that our data are our children and we cannot, must not endanger them!
please, for the innocent children, i beg you, create a new flag!
Do you have a practical case when this change might affect this much?
From my experience, and I have been using this tag since I found out about it. The change is only making it better, more robust and negating any unexpected result.
So there is no practical case for the use of trimming leading/trailing whitespaces method.
I understand your concern, but it is a bit far fetched from a realistic point of view. We can't use the approach of creating a new flag everytime we need to make a change (might be useful in testing but certainly not in production).
I could make a smaller change which would just stop stripping the spaces on lines which would solve the original issue.
Another alternative would be to introduce a new flag like xargs does (help from xargs here)
-0, --null items are separated by a null, not whitespace;
disables quote and backslash processing and
logical EOF processing
-d, --delimiter=CHARACTER items in input stream are separated by CHARACTER,
not by whitespace; disables quote and backslash
processing and logical EOF processing
Or make a --files-from-raw flag which doesn't interpret the file at all...
I would be perfectly ok to just make the change and let it be what it is. I think the problems this could cause would be small and easily solvable. Most importantly I think we should not hold back on improvements just for the sake of 100% backwards compatibility. If we do that - we quickly run out of options due to being chained by earlier mistakes and sub-optimal choices.
But having --files-from-raw seems like an ok comprimise as well. Primarily because --files-from mostly works ok as-is and it won't matter much to most users and most use-cases - and the advanced users will hopefully be able to find what they need in the rare cases where this is required.
I guess another option would be to make the change so --files-from processes the file in a raw way, but warn the user for a while something like this if it detects a commented line or a line with leading/trailing whitespace
NOTICE in rclone versions before v1.51 lines in --files-from starting with '#' or ';' were ignored
NOTICE in rclone versions before v1.51 lines in --files-from had whitespace stripped
imho, i do not think that is a good idea but it is your choice, of course
to make a change in a flag's behaviour and then after the command is processing real data, to warn them.
and the notice would be in a log file? if so, the end-user might not notice the NOTICE and may not use a log file.
I appreciate your opinion - that is why I opened this thread to discuss with real users.
In general I don't like making backwards incompatible changes. For example I'd really like to change the interface to the ls command so that it works much more like the lsf command (doesn't recurse by default) etc, but I haven't because I don't want to break people's scripts.
Maybe in version 2.0.0 I could make some backwards incompatible changes...
I think for this case the smallest change might be to add a --files-from-delimeter flag. If it isn't set then --files-from works exactly as it is. If you do set it then files-from just splits at that delimiter.
The unix way would be to say --files-from-delimiter '\n'. Windows users might say --files-from-delimiter '\r\n' and power users --files-from-delimiter '\000'
As long as the problems are solved, I am completely happy with it.
However since you mention some thoughts about upcoming update. From my experience, expanding overlap flags (which only differs with slight changes) is a recipe for nightmare because there are always group of people who try new stuffs with new flags and become familiar with it. It would be a very difficult thing to reverse though.
)