RClone skips files ending in unicode whitespace when using ---files-from

Suspected Bug
1

What is the problem you are having with rclone?

If the source has a file that ends with a unicode whitespace character (eg: u'\u2028'), rclone would skip that file when using --files-from

What is your rclone version (output from rclone version)

v1.50.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18.04 (though OS is irrelevant)

Which cloud storage system are you using? (eg Google Drive)

Issue can be replicated irrespective of OS as this is an issue in the filtering

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Perform following commands:

rclone lsf <source_path> > include-file.txt
rclone copy <source_path> <remote_name>:<remote_path> --files-from include-file.txt -vv

The issue is because while adding files to the FilesFrom map during filters creation, we use strings.TrimSpace, which also trims the unicode whitespace characters. So the FilesFrom map now contains incorrect file names, and later when walking, we try to find the correct file name (with the unicode whitespace) in the FilesFrom map, and when we do not find it, the file is excluded.

2

This has been discussed before in the forum without getting to a conclusion

Here is the code

I'd quite like to not trim spaces or check for comments but I'm a little concerned about backwards compatibility.

One alternative would be an alternate flag, say --files-from-raw which just takes a \n delimited file (or maybe \r\n) and does no processing.

Do you fancy working on this?

3

Sure. I'll send over a PR soon.

4

Thank you very much :slight_smile:

5

Going through the original thread, I just realized this might not be the first time I ran into this issue and read that thread. Maybe just the first time I decided to actually report it. :smiley:

6

Here's the PR:

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.