Exclude Icon␍OSX files

Help and Support
1

What is the problem you are having with rclone?

Hi, this is more of a "what am I doing wrong" sort of question, but how do I exclude errors relating to those wretched OSX Icon\r files appearing in my log file ? I understand these files are used by OSX where a custom folder icon has been used. I also understand that these filenames are illegal. I don't want to backup the files, I'd just like to filter then so they don't appear in my rclone log as errors, forcing a retry.

I'm backing up a number of macs to an RPi running rsyncd, then using rclone to back that up to dropbox

In my exclude file I've tried

*Icon\r

Which is the syntax I use for Thumbs.db files and works fine

I'd rather not use Icon? as that syntax is too broad

I saw threads suggesting this was resolved in 1.50, but perhaps I'm misunderstanding how to use it. I also not some people are using filter files rather than exclude files, I'm not sure of the difference ?

Thank you

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: linux/arm

  • go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian RPi 32 bit

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -q --tpslimit=9 --ignore-errors --exclude-from /etc/rclone_exclude.txt /Archive/rsync Dropbox:Pi/rsync >> /var/tmp/log/rclone.log 2>&1

The rclone config contents with secrets removed.

A log from the command with the -vv flag

2021/01/16 11:37:12 ERROR : Icon␍: Failed to copy: upload failed: path/disallowed_name/.
2021/01/16 11:37:15 ERROR : Camera Uploads/Icon␍: Failed to copy: upload failed: path/disallowed_name/.
2021/01/16 11:37:16 ERROR : Photos/Icon␍: Failed to copy: upload failed: path/disallowed_name/
2021/01/16 11:37:17 ERROR : Public/Icon␍: Failed to copy: upload failed: path/disallowed_name/..
2021/01/16 11:37:20 ERROR : User Library/Icon␍: Failed to copy: upload failed: path/disallowed_name/.
2

You can test with rclone ls.

Can you share what the actual output of the file name is using rclone ls? You can use a filter file / exclude or whatever you want, but just share the full command and the filter file if you are using it.

3

if I run rclone with the ls flag, the offending files look like this when I vi the log file:

0 <directory path>/Icon␍

The problem is that the offending filenames contain a carriage return (^M) for some strange reason known only to apple.

Thank you, my apologies, here is the syntax from my exclude file:

*Thumbs.db
*.DS_Store
*.dropbox
*.~lock.*
*~$*
*desktop.ini*
iTunes/**
GarageBand/**
Downloads/**
Library/**
.Trash/**
*Icon\r
Applications/**
4

Can you pick one directory and share the full rclone ls output for that directory?

felix@gemini:~/test$ rclone ls /home/felix/test -vv
2021/01/16 07:38:10 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "ls" "/home/felix/test" "-vv"]
2021/01/16 07:38:10 DEBUG : Creating backend with remote "/home/felix/test"
2021/01/16 07:38:10 DEBUG : Using config file from "/opt/rclone/rclone.conf"
      225 hosts
        0 test1/blah
        0 test1/test2/blah2
2021/01/16 07:38:10 DEBUG : 2 go routines active
5 
2021/01/16 12:46:30 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "ls" "/Archive/rsync/George/Dropbox/Public/" "-vv"]
2021/01/16 12:46:30 DEBUG : Creating backend with remote "/Archive/rsync/George/Dropbox/Public/"
2021/01/16 12:46:30 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
      675 How to use the Public folder.txt
        0 Icon␍
2021/01/16 12:46:30 DEBUG : 2 go routines active```
6

Can you try that same ls with

--exclude 'Icon.'

I'd wonder how it's globbing that return but a "." should match any one character.

7

Thank you. So can I also use that syntax in an excludes file ?

I will try it, but a potential problem is that it would also match potentially legitimate files/directores called Icons ?

This thread appears to be the same problem, so I'm interested to understand how this was addressed in 1.50

https://forum.rclone.org/t/filtering-files-with-r/2296/4

8

No, it won't match directories.

I'd actually try a few things:

The . may match, but I'd try

--exclude Icons\r\n

As an example:

felix@flux:~/test$ rclone ls /home/felix/test --exclude 'Icons1'
        0 Icons1/Icons2
        0 Icons1/blah
felix@flux:~/test$ rclone ls /home/felix/test
        0 Icons1/Icons1
        0 Icons1/Icons2
        0 Icons1/blah
9

You can use a character class to match \r is probably the neatest way, so

--exclude 'Icons[\r]'

Or this in your exclude file

Icons[\r]

Will match one character \r only.

You can put these things in a character class (from the go regexp docs)

x              single character
A-Z            character range (inclusive)
\d             Perl character class
[:foo:]        ASCII character class foo
\p{Foo}        Unicode character class Foo
\pF            Unicode character class F (one-letter name)

That said, in the internal encoding rclone uses, \r will have been transformed to a unicode ␍ character, so I think

Icon␍

Might be what you want.

10

Thank you. Last night I ran it with the exclude file entry of

*Icon?

And that worked, the rclone ran without error, but tonight I will try the following which avoids the remote possibility of false positives i.e Icons

*Icon[\r]
11

I think "Icon␍" is what you want - I created an Icon\r file with python and then did this test

$ ls
'Icon'$'\r'
$ rclone lsf .
Icon␍
$ rclone lsf . --exclude "Icon␍"
$ rclone lsf . --exclude "Icon[\r]"
Icon␍
12

Thank you Nick, yes I can confirm that last nights cron job gave the same errors when I had

*Icon[\r]

in my exclude file. Tonight I will use the syntax you suggest with the actual ␍ character. Thank you

1 Like
13

OK so in my exclude file I tried Nicks suggestion

*Icon␍

And this worked perfectly, so many thanks for that

2 Likes
14

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.