Issues with exclude and symlinks

Help and Support
1

I’m trying to sync a folder which contains node_modules directory. This directory contains symlinks.

I’m trying to exclude the node_modules directory. I’m using

/usr/local/bin/rclone --exclude "**node_modules/" -v copy $HOME/Documents/ gdrive-crypt:Documents/

Yet, I still get error messages like,

2017/05/11 16:39:30 NOTICE: Github Repos/hello-electron-with-pouchdb/node_modules/.bin/window-size: Can’t follow symlink without -L/–copy-links

To test the --exclude, I have tried,

rclone ls ~/Documents/Github\ Repos/hello-electron-with-pouchdb/ --exclude "**node_modules/"

And it works.

All I can think of is that it’s validating the symlink before excluding?

Not sure if anyone has insight into it. I’m just trying to optimize my script, and found this.

1 Like
2

Try doing
".*node_modules.*"

3

This should work.

--exclude="**/node_modules/**"

4

You need two stars.

HS -> rclone ls Repos --exclude=".*node_modules.*"
        0 hello-electron-with-pouchdb/qweqeqeq
        0 hello-electron-with-pouchdb/node_modules/.bin/window-size/dsf
5

That’s goofy. My regex should work without an issue, but it doesn’t.

felix@plex:~$ rclone ls ACD:test --exclude '.*node_modules.*'
  2108949 node_modules/image002.png
felix@plex:~$ rclone ls ACD:test | grep -v '.*node_modules.*'
felix@plex:~$

Docs listed seems to be that it is good -> https://golang.org/pkg/regexp/syntax/

6

Yeah it’s a little odd. The star match anything but slashes. To match including slashes it’s two stars. It isn’t regex. It’s based on unix style file globs according to the docs with some oddities like the two stars.

7

The * should match 0 or more characters of anything before it.

With the “.” before it, that should be any character. I’m guessing the “.” isn’t really any character.

8

With regex, yes. But it’s not regex.

9

Based on the linked GO regex page, it says “Any character”. I was basing off that not the regeex docs.

10

Yep. But the only reference on rclone site for the go regex docs is for character classes. It isn’t full regex implementation in rclone…

https://rclone.org/filtering/

11

Okay, first recommendation,

rclone ls ~/Documents/Github\ Repos/hello-electron-with-pouchdb/ --exclude "*.node_modules.*" | tail

is not working,

2017/05/11 22:35:52 NOTICE: node_modules/sqlite3/node_modules/node-pre-gyp/node_modules/request/node_modules/http-signature/node_modules/.bin/sshpk-sign: Can't follow symlink without -L/--copy-links
2017/05/11 22:35:52 NOTICE: node_modules/sqlite3/node_modules/node-pre-gyp/node_modules/request/node_modules/http-signature/node_modules/.bin/sshpk-verify: Can't follow symlink without -L/--copy-links
       41 node_modules/lodash/fp/zipObj.js
      164 node_modules/lodash/fp/trimCharsEnd.js
      171 node_modules/lodash/fp/zipObjectDeep.js
      159 node_modules/lodash/fp/zipWith.js```

---------------
Second, `--exclude="**/node_modules/**"

`rclone ls ~/Documents/Github\ Repos/hello-electron-with-pouchdb/ --exclude "**/node_modules/**" | tail`

Not working,

```17/05/11 22:33:36 NOTICE: node_modules/electron/dist/Electron.app/Contents/Frameworks/Squirrel.framework/Resources: Can't follow symlink without -L/--copy-links
2017/05/11 22:33:36 NOTICE: node_modules/electron/dist/Electron.app/Contents/Frameworks/Squirrel.framework/Squirrel: Can't follow symlink without -L/--copy-links
2017/05/11 22:33:36 NOTICE: node_modules/electron/dist/Electron.app/Contents/Frameworks/ReactiveCocoa.framework/Versions/Current: Can't follow symlink without -L/--copy-links
2017/05/11 22:33:36 NOTICE: node_modules/electron/dist/Electron.app/Contents/Frameworks/Squirrel.framework/Versions/Current: Can't follow symlink without -L/--copy-links
       41 node_modules/lodash/fp/zipObj.js
      154 node_modules/lodash/fp/zipAll.js
      163 node_modules/lodash/fp/zipObject.js
      171 node_modules/lodash/fp/zipObjectDeep.js
      199 node_modules/lodash/fp/wrapperLodash.js
      151 node_modules/lodash/fp/xor.js
      191 node_modules/lodash/fp/upperCase.js
      159 node_modules/lodash/fp/zipWith.js
      159 node_modules/lodash/fp/without.js
      199 node_modules/lodash/fp/toSafeInteger.js```


-------

@calisro could you check creating a test symlink. 

    mkdir test-rclone
    touch a b c d
    mkdir node_modules
    touch node_modules/{a,b,c}
    ln -s node_modules/c node_modules/d
    rclone ls -v rclone-test/ --exclude="**/node_modules/**"

That should still trigger the error,

    rclone ls -v rclone-test/ --exclude="**/node_modules/**"
    2017/05/11 22:40:28 INFO  : Local file system at /Users/me/tmp/rclone-test: Modify window is 1s
        0 a
        0 c
        0 b
    2017/05/11 22:40:28 NOTICE: node_modules/d: Can't follow symlink without -L/--copy-links
        0 node_modules/a
        0 node_modules/c
        0 node_modules/b

Using `rclone ls -v rclone-test/ --exclude="**node_modules**"` works.

 

But it doesn't work with a `copy`. The message regarding `node_modules/d` is still there.

    rclone copy -v ./rclone-test/ gdrive-crypt:/rclone-test/
    2017/05/11 22:45:46 INFO  : Encrypted Google drive root 'rclone-crypt/dpgovukoo3qomo1rtol7tje29k': Modify window is 1s
    2017/05/11 22:45:47 NOTICE: node_modules/d: Can't follow symlink without -L/--copy-links
    2017/05/11 22:45:47 INFO  : Encrypted Google drive root 'rclone-crypt/dpgovukoo3qomo1rtol7tje29k': Waiting for checks to finish
    2017/05/11 22:45:47 INFO  : Encrypted Google drive root 'rclone-crypt/dpgovukoo3qomo1rtol7tje29k': Waiting for transfers to finish
    2017/05/11 22:45:49 INFO  : node_modules/a: Copied (new)
    2017/05/11 22:45:49 INFO  : b: Copied (new)
    2017/05/11 22:45:49 INFO  : c: Copied (new)
    2017/05/11 22:45:50 INFO  : a: Copied (new)
    2017/05/11 22:45:50 INFO  : node_modules/b: Copied (new)
    2017/05/11 22:45:50 INFO  : node_modules/c: Copied (new)
    2017/05/11 22:45:50 INFO  :
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            6
    Elapsed time:        6.4s
12

It seems to be working.

In this case, you started the ls WITHIN that directory. Because of that, it won’t trigger because there is no slash prefixing the path.

rclone ls -v rclone-test/ --exclude="**/node_modules/**"
2017/05/11 22:40:28 INFO  : Local file system at /Users/me/tmp/rclone-test: Modify window is 1s
    0 a
    0 c
    0 b
2017/05/11 22:40:28 NOTICE: node_modules/d: Can't follow symlink without -L/--copy-links
    0 node_modules/a
    0 node_modules/c
    0 node_modules/b

But if you change it to this (based on your starting path) this will work for THAT use case.

rclone ls -v rclone-test/ --exclude="**node_modules/**"

If you know for certain that node_modules will always be in the root and you want to anchor it that way, then this would be okay too:

rclone ls -v rclone-test/ --exclude="node_modules/**"

1 Like