I'm trying to sync a folder which contains node_modules directory. This directory contains symlinks.
I'm trying to exclude the node_modules directory. I'm using
/usr/local/bin/rclone --exclude "**node_modules/" -v copy $HOME/Documents/ gdrive-crypt:Documents/
Yet, I still get error messages like,
2017/05/11 16:39:30 NOTICE: Github Repos/hello-electron-with-pouchdb/node_modules/.bin/window-size: Can't follow symlink without -L/--copy-links
To test the --exclude, I have tried,
rclone ls ~/Documents/Github\ Repos/hello-electron-with-pouchdb/ --exclude "**node_modules/"
And it works.
All I can think of is that it's validating the symlink before excluding?
Not sure if anyone has insight into it. I'm just trying to optimize my script, and found this.
1 Like
Try doing".*node_modules.*"
calisro
May 11, 2017, 11:28pm
3
This should work.
--exclude="**/node_modules/**"
calisro
May 11, 2017, 11:32pm
4
Animosity022:
".node_modules. "
You need two stars.
HS -> rclone ls Repos --exclude=".*node_modules.*"
0 hello-electron-with-pouchdb/qweqeqeq
0 hello-electron-with-pouchdb/node_modules/.bin/window-size/dsf
That’s goofy. My regex should work without an issue, but it doesn’t.
felix@plex:~$ rclone ls ACD:test --exclude '.*node_modules.*'
2108949 node_modules/image002.png
felix@plex:~$ rclone ls ACD:test | grep -v '.*node_modules.*'
felix@plex:~$
Docs listed seems to be that it is good -> https://golang.org/pkg/regexp/syntax/
calisro
May 12, 2017, 12:00am
6
Yeah it’s a little odd. The star match anything but slashes. To match including slashes it’s two stars. It isn’t regex. It’s based on unix style file globs according to the docs with some oddities like the two stars.
The * should match 0 or more characters of anything before it.
With the “.” before it, that should be any character. I’m guessing the “.” isn’t really any character.
calisro
May 12, 2017, 12:06am
8
With regex, yes. But it’s not regex.
Based on the linked GO regex page, it says “Any character”. I was basing off that not the regeex docs.
calisro
May 12, 2017, 12:11am
10
Yep. But the only reference on rclone site for the go regex docs is for character classes. It isn’t full regex implementation in rclone…
https://rclone.org/filtering/
Okay, first recommendation,
rclone ls ~/Documents/Github\ Repos/hello-electron-with-pouchdb/ --exclude "*.node_modules.*" | tail
is not working,
2017/05/11 22:35:52 NOTICE: node_modules/sqlite3/node_modules/node-pre-gyp/node_modules/request/node_modules/http-signature/node_modules/.bin/sshpk-sign: Can't follow symlink without -L/--copy-links
2017/05/11 22:35:52 NOTICE: node_modules/sqlite3/node_modules/node-pre-gyp/node_modules/request/node_modules/http-signature/node_modules/.bin/sshpk-verify: Can't follow symlink without -L/--copy-links
41 node_modules/lodash/fp/zipObj.js
164 node_modules/lodash/fp/trimCharsEnd.js
171 node_modules/lodash/fp/zipObjectDeep.js
159 node_modules/lodash/fp/zipWith.js```
---------------
Second, `--exclude="**/node_modules/**"
`rclone ls ~/Documents/Github\ Repos/hello-electron-with-pouchdb/ --exclude "**/node_modules/**" | tail`
Not working,
```17/05/11 22:33:36 NOTICE: node_modules/electron/dist/Electron.app/Contents/Frameworks/Squirrel.framework/Resources: Can't follow symlink without -L/--copy-links
2017/05/11 22:33:36 NOTICE: node_modules/electron/dist/Electron.app/Contents/Frameworks/Squirrel.framework/Squirrel: Can't follow symlink without -L/--copy-links
2017/05/11 22:33:36 NOTICE: node_modules/electron/dist/Electron.app/Contents/Frameworks/ReactiveCocoa.framework/Versions/Current: Can't follow symlink without -L/--copy-links
2017/05/11 22:33:36 NOTICE: node_modules/electron/dist/Electron.app/Contents/Frameworks/Squirrel.framework/Versions/Current: Can't follow symlink without -L/--copy-links
41 node_modules/lodash/fp/zipObj.js
154 node_modules/lodash/fp/zipAll.js
163 node_modules/lodash/fp/zipObject.js
171 node_modules/lodash/fp/zipObjectDeep.js
199 node_modules/lodash/fp/wrapperLodash.js
151 node_modules/lodash/fp/xor.js
191 node_modules/lodash/fp/upperCase.js
159 node_modules/lodash/fp/zipWith.js
159 node_modules/lodash/fp/without.js
199 node_modules/lodash/fp/toSafeInteger.js```
-------
@calisro could you check creating a test symlink.
mkdir test-rclone
touch a b c d
mkdir node_modules
touch node_modules/{a,b,c}
ln -s node_modules/c node_modules/d
rclone ls -v rclone-test/ --exclude="**/node_modules/**"
That should still trigger the error,
rclone ls -v rclone-test/ --exclude="**/node_modules/**"
2017/05/11 22:40:28 INFO : Local file system at /Users/me/tmp/rclone-test: Modify window is 1s
0 a
0 c
0 b
2017/05/11 22:40:28 NOTICE: node_modules/d: Can't follow symlink without -L/--copy-links
0 node_modules/a
0 node_modules/c
0 node_modules/b
Using `rclone ls -v rclone-test/ --exclude="**node_modules**"` works.
But it doesn't work with a `copy`. The message regarding `node_modules/d` is still there.
rclone copy -v ./rclone-test/ gdrive-crypt:/rclone-test/
2017/05/11 22:45:46 INFO : Encrypted Google drive root 'rclone-crypt/dpgovukoo3qomo1rtol7tje29k': Modify window is 1s
2017/05/11 22:45:47 NOTICE: node_modules/d: Can't follow symlink without -L/--copy-links
2017/05/11 22:45:47 INFO : Encrypted Google drive root 'rclone-crypt/dpgovukoo3qomo1rtol7tje29k': Waiting for checks to finish
2017/05/11 22:45:47 INFO : Encrypted Google drive root 'rclone-crypt/dpgovukoo3qomo1rtol7tje29k': Waiting for transfers to finish
2017/05/11 22:45:49 INFO : node_modules/a: Copied (new)
2017/05/11 22:45:49 INFO : b: Copied (new)
2017/05/11 22:45:49 INFO : c: Copied (new)
2017/05/11 22:45:50 INFO : a: Copied (new)
2017/05/11 22:45:50 INFO : node_modules/b: Copied (new)
2017/05/11 22:45:50 INFO : node_modules/c: Copied (new)
2017/05/11 22:45:50 INFO :
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 0
Transferred: 6
Elapsed time: 6.4s
calisro
May 12, 2017, 1:00pm
12
It seems to be working.
In this case, you started the ls WITHIN that directory. Because of that, it won’t trigger because there is no slash prefixing the path.
rclone ls -v rclone-test/ --exclude="**/node_modules/**"
2017/05/11 22:40:28 INFO : Local file system at /Users/me/tmp/rclone-test: Modify window is 1s
0 a
0 c
0 b
2017/05/11 22:40:28 NOTICE: node_modules/d: Can't follow symlink without -L/--copy-links
0 node_modules/a
0 node_modules/c
0 node_modules/b
But if you change it to this (based on your starting path) this will work for THAT use case.
rclone ls -v rclone-test/ --exclude="**node_modules/**"
If you know for certain that node_modules will always be in the root and you want to anchor it that way, then this would be okay too:
rclone ls -v rclone-test/ --exclude="node_modules/**"
1 Like