thiagocosta
September 21, 2023, 6:18pm
1
Hello, I am using rclone 1.64 and I am trying to execute npm install into a mounted folder. The source folder is a NFS mounted directory. My mount command is:
rclone mount --cache-dir /opt/cachefs/cache/b7b4b8a8c68843e0b24c59fcc1dd5614/WorkCopy --file-perms 0777 --dir-perms 0777 --vfs-write-back 10 --attr-timeout 1 --allow-other --transfers 10 --vfs-cache-mode full -vv --links /opt/fsx/Integration/WorkCopy/b7b4b8a8c68843e0b24c59fcc1dd5614 /opt/cachefs/data/b7b4b8a8c68843e0b24c59fcc1dd5614/WorkCopy
I am trying top execute
npm install cronapp-lib-js@2.9.6-SP.8
and I am getting
npm ERR! code EIO
npm ERR! syscall symlink
npm ERR! path ../terser/bin/terser
npm ERR! dest /opt/techne/cronoside/cachefs/data/b7b4b8a8c68843e0b24c59fcc1dd5614/WorkCopy/src/main/webapp/node_modules/.bin/terser
npm ERR! errno -5
npm ERR! EIO: i/o error, symlink '../terser/bin/terser' -> '/opt/techne/cronoside/cachefs/data/b7b4b8a8c68843e0b24c59fcc1dd5614/WorkCopy/src/main/webapp/node_modules/.bin/terser'
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2023-09-21T18_10_39_619Z-debug-0.log
The -vv log tail is:
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/core/icons/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/: Remove: name="doc"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris: Added virtual directory entry vDel: "doc"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/core/icons/: Remove: name="tiny_mce"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/core/icons: Added virtual directory entry vDel: "tiny_mce"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/core/icons/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/core/: Remove: name="icons"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/core: Added virtual directory entry vDel: "icons"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/core/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/: Remove: name="core"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris: Added virtual directory entry vDel: "core"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/tiny_mce_wiris/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/: Remove: name="tiny_mce_wiris"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins: Added virtual directory entry vDel: "tiny_mce_wiris"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/plugins/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/: Remove: name="plugins"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce: Added virtual directory entry vDel: "plugins"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/tinymce/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/: Remove: name="tinymce"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins: Added virtual directory entry vDel: "tinymce"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/plugins/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/: Remove: name="plugins"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce: Added virtual directory entry vDel: "plugins"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/tinymce/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/: Remove: name="tinymce"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js: Added virtual directory entry vDel: "tinymce"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/js/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/: Remove: name="js"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist: Added virtual directory entry vDel: "js"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/dist/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/: Remove: name="dist"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js: Added virtual directory entry vDel: "dist"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/cronapp-lib-js/: >Remove: err=<nil>
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/: Remove: name="cronapp-lib-js"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules: Added virtual directory entry vDel: "cronapp-lib-js"
2023/09/21 18:17:36 DEBUG : src/main/webapp/node_modules/: >Remove: err=<nil>
What is the purpose of this mount?
You have NFS mounted directory:
/opt/fsx/Integration/WorkCopy/b7b4b8a8c68843e0b24c59fcc1dd5614
and you mount it using rclone to another local directory(?):
/opt/cachefs/data/b7b4b8a8c68843e0b24c59fcc1dd5614/WorkCopy
with cache in another local directory:
--cache-dir /opt/cachefs/cache/b7b4b8a8c68843e0b24c59fcc1dd5614/WorkCopy
thiagocosta
September 21, 2023, 6:36pm
3
The source directory in NFS is a very slow directory. I want when reading, copy to local disk and when writing, write to local disk and send to remove async
I think in order to proceed you have to find out what the issue is exactly. Based on your npm error it looks like there is some issue with symlinks.
Try mount without --links
Try manually creating symlinks in your mount etc.
I am afraid that npm installation debugging is beyond this forum scope so you have to identify rclone related issue first.
On the second thought IMO it is impossible to make it work as mount does not support symlinks and your npm program needs them.
thiagocosta
September 22, 2023, 10:44am
6
I have tried with —links and —copy-links and no look. I will try create a symlink manually and send here
You can but I am pretty much sure it won't work. I have tried on my machine using only local directories and it does not work.
You better spend time on trying to improve speed of your NFS mount.
It was clever and creative idea you had about utilising rclone as NFS speed booster but unfortunately links are not supported.
ncw
September 22, 2023, 2:37pm
8
True. However there is a PR that I'd like to merge which does implement this!
1 Like
thiagocosta
September 22, 2023, 3:09pm
9
Can you send the link of PR ?
I believe this is the one:
rclone:master ← pasnox:master
opened 05:53PM - 24 Jul 22 UTC
#### What is the purpose of this change?
Add mount / vfs symlink support
Ple… ase, note that it has only been tested with storj backend on linux.
Especially, macOS, windows were not tested.
Please do test those platforms and any other backends !!!
#### Was the change discussed in an issue or in the forum before?
https://github.com/rclone/rclone/issues/2975
#### Checklist
- [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix).
- [x] I have added tests for all changes in this PR if appropriate.
- [ ] I have added documentation for the changes if appropriate.
- [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [x] I'm done, this Pull Request is ready for review :-)
#### Notes
The whole work has been (trashed mostly) refactored. They were flaws in the previous implementation, especially trying to use truncated names in the vfs dir internal nodes names which lead to inconsistency when it comes to push changes to the remotes.
For this reason, the refactoring went in 2 parts:
1. Add symlinks support in the VFS itself, in the form of always storing symlinks with the .rclonelink suffix, those being regular files.
The VFS Readlink and Symlink additions allow support for symlinks, even if the --links command line switch is not enabled, it's just a mater of creating a .rclonelink regular file with the target inside.
The only difference is that when --link is enabled, the VFS nodes Mode() would report the **os.ModeSymlink** flag hint.
This approach limit adding regressions in the VFS itself as there is no more names tweaking.
Some more checks were also added, to avoid to allow creating / moving files / links / directories names that would clash.
By example, the VFS must not be able to create in the same directory something called **Hello.txt** and something called **Hello.txt.rclonelink**, this is because when using the mount operation with --links enabled, we would end up with both files claiming the same node name: **Hello.txt**.
2. Add symlinks support in the mount backends, they are required to translate from the full vfs names into truncated names to the end user: **Test.rclonelink** would be seen as **Test**.
When --links command line switch is enabled, the file system will translate any ***.rclonelink** files into * links.
On the other hand, if --links is disabled, the symlinks will be seen as regular files, with their .rclonelink suffix which content contains the target file path.
Those changes comes with a rather big TestSymlinks unit test, which test all possible api and combinations.
The tests are ran for a VFS and each backends, with and without --links command line switch enabled, covering all possible scenarios.
I did run the tests successfully using this bash script:
```
#!/bin/bash -e
# Run the VFS tests in all subdirectories
pushd vfs
go test -tags cmount -v ./...
#go test -tags cmount -v -run TestFunctional
popd
# This test uses the tests in vfs/vfstest
pushd cmd/cmount
go test -tags cmount -v
popd
# This test uses the tests in vfs/vfstest
pushd cmd/mount
go test -v
popd
# This test uses the tests in vfs/vfstest
pushd cmd/mount2
go test -v
popd
```
If you are familiar with git and building from source you could give it a try for your use case.
thiagocosta
September 25, 2023, 1:19pm
11
I will try and give you a feedback
thiagocosta
September 25, 2023, 2:30pm
12
Hello,
The symlink error seems to be resolved, but now I am having errors with remove:
$ npm install cronapp-lib-js@2.9.6-SP.8
npm WARN webapp@1.0.0 No description
npm WARN webapp@1.0.0 No repository field.
npm ERR! code EIO
npm ERR! syscall rmdir
npm ERR! path /home/project/src/main/webapp/node_modules/.staging/@uirouter
npm ERR! errno -5
npm ERR! EIO: i/o error, rmdir '/home/project/src/main/webapp/node_modules/.staging/@uirouter'
npm ERR! A complete log of this run can be found in:
npm ERR! /home/.npm/_logs/2023-09-25T14_26_36_561Z-debug.log
$ rm -rf node_modules/*
rm: cannot remove 'node_modules/@cronapp/framework-core': Input/output error
rm: cannot remove 'node_modules/chart.js/node_modules/moment/src/locale': Input/output error
rm: cannot remove 'node_modules/chart.js/test/fixtures/core.scale': Input/output error
rm: cannot remove 'node_modules/chart.js/test/fixtures/plugin.filler': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/SourceSansPro': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/WorkSans': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/arial-narrow': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/calibri': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/georgia': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/glyphicons-halflings': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/helvetica-neue': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/inter': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/lato': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/material-icons': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/news-cycle': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/open-sans': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/raleway': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/rawline': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/roboto': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/times-new-roman': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/fonts/ubuntu': Input/output error
rm: cannot remove 'node_modules/cronapp-framework-js/css/privacy': Input/output error
...
I think you should share your findings with this PR author.
Actually I see he is here - @pasnox something for you.
thiagocosta
September 25, 2023, 2:49pm
14
The same error occurs on npm repositories without symlinks and using rclone 1.64. @pasnox can help ?
The error you shared doesn't mean much as there's no rclone debug log.
Pretty please re-create the issue and share the full rclone debug log.
system
September 28, 2023, 2:54pm
16
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.