schklom
September 14, 2022, 2:27am
1
Hello,
Nextcloud has a client that automatically chunks files during sync.
I can do something similar with the WebDAV
and chunker
remotes, but the problem is that the unchunked files can only be retrieved through Rclone.
The Nextcloud client manages to unchunk on upload to the server and on download to the client. The chunks are transparent to the user.
Can this be done with Rclone? I want to have an automatic upload only (one-way sync) setup to a Nextcloud server, and the Nextcloud client cannot do this.
I also want the files on the server to be readable by people who do not use Rclone, in other words I would like the chunker
remote to be transparent instead of leaving chunks on the server.
A solution could be a general unchunk
remote/command, or a Nextcloud_chunker
remote. The Nextcloud client code is open-source (GitHub - nextcloud/desktop: 💻 Desktop sync client for Nextcloud ), so this should be possible.
Is either option doable?
Thanks
ncw
(Nick Craig-Wood)
September 14, 2022, 9:46am
2
There is a nextcloud chunked upload stuck in pull request review:
rclone:master
← devnoname120:feat-webdav-nextcloud-parallel-chunk-upload
opened 12:09PM - 17 May 22 UTC
Follow-up of https://github.com/rclone/rclone/pull/6133
This will pave the wa… y as an example of implementation for https://github.com/rclone/rclone/issues/4798 and https://github.com/rclone/rclone/issues/5156.
In my tests I was able to max out the download speed, and the upload workers were not used as much as they could. This is due to the fact that there is a single download thread (rclone limitation).
**TODO**:
- [ ] Backport extra changes & refactoring from https://github.com/rclone/rclone/pull/6133
- [ ] Fix chunks purging on `SIGTERM`. See https://gist.github.com/echohes/8f6b1002f99f2305456cc1fa6b224438 for inspiration.
- [x] Stop using `numParallelUploads` for the number of workers to use for each upload. Instead find if rclone provides a config value, and otherwise use heuristics / remote config.
- Added an extra config to the remote: `nextcloud_chunk_parallel_transfers`
- [x] Fix download thread downloading an unlimited number of chunks even if the upload workers are far behind. If download is faster than upload, then it hogs up all the available memory.
<!--
Thank you very much for contributing code or documentation to rclone! Please
fill out the following questions to make it easier for us to review your
changes.
You do not need to check all the boxes below all at once, feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box.
-->
#### What is the purpose of this change?
<!--
Describe the changes here
-->
#### Was the change discussed in an issue or in the forum before?
<!--
Link issues and relevant forum posts here.
-->
#### Checklist
- [ ] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix).
- [ ] I have added tests for all changes in this PR if appropriate.
- [ ] I have added documentation for the changes if appropriate.
- [ ] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [ ] I'm done, this Pull Request is ready for review :-)
You could give that a try
1 Like
system
(system)
Closed
November 13, 2022, 9:47am
3
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.