Chunker wastes time and traffic

Feature
1

What is the problem you are having with rclone?

chunker does not discover already uploaded chunks and therefore does not help in case of connection issues.

I try to upload a 6 GB file, and when something goes wrong rclone cancels any progress it had, even if there were any chunks successfully uploaded before that. It then tries to upload all the same chunks again and again. Lucky if it actually gets the job done, but it wastes tons of time and traffic in the process.

See chunker - broken or intended? · Issue #7052 · rclone/rclone · GitHub

Run the command 'rclone version' and share the full output of the command.

v1.63.0-beta.7056.5d82e90ff.fix-7044-yandex-bad-request

Which cloud storage system are you using? (eg Google Drive)

yandex

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -vv --dump responses --contimeout 10m E:\\folder\\6Gfile chunker-test:temp --log-file rclone.log --disable-http-keep-alives

The rclone config contents with secrets removed.

[yd-test]
type = yandex
token = *redacted*

[chunker-test]
type = chunker
remote = yd-test:chunker
chunk_size = 512Mi
hash_type = none

A log from the command with the -vv flag

rclone-chunker.log (1.6 MB)

2

This is how chunker works at the moment. It lacks functionality to resume interrupted transfers.

3

Any chance this gets attention and eventually gets improved?

Yes, developers may assume there are too few users affected. But just think about this: a user will not be able to upload a big file at all, no matter how small the chunk size they choose. With this amount of supported remotes, there is no way to be sure they all work fine and not disconnect randomly.

Think about why rclone must remove what it uploaded successfully, and upload again.

I don't tnink it is difficult to make rclone create temporary file somewhere to understand which chunks for current file were uploaded successfully to avoid removing them from the remote and uploading again.

4

The issue is recorded here and on github and anybody can try to fix it.

Feel free to implement required changes and submit PR - it is open source project at the end.

5

Yeah, this is the issue that also leads to orphaned chunks too...

6

Yes this is known issue. One of the few recorded for chunker.

Nothing is perfect:) Maybe now with growing popularity of box somebody will try to fix it.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.