Why RCLONE would discard all the parts of a large file when it failed to upload just one part due to network connection lost?

Help and Support
1

What is the problem you are having with rclone?

RCLONE would discard all the parts of a large file when it failed to upload just one part due to network connection lost.
When RCLONE discards file parts, all the parts will be put in the recycle bin of OneDrive. I checked discarded parts, and these parts are OK and the data are not corrupt.

Expected RCLONE to do:
When RCLONE fails to upload just one part, just discard this part, and keep all parts uploaded successfully. Then retry to upload the failed one.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0

  • os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
  • os/kernel: 10.0.19045.3086 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.5
  • go/linking: static
  • go/tags: cmount

Currently, the latest RCLONE version offered on the download page is 1.63.1

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --vfs-cache-mode full --bwlimit 500k:off --temp-dir=F:\temp\rclone chunker: f:\dsshare\ds

The rclone config contents with secrets removed.

[ds]
type = onedrive
token = XXXXXXXXXX
drive_type = business

[crypt]
type = crypt
remote = ds:
filename_encryption = off
directory_name_encryption = false
password = xxxxxxx
password2 = xxxxxxx

[chunker]
type = chunker
remote = crypt:
chunk_size = 1Gi

A log from the command with the -vv flag

2023/07/18 16:56:46 DEBUG : DigitalStudio/XXXXXXXXX/4H8A1611.MOV.rclone_chunk.005_yrw1zz.bin: Cancelling multipart upload: Put "https://rococobee-my.sharepoint.com/personal/rclonedigitalstudio_rococobee_onmicrosoft_com/_api/v2.0/drives/b!n_5O43vC2E-0cXXXXXXXXXXXXXXXXXXX": http2: client connection force closed via ClientConn.Close
2023/07/18 16:56:46 DEBUG : DigitalStudio/XXXXXXXXXX/4H8A1272.MOV.rclone_chunk.005_yrw0xr.bin: Cancelling multipart upload: Put "https://rococobee-my.sharepoint.com/personal/rclonedigitalstudio_rococobee_onmicrosoft_com/_api/v2.0/drives/b!n_5O43vC2E-0ci3ssKHhRXH2URSo46BEsawnFrFHxp7t1VxLgT6wQqMkMzbSRqF1/items/01AGAADGY2BB2TFLV25NEZMKF4QREDDUYB/uploadSession?guid='7fa4f73c-be6f-44d7-b8cc-49d4e19244b6'&overwrite=True&rename=False&dc=0&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvcm9jb2NvYmVlLW15LnNoYXJlcG9pbnQuY29tQDgxNjdjMjc2LWZlZTgtXXXXXXXXXXXXXXXXXXXXX": http2: client connection force closed via ClientConn.Close
2023/07/18 16:56:46 DEBUG : DigitalStudio/xxxxxxxxxx/4H8A3672.MOV.rclone_chunk.005_yrw26x.bin: Cancelling multipart upload: Put 
........
.....cafOA8Un5rts8zmsGrJKxMc": http2: client connection force closed via ClientConn.Close
2023/07/18 16:56:46 DEBUG : pacer: low level retry 1/10 (error Put "https://rococobee-my.sharepoint.com/personal/rclonedigitalstudio_rococobee_onmicrosoft_com/_api/v2.0/drives/b!n_5O43vC2E-XXXXXXXXXXXXXXXXXXXXXXXXX": write tcp 192.168.50.249:60317->13.107.138.8:443: wsasend: An established connection was aborted by the software in your host machine.)
2023/07/18 16:56:46 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2023/07/18 16:56:47 DEBUG : pacer: Reducing sleep to 15ms
2023/07/18 16:56:47 DEBUG : pacer: Reducing sleep to 11.25ms
2023/07/18 16:56:47 DEBUG : pacer: Reducing sleep to 10ms
2

Not bug. This is by design unfortunately and is clearly documented:

When upload completes, temporary chunk files are finally renamed. This scheme guarantees that operations can be run in parallel and look from outside as atomic. A similar method with hidden temporary chunks is used for other operations (copy/move/rename, etc.). If an operation fails, hidden chunks are normally destroyed, and the target composite file stays intact.

It can be only properly implemented when this PR is finished:

It has been open for years.... If you know go maybe you can help with it.

What it worse with chunker is that chunks can be left behind and stay invisible forever. In your case you use ClientConn.Close so rclone gracefully cleans things. But when termination is more abrupt things will be left behind...

I have just posted workaround for this problem:

1 Like
3

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.