Corrupted on transfer while syncing from NextCloud

Help and Support
1

What is the problem you are having with rclone?

I am trying to sync a Nextcloud remote and a local copy, and some files do not sync properly, and fail with a "corrupted data" message.

Run the command 'rclone version' and share the full output of the command.

$ rclone --version
rclone v1.66.0
- os/version: raspbian 12.5 (64 bit)
- os/kernel: 6.6.20+rpt-rpi-v8 (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.22.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Using NextCloud 28.0.4, on a self-hosted server, with NGINX (PHP-FPM) and PHP 8.2.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

viana@nas-viana:~/Downloads $ rclone sync nextcloud:"Music/AC_DC/Blow Up Your Video/06 Nick Of Time.mp3" . -v -v

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

viana@nas-viana:~/Downloads $ rclone config redacted
[nextcloud]
type = webdav
url = https://server/remote.php/dav/files/username
vendor = nextcloud
user = XXX
description = nextcloud
bearer_token = XXX
pass = XXX
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2024/04/12 14:44:46 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "sync" "nextcloud:Music/AC_DC/Blow Up Your Video/06 Nick Of Time.mp3" "." "-v" "-v"]
2024/04/12 14:44:46 DEBUG : Creating backend with remote "nextcloud:Music/AC_DC/Blow Up Your Video/06 Nick Of Time.mp3"
2024/04/12 14:44:46 DEBUG : Using config file from "/home/viana/.config/rclone/rclone.conf"
2024/04/12 14:44:46 DEBUG : found headers: 
2024/04/12 14:44:46 DEBUG : Chunks temporary upload directory: https://server/remote.php/dav/uploads/orestes/
2024/04/12 14:44:46 DEBUG : fs cache: adding new entry for parent of "nextcloud:Music/AC_DC/Blow Up Your Video/06 Nick Of Time.mp3", "nextcloud:Music/AC_DC/Blow Up Your Video"
2024/04/12 14:44:46 DEBUG : Creating backend with remote "."
2024/04/12 14:44:46 DEBUG : fs cache: renaming cache item "." to be canonical "/home/viana/Downloads"
2024/04/12 14:44:46 DEBUG : 06 Nick Of Time.mp3: Need to transfer - File not found at Destination
2024/04/12 14:44:49 DEBUG : 06 Nick Of Time.mp3: sha1 = 061a4ad4d97814247f56794282d2c01578eaa9fe (webdav root 'Music/AC_DC/Blow Up Your Video')
2024/04/12 14:44:49 DEBUG : 06 Nick Of Time.mp3.moqodox9.partial: sha1 = 26de37113b5145cea61506a3afdc1ceb585e405f (Local file system at /home/viana/Downloads)
2024/04/12 14:44:49 ERROR : 06 Nick Of Time.mp3.moqodox9.partial: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 14:44:49 INFO  : 06 Nick Of Time.mp3.moqodox9.partial: Removing failed copy
2024/04/12 14:44:49 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 14:44:49 DEBUG : 06 Nick Of Time.mp3: Need to transfer - File not found at Destination
2024/04/12 14:44:51 DEBUG : 06 Nick Of Time.mp3: sha1 = 061a4ad4d97814247f56794282d2c01578eaa9fe (webdav root 'Music/AC_DC/Blow Up Your Video')
2024/04/12 14:44:51 DEBUG : 06 Nick Of Time.mp3.wiqariv5.partial: sha1 = 26de37113b5145cea61506a3afdc1ceb585e405f (Local file system at /home/viana/Downloads)
2024/04/12 14:44:51 ERROR : 06 Nick Of Time.mp3.wiqariv5.partial: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 14:44:51 INFO  : 06 Nick Of Time.mp3.wiqariv5.partial: Removing failed copy
2024/04/12 14:44:51 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 14:44:51 DEBUG : 06 Nick Of Time.mp3: Need to transfer - File not found at Destination
2024/04/12 14:44:52 DEBUG : 06 Nick Of Time.mp3: sha1 = 061a4ad4d97814247f56794282d2c01578eaa9fe (webdav root 'Music/AC_DC/Blow Up Your Video')
2024/04/12 14:44:52 DEBUG : 06 Nick Of Time.mp3.wotabix5.partial: sha1 = 26de37113b5145cea61506a3afdc1ceb585e405f (Local file system at /home/viana/Downloads)
2024/04/12 14:44:52 ERROR : 06 Nick Of Time.mp3.wotabix5.partial: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 14:44:52 INFO  : 06 Nick Of Time.mp3.wotabix5.partial: Removing failed copy
2024/04/12 14:44:52 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 14:44:52 INFO  : 
Transferred:       24.748 MiB / 24.748 MiB, 100%, 3.915 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         6.6s

2024/04/12 14:44:52 DEBUG : 7 go routines active
2024/04/12 14:44:52 Failed to sync: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"

Initial research

The target folder is in an ext4fs filesystem on a Raspberry Pi 4B, running RaspberryPi OS (64-bit) Bookworm:

Linux nas-viana 6.6.20+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux

I have checked that the problem is the way rclone is reporting the remote hash, here it is the hashsum output for the remote:

viana@nas-viana:~/Downloads $ rclone hashsum sha1 nextcloud:"Music/AC_DC/Blow Up Your Video/06 Nick Of Time.mp3" -v
061a4ad4d97814247f56794282d2c01578eaa9fe  06 Nick Of Time.mp3

If I manually copy the remote file to local with scp -p:

viana@nas-viana:~/Downloads $ scp -p server:"/Music/AC_DC/Blow Up Your Video/06 Nick Of Time.mp3" .
06 Nick Of Time.mp3                                                             100% 8447KB   2.9MB/s   00:02

And I check the hash of the transferred file it is:

viana@nas-viana:~/Downloads $ rclone hashsum sha1 06\ Nick\ Of\ Time.mp3 
26de37113b5145cea61506a3afdc1ceb585e405f  06 Nick Of Time.mp3

If I run sha1sum in a console on the remote host:

server$ sha1sum 06\ Nick\ Of\ Time.mp3 
26de37113b5145cea61506a3afdc1ceb585e405f  06 Nick Of Time.mp3

So the sha1sum on the remote console, and the hashsum sha1 on the local scp-copied file are the same,
but the hashsum of the remote file is different.

There is something in the middle that is "changing" the file and thus making the hash calculation differ.

I am not sure if it is Nextcloud (or any other component in the middle) or rclone.

2

welcome to the forum,

the odds are there is something wrong with your setup.

for testing, can you simplify your setup as much as possible.
run rclone on the server itself and copy direct to nextcloud, without middle layers such as NGINX (PHP-FPM) and PHP 8.2

3

I am afraid I can't give up NGINX, it is the web server that serves Nextcloud (NGINX configuration — Nextcloud latest Administration Manual latest documentation). And PHP 8.2 is the supported version (System requirements — Nextcloud latest Administration Manual latest documentation). I provided those details to make sure I shared my current setup.

4

what other webdav clients have you tested?

5

Nextcloud desktop client for MacOS, I've been using it for years, and it serves that MP3 files without any problem.
The Music app in Nextcloud also plays the file without any problem. And I can play it also locally if I download it.

7

yes, i can download it, using for example the MacOS Finder client.

9

ok, and the checksums match?

10

Yes, they match.

11

ok, then not sure what the issue is.

is this the first time using rclone with the nextcloud server?

12

No, I have been using it to sync more than 100Gb and it fails on ALL the MP3 files. I have thousand of successfully synced files.

13

ok, did not mention that before.

can you rename 06 Nick Of Time.mp3 to 06 Nick Of Time.bin and then download with rclone.
and rename another file, something like file.txt to file.mp3 and then download with rclone.

to keep the debug log small, use --retries=1

14

Sorry, I meant to say that ALL the files that fail are MP3.

But I have succesfully downloaded other MP3 files.

15

I just add --ignore-checksum and it is syncing without problems, but I am concerned about disabling checksums, since they are key to keeping it properly in sync.

16

I have run it as you suggested:

rclone sync remote:/Music/AC_DC /target/Music/AC_DC --error ~/error-nextcloud.txt --progress --include="06 Nick Of Time.bin" --include="06 Nick Of Time.mp3" --retries=1
2024/04/12 18:10:16 ERROR : Blow Up Your Video/06 Nick Of Time.mp3.kenelaf6.partial: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 18:10:16 ERROR : Local file system at /target/Music/AC_DC: not deleting files as there were IO errors
2024/04/12 18:10:16 ERROR : Local file system at /target/Music/AC_DC: not deleting directories as there were IO errors
2024/04/12 18:10:16 ERROR : Attempt 1/1 failed with 1 errors and: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
Transferred:       16.498 MiB / 16.498 MiB, 100%, 3.582 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Transferred:            1 / 1, 100%
Elapsed time:         4.5s
2024/04/12 18:10:16 Failed to sync: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"

Here it is the checksum of the bin file, both local and remote:

rclone hashsum sha1 /target/Music/AC_DC/Blow\ Up\ Your\ Video/06\ Nick\ Of\ Time.bin 
26de37113b5145cea61506a3afdc1ceb585e405f  06 Nick Of Time.bin

rclone hashsum sha1 remote:Music/AC_DC/Blow\ Up\ Your\ Video/"06 Nick Of Time.bin"
rclone hashsum sha1 nextcloud:Music/AC_DC/Blow\ Up\ Your\ Video/"06 Nick Of Time.bin"
                                          06 Nick Of Time.bin

Surprisingly, there is no checksum on remote!!!

In addition, I have run sha1sum on the Nextcloud server, on both files:

sha1sum 06\ Nick\ Of\ Time.*
26de37113b5145cea61506a3afdc1ceb585e405f  06 Nick Of Time.bin
26de37113b5145cea61506a3afdc1ceb585e405f  06 Nick Of Time.mp3
17

Depending on the exact version of Owncloud or Nextcloud hashes may appear on all objects, or only on objects which had a hash uploaded with them.

18

The point is that I have downloaded the file using scp, put it in the appropiate folder, and then run rclone sync again, and it does not fail. So in this case, the local and the remote hashsum are equal. See my initial message for it.

19

can you pick one file that rclone has an issue with and then test another webdav client.
did you do that yet, sorry, maybe i missed your post about that?

20

I did it using MacOS Finder Webdav client. I mounted the remote webdav folder and copied 06 Nick Of Time.mp3 file to local without any problem.

Is there a way to hook into rclone and see how it transfers the file to a temporal location and then how it calculates the checksum?

21

could try --dump

https://rclone.org/commands/rclone_sha1sum/

and not sure it will matter, but might try --inplace

22

I have run a complete sync using --ignore-checksum, and everything synced properly, so I have all the remote files in my local folder:

rclone sync remote:/Music /target/Music --error ~/error-nextcloud.txt --progress --ignore-checksum
Transferred:        9.188 GiB / 9.188 GiB, 100%, 5.704 MiB/s, ETA 0s
Checks:              1797 / 1797, 100%
Transferred:          977 / 977, 100%
Elapsed time:     26m34.7s

Then I immediately have run another sync without --ignore-checksum, and everything run smoothly. Since everything was already in local folder, there was no real transfer, only checks:

rclone sync remote:/Music /target/Music --error ~/error-nextcloud.txt --progress
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:              2774 / 2774, 100%
Elapsed time:      1m22.6s

Then I removed the MP3 file and the bin file, and synced again (using checksums), and it failed :frowning: for the MP3 file but not for the bin file:

rclone sync remote:/Music /Music --error ~/error-nextcloud.txt --progress
2024/04/12 19:08:31 ERROR : AC_DC/Blow Up Your Video/06 Nick Of Time.mp3.hahofuk2.partial: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 19:09:44 ERROR : Local file system at /target/Music: not deleting files as there were IO errors
2024/04/12 19:09:44 ERROR : Local file system at /target/Music: not deleting directories as there were IO errors
2024/04/12 19:09:44 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
2024/04/12 19:09:55 ERROR : AC_DC/Blow Up Your Video/06 Nick Of Time.mp3.cahedin7.partial: corrupted on transfer: sha1 hash differ "061a4ad4d97814247f56794282d2c01578eaa9fe" vs "26de37113b5145cea61506a3afdc1ceb585e405f"
Transferred:       24.748 MiB / 24.748 MiB, 100%, 47.671 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:              5214 / 5214, 100%
Transferred:            1 / 1, 100%

There is something in the way those MP3 files are handled by rclone or Nextcloud that breaks the transfer. I will give a try to dump headers.