Corrupted files with windows net drive

Help and Support
1

What is the problem you are having with rclone?

Lots of corrupted files when copying to a net drive on Windows.
On average I see an error for every 40~50 files.

My setup:

  • C: is a local disk on a window machine
  • Z: is a windows net drive, mapped from a samba share on an Ubuntu machine
  • The source directories contains lots of small image files, typically between 256KB and 512KB.

To rule out other factors, I have also tried the following and found no errors:

  • Copy from from C: to Z: using Windows Explorer, then check hashsum
  • Use rclone to copy from C:\files to C:\files2

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: Microsoft Windows 10 Home 2009 (64 bit)
- os/kernel: 10.0.19042.1526 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Windows net drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy C:\files Z:\files

The rclone config contents with secrets removed.

N/A

A log from the command with the -vv flag

2022/02/20 20:12:11 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\Users\\user\\bin\\rclone.exe" "--progress" "-vv" "--log-file" "C:\\Users\\user\\Desktop\\rclone.log" "copy" "C:\\Users\\user\\files" "Z:\\user\\files"]
2022/02/20 20:12:11 DEBUG : Creating backend with remote "C:\\Users\\user\\files"
2022/02/20 20:12:11 DEBUG : Using config file from "C:\\Users\\user\\AppData\\Roaming\\rclone\\rclone.conf"
2022/02/20 20:12:11 DEBUG : fs cache: renaming cache item "C:\\Users\\user\\files" to be canonical "//?/C:/Users/user/files"
2022/02/20 20:12:11 DEBUG : Creating backend with remote "Z:\\user\\files"
2022/02/20 20:12:11 DEBUG : fs cache: renaming cache item "Z:\\user\\files" to be canonical "//?/Z:/user/files"
...
2022/02/20 20:12:14 DEBUG : test-rclone/0006296.jpg: md5 = a768e96e5a720081c2dd496304de8bd7 OK
2022/02/20 20:12:14 INFO  : test-rclone/0006296.jpg: Copied (new)
2022/02/20 20:12:14 ERROR : test-rclone/0006297.jpg: corrupted on transfer: sizes differ 328388 vs 328374
2022/02/20 20:12:14 DEBUG : test-rclone/0006298.jpg: md5 = b0ebdaced2b552b8f2824f35109580fc OK
2022/02/20 20:12:14 INFO  : test-rclone/0006297.jpg: Removing failed copy
2022/02/20 20:12:14 INFO  : test-rclone/0006298.jpg: Copied (new)
2022/02/20 20:12:14 DEBUG : test-rclone/0006299.jpg: md5 = 14551cf849ac842356c750850c46a511 OK
2022/02/20 20:12:14 INFO  : test-rclone/0006299.jpg: Copied (new)
2022/02/20 20:12:14 DEBUG : test-rclone/0006300.jpg: md5 = ebec29cbfc8fdbf4a76f615b692a0b6e OK
2022/02/20 20:12:14 INFO  : test-rclone/0006300.jpg: Copied (new)
2022/02/20 20:12:14 DEBUG : test-rclone/0006302.jpg: md5 = 523414785e70e52c45cf3b53f0c6e307 OK
2022/02/20 20:12:14 INFO  : test-rclone/0006302.jpg: Copied (new)
2022/02/20 20:12:14 DEBUG : test-rclone/0006301.jpg: md5 = 1f4a92ff3aaa3f4cdf7d248123899fd7 OK
2022/02/20 20:12:14 INFO  : test-rclone/0006301.jpg: Copied (new)
2022/02/20 20:12:14 DEBUG : test-rclone/0006303.jpg: md5 = 25a24b32a3e07339139a325e9c411bf6 OK
2022/02/20 20:12:14 INFO  : test-rclone/0006303.jpg: Copied (new)
2022/02/20 20:12:14 DEBUG : test-rclone/0006304.jpg: md5 = 65ce75d5f243d7efbc149e22dc5b1c8a OK
2022/02/20 20:12:14 INFO  : test-rclone/0006304.jpg: Copied (new)
2022/02/20 20:12:14 DEBUG : test-rclone/0006306.jpg: md5 = 741b6a6219d27569b9f1b16af43fc659 OK
2022/02/20 20:12:14 INFO  : test-rclone/0006306.jpg: Copied (new)
2022/02/20 20:12:14 DEBUG : test-rclone/0006305.jpg: md5 = 856d265a8621b2c903577503b1e106f0 OK
2022/02/20 20:12:14 INFO  : test-rclone/0006305.jpg: Copied (new)
2022/02/20 20:12:14 ERROR : test-rclone/0006307.jpg: corrupted on transfer: sizes differ 328309 vs 328302
2022/02/20 20:12:14 INFO  : test-rclone/0006307.jpg: Removing failed copy
...
2

hi,

while rclone copy is running
is there a chance that any of the source files are in-use, might change?

3

I don't think so, since I tried (and mixed) all different copying methods several times.

4

so that is the unedited debug log?

5

Right. Except that most lines about successful copies were removed.

6

are the same files failing each time?

7

i am confused, here is you state the rclone copy from c: to z:, no errors.
but your rclone command is rclone copy C:\files Z:\files which has errors???

8

No, it will be OK after a few attempts (usually it'd be ok on the second try). I suppose this (a file is corrupted) is a random event.

9

Sorry, my mistake. I updated the post.
I tested copying the files locally on the same disk, everything worked fine.

10

rclone can stress out a system, as compared to windows explorer.

test using --transfers=1 --checkers=1, which will more closely emulate widows explorer.

11

Hmm, funny. I tried again and now I cannot reproduce the issue.
I just copied 5k files without any errors.
Perhaps it's some transient errors on the server side.

12

using your command as is or with the changes i suggested?

13

Using my original command.
(I wanted to compare with --transfer=1)

14

And thanks for your help!

15

This error looks like the net drive is running behind somehow. What rclone does is copies the file, then queries the file to see how long it is. If it is the wrong length then rclone declares it corrupted.

So what it looks like here is that the net drive showed an old size somehow.

I wouldn't have thought it should ever do that though!

16

I did check the hashsum on the server side, the file is indeed smaller.
I could also clearly see the image was missing pixels (i.e. that lower 1/4 part became gray) when the difference is large.

I had suspected that it has something to do with the issue betwen go runtime and samba. But now I'm hestitating because I'm using Windows and I'm not using rclone mount

17

What's the OS running samba? Older kernels distros are bad with Samba bugs.

18

rclone is not interacting with samba.

rclone -> windows os -> ubuntu -> samba

19

It's Ubuntu 20.04

20

Running the HWE kernel?

uname -a
Linux gemini 5.13.0-30-generic #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux