'rclone copy' fails to WSL2, dos copy works

asdffdsa · May 20, 2023, 9:42pm

What is the problem you are having with rclone?

rclone copy fails with corrupted on transfer, but dos copy works.
in addition, after successful dos copy, these rclone command output correct results.
rclone lsl on source and dest
rclone check --size-only
rclone check

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
- os/kernel: 10.0.22621.1702 Build 22621.1702.1702 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

source: windows
dest: WSL2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy e:\themod.py \\wsl$\ubuntu01\home\user01\cp\circuitpython\ports\atmel-samd\build-neopixel_trinkey_m0\frozen_mpy --retries=1 -vv

The rclone config contents with secrets removed.

no config used

A log from the command with the `-vv` flag

note: lightly edited for brevity
------------------------------------

rclone lsl e:\ --include=themod.py 
     6411 2023-05-20 16:27:30.000000000 themod.py

rclone copy e:\themod.py \\wsl$\ubuntu01\home\user01\cp\circuitpython\ports\atmel-samd\build-neopixel_trinkey_m0\frozen_mpy --retries=1 -vv  
DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\data\\rclone\\rclone" "copy" "e:\\themod.py" "\\\\wsl$\\ubuntu01\\home\\user01\\cp\\circuitpython\\ports\\atmel-samd\\build-neopixel_trinkey_m0\\frozen_mpy" "--retries=1" "-vv"]
DEBUG : Creating backend with remote "e:\\themod.py"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : fs cache: adding new entry for parent of "e:\\themod.py", "//?/e:/"
DEBUG : Creating backend with remote "\\\\wsl$\\ubuntu01\\home\\user01\\cp\\circuitpython\\ports\\atmel-samd\\build-neopixel_trinkey_m0\\frozen_mpy"
DEBUG : fs cache: renaming cache item "\\\\wsl$\\ubuntu01\\home\\user01\\cp\\circuitpython\\ports\\atmel-samd\\build-neopixel_trinkey_m0\\frozen_mpy" to be canonical "//?/UNC/wsl$/ubuntu01/home/user01/cp/circuitpython/ports/atmel-samd/build-neopixel_trinkey_m0/frozen_mpy"
DEBUG : themod.py: Need to transfer - File not found at Destination
ERROR : themod.py: corrupted on transfer: sizes differ 6411 vs 8192
INFO  : themod.py: Removing failed copy
ERROR : Attempt 1/1 failed with 1 errors and: corrupted on transfer: sizes differ 6411 vs 8192
INFO  : 
Transferred:   	    6.261 KiB / 6.261 KiB, 100%, 0 B/s, ETA -

verify on 

copy e:\themod.py \\wsl$\ubuntu01\home\user01\cp\circuitpython\ports\atmel-samd\build-neopixel_trinkey_m0\frozen_mpy\themod.py 
        1 file(s) copied.

echo ERRORLEVEL=%ERRORLEVEL%
ERRORLEVEL=0

rclone lsl \\wsl$\ubuntu01\home\user01\cp\circuitpython\ports\atmel-samd\build-neopixel_trinkey_m0\frozen_mpy --include=themod.py 
     6411 2023-05-20 16:27:30.000000000 themod.py

rclone check --size-only e:\ \\wsl$\ubuntu01\home\user01\cp\circuitpython\ports\atmel-samd\build-neopixel_trinkey_m0\frozen_mpy\ --include=themod.py -vv 
INFO  : Using md5 for hash comparisons
DEBUG : themod.py: OK
NOTICE: Local file system at //?/UNC/wsl$/ubuntu01/home/user01/cp/circuitpython/ports/atmel-samd/build-neopixel_trinkey_m0/frozen_mpy: 0 differences found
NOTICE: Local file system at //?/UNC/wsl$/ubuntu01/home/user01/cp/circuitpython/ports/atmel-samd/build-neopixel_trinkey_m0/frozen_mpy: 1 matching files

rclone check e:\ \\wsl$\ubuntu01\home\user01\cp\circuitpython\ports\atmel-samd\build-neopixel_trinkey_m0\frozen_mpy\ --include=themod.py -vv 
INFO  : Using md5 for hash comparisons
DEBUG : themod.py: md5 = 998b9361c5b252338ea32975f8d86304 OK
DEBUG : themod.py: OK
NOTICE: Local file system at //?/UNC/wsl$/ubuntu01/home/user01/cp/circuitpython/ports/atmel-samd/build-neopixel_trinkey_m0/frozen_mpy: 0 differences found
NOTICE: Local file system at //?/UNC/wsl$/ubuntu01/home/user01/cp/circuitpython/ports/atmel-samd/build-neopixel_trinkey_m0/frozen_mpy: 1 matching files

ncw · May 21, 2023, 10:45am

Looks like the file read as being 8192 bytes long instead of the 6411 bytes it should have been. Maybe just for an instant?

This is almost certainly a bug in wsl file system driver.

asdffdsa · May 21, 2023, 5:20pm

i want this to be a wsl bug but strange that only rclone produces incorrect results.
something changed the rclone copied file, but that something did not change the dos copied file.

this is a new example, using --ignore-size so rclone will actually copy the file.

the source file is 1 Byte in size.

dos copy : as expected, dest file size is 1 Byte.
rclone copy : dest file size is 4096 Bytes, the physical sector size.

# list the source file
rclone lsl . --include=file.txt 
        1 2023-05-21 11:44:39.562267600 file.txt

# dos copy
copy .\file.txt \\wsl$\ubuntu01\home\user01\test\dos.txt 
        1 file(s) copied.

# rclone copyto
rclone copyto .\file.txt \\wsl$\ubuntu01\home\user01\test\rclone.txt --ignore-size -vv  
DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\data\\rclone\\rclone.exe" "copyto" ".\\file.txt" "\\\\wsl$\\ubuntu01\\home\\user01\\test\\rclone.txt" "--ignore-size" "-vv"]
DEBUG : Creating backend with remote ".\\file.txt"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : fs cache: adding new entry for parent of ".\\file.txt", "//?/D:/data/l/adafruit/CircuitPython/mpycross"
DEBUG : Creating backend with remote "//wsl$/ubuntu01/home/user01/test/"
DEBUG : fs cache: renaming cache item "//wsl$/ubuntu01/home/user01/test/" to be canonical "//?/UNC/wsl$/ubuntu01/home/user01/test"
DEBUG : file.txt: Need to transfer - File not found at Destination
DEBUG : file.txt: md5 = c4ca4238a0b923820dcc509a6f75849b OK
INFO  : file.txt: Copied (new) to: rclone.txt
INFO  : 
Transferred:   	          1 B / 1 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

# result - the rclone copied file has has size changed
rclone lsl \\wsl$\ubuntu01\home\user01\test 
        1 2023-05-21 11:44:39.562267600 dos.txt
     4096 2023-05-21 11:44:39.562267600 rclone.txt

ncw · May 22, 2023, 12:20pm

Try disabling preallocations - that could be the cause of the problem.

  --local-no-preallocate   Disable preallocation of disk space for transferred files

If that works then the WSL driver is implementing preallocations wrong I think.

asdffdsa · May 22, 2023, 1:36pm

good idea to use --local-no-preallocate, the issue does not occur.

i think this is the issue?
https://github.com/Microsoft/WSL/issues/2626

ncw · May 22, 2023, 7:17pm

Great, glad that fixed it.

That issue describes the problem perfectly. I added a note to it that it breaks rclone too and the workaround.

asdffdsa · May 22, 2023, 7:23pm

oh, good, you beat me to the punch, better you than me.

but at least, i added two very important emoji ;wink

system · May 25, 2023, 7:23pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.