Spurious mkdir errors on mounted fileshare (+ size check errors)

Suspected Bug
#1

What is the problem you are having with rclone?

When syncing a bigger amount of files to a samba fileshare on windows, there are various errors. While I could ignore the size check errors (the files come out just fine and windows reports the exact same size), the mkdir errors are a problem. They are spurious in a sense that a) they only happen when transmitting enough files at once and b) each time, they are on different files.

These errors do not happen when copying the file via windows explorer. running xcopy right after into a folder on the same drive with the same rights also works just fine. If not for that, I would suspect a smb problem, but thats the point - only rclone seems to have it.

The errors also only affect the mounted network drive - copying to a local folder works fine.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.1766 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.9
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

no cloud storage, local filesystem. The drive that I'm copying to is a network drive (samba on debian), though.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -vv "C:\\test" Z://test5

The rclone config contents with secrets removed.

<empty>

A log from the command with the -vv flag

Note: "Das System kann den angebenen Pfad nicht finden" translates to "The system cannot find the path specified". "Zugriff verweigert" translates to "Access denied".

#2

hello and welcome to the forum,

if you copy the same files not using rclone, instead using another copy tool, what is the result.

#3

xcopy works like a charm

#4

Same terminal, btw. So no problem with access rights.

#5

An additional note: No problem when copying to a local drive!

Thanks for the welcome by the way.

#6

as i would never used rclone for local, not sure what the issue is.
tho there have been a bunch of topics about issues with rclone and samba.

as a test,
--- try --transfers=1 --checkers=1
--- try \\server\share instead of Z://test5
--- try single slashes instead of double slashes - C:\test instead of C:\\test

#7

ok, interesting results incoming:

  1. --transfers=1 --checkers=1 results in only one file in every sub-folder of C://test getting copied.
  2. results in //?/UNC///test8 and consequently for some reason a syntax error
  3. no difference
#8

need to see a debug log with all the changes i suggested.

  1. very strange result, as those flags means that rclone will copy a single file at a time and check a single file at a time. but all files should be copied.
  2. use simple \\server\share format
#9

Output of --transfers=1 --checkers=1:

The other output I cannot give right now, as I just remembered: The network drive has a different username and password than my local machine anyways - It cant work like that right now. I didnt expect an answer so quickly and have to go to an appointment- I will answer later on this evening or tomorrow.

#10

Ouput of

rclone sync --transfers=1 --checkers=1 -vv C:\test \\192.168.20.102\data\test

is (where the error message indicates "invalid syntax for File name, directory name or drive name")

2022/06/21 22:29:47 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "sync" "--transfers=1" "--checkers=1" "-vv" "C:\\test" "\\\\192.168.20.102\\data\\test"]
2022/06/21 22:29:47 DEBUG : Creating backend with remote "C:\\test"
2022/06/21 22:29:47 DEBUG : Using config file from "C:\\Users\\<username>\\AppData\\Roaming\\rclone\\rclone.conf"
2022/06/21 22:29:47 DEBUG : fs cache: renaming cache item "C:\\test" to be canonical "//?/C:/test"
2022/06/21 22:29:47 DEBUG : Creating backend with remote "\\\\192.168.20.102\\data\\test"
2022/06/21 22:29:49 DEBUG : fs cache: renaming cache item "\\\\192.168.20.102\\data\\test" to be canonical "//?/UNC/192.168.20.102/data/test"
2022/06/21 22:29:49 DEBUG : Local file system at //?/UNC/192.168.20.102/data/test: Waiting for checks to finish
2022/06/21 22:29:49 DEBUG : Local file system at //?/UNC/192.168.20.102/data/test: Waiting for transfers to finish
2022/06/21 22:29:49 ERROR : 2017 06 07/P1150642.JPG: Failed to copy: mkdir \\?\UNC\: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
2022/06/21 22:29:49 ERROR : 2017 06 07/P1150643.JPG: Failed to copy: mkdir \\?\UNC\: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
2022/06/21 22:29:49 ERROR : 2017 06 07/P1150644.JPG: Failed to copy: mkdir \\?\UNC\: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
2022/06/21 22:29:49 ERROR : 2017 06 07/P1150645.JPG: Failed to copy: mkdir \\?\UNC\: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
<many instances of the same error ommitted, in between several instances of>
2022/06/21 22:29:51 ERROR : Local file system at //?/UNC/192.168.20.102/data/test: not deleting files as there were IO errors
2022/06/21 22:29:51 ERROR : Local file system at //?/UNC/192.168.20.102/data/test: not deleting directories as there were IO errors
2022/06/21 22:29:51 ERROR : Attempt 1/3 failed with 391 errors and: mkdir \\?\UNC\: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
2022/06/21 22:29:51 DEBUG : Local file system at //?/UNC/192.168.20.102/data/test: Waiting for checks to finish
2022/06/21 22:29:51 DEBUG : Local file system at //?/UNC/192.168.20.102/data/test: Waiting for transfers to finish
<after that: rinse and repeat, in the end then>
2022/06/21 22:35:46 ERROR : Local file system at //?/UNC/192.168.20.102/data/test: not deleting files as there were IO errors
2022/06/21 22:35:46 ERROR : Local file system at //?/UNC/192.168.20.102/data/test: not deleting directories as there were IO errors
2022/06/21 22:35:46 ERROR : Attempt 3/3 failed with 391 errors and: mkdir \\?\UNC\: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
2022/06/21 22:35:46 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:               391 (retrying may help)
Elapsed time:         7.9s

2022/06/21 22:35:46 DEBUG : 3 go routines active
2022/06/21 22:35:46 Failed to sync with 391 errors: last error was: mkdir \\?\UNC\: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
#11

from the debug log, this is not correct
Failed to copy: mkdir \\?\UNC\:

might try to create a remote such as

[nounc]
type = local
nounc = true

and a command such as
rclone sync C:\test nounc:\\192.168.20.102\data\test -vv

and/or might try using hostname, not ip address

#12

That yields

For some reason, using the hostname made it possible to have more than one file per directory - but there are still 55 errors after 3 retries:

#13

really, very strange results.
can you create a net share on a windows machine and run a test with that?

try --retries=1, that will reduce the retries and the log file.

#14

Agree, though I haven't read the thread in detail.

Perhaps a kernel/driver issue that can be worked around like this:

#15

A share on a windows vm on the same server works just fine. This seems to be a samba-specific error then.
I tried to disable opportunistic locking in the registry, did not help.
Running one of the failing commands (cannot find path) manually (with the path quoted of course) yields the proper result (file already exists).

#16

That changes the result maybe slightly:

#17

and just to understand:

you get a similar result but with different folders failing if you perform the command again? (to ...\test7 with GODEBUG=asyncpreemptoff=1)

#18

Sometimes it works, sometimes it doesnt.

#19

It only syncs the remaining files - and sometimes, no error occures and all files get synced. Most times though, there still is an error for at least some of the files not synced the first time. Even with the standard 3 retries.

#20

OK, I get it.

I agree it looks like a smb driver problem (race condition) somehow provoked by something rclone does different than the other tools tried. It may well be very difficult/impossible to pinpoint and I don't have that much time, but here are a few ideas to exclude issues with your source:

Can you reproduce the issue if you remove all spaces and special characters from your folder/file names?

Can you reproduce if you copy small set local testdata generated with rclone test makefiles .\testfolder with appropriate parameters?

Can you reproduce if data are synced from a (somewhat slow) cloud remote e.g. OneDrive?