`rclone copy t:\ --include=filename` does NOT work, but `rclone copy t:\filename` works

asdffdsa · October 8, 2023, 9:36pm

What is the problem you are having with rclone?

rclone copy displays different behavior depending on how the source is specified.

rclone will copy the source file
rclone copy t:\filename
rclone will NOT copy the source file
rclone copy t:\ --include=filename

note: from the log below, size changed from 1147752 to 1213336
the correct filesize is 1147752

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.64.0
- os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
- os/kernel: 10.0.22621.2361 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

local to local - the source and dest are both local drive letters on the same machine.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

please see the logs below

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

no config, the source and dest are both local.

A log from the command that you were trying to run with the `-vv` flag

# rclone will copy the source file
rclone copy t:\filename d:\test  -vv --retries=1 
DEBUG : rclone: Version "v1.64.0" starting with parameters ["c:\\data\\rclone\\rclone.exe" "copy" "t:\\filename" "d:\\test" "-vv" "--retries=1"]
DEBUG : Creating backend with remote "t:\\filename"
DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
DEBUG : fs cache: adding new entry for parent of "t:\\filename", "//?/t:/"
DEBUG : Creating backend with remote "d:\\test"
DEBUG : fs cache: renaming cache item "d:\\test" to be canonical "//?/d:/test"
DEBUG : filename: Need to transfer - File not found at Destination
DEBUG : filename: md5 = 1bacd6890eed1db4f4debffb5240464a OK
DEBUG : filename.jenopeq7.partial: renamed to: filename
INFO  : filename: Copied (new)

# rclone will **NOT** copy the source file
rclone.exe copy t:\ d:\test --include=filename -vv --retries=1 
DEBUG : rclone: Version "v1.64.0" starting with parameters ["c:\\data\\rclone\\rclone.exe" "copy" "t:\\" "d:\\test" "--include=filename" "-vv" "--retries=1"]
DEBUG : Creating backend with remote "t:\\"
DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
DEBUG : fs cache: renaming cache item "t:\\" to be canonical "//?/t:/"
DEBUG : Creating backend with remote "d:\\test"
DEBUG : fs cache: renaming cache item "d:\\test" to be canonical "//?/d:/test"
DEBUG : filename: Need to transfer - File not found at Destination
NOTICE: filename.yuyawer8.partial: Removing partially written file on error: can't copy - source file is being updated (size changed from 1147752 to 1213336)
ERROR : filename: Failed to copy: can't copy - source file is being updated (size changed from 1147752 to 1213336)
DEBUG : Local file system at //?/d:/test: Waiting for checks to finish
DEBUG : Local file system at //?/d:/test: Waiting for transfers to finish
ERROR : Attempt 1/1 failed with 1 errors and: can't copy - source file is being updated (size changed from 1147752 to 1213336)

ncw · October 9, 2023, 7:46pm

Is the file changing in size?

If not, is it just this one file or is it any file?

What is special about it? Are there files or directories with similar names?

asdffdsa · October 9, 2023, 9:02pm

never. in fact, the source is write-protected, i cannot edit it, cannot delete it

well, the source dir is a vss snapshot ;wink

happens to many dozens, hundreds of files. easily reproduced.
i tested this over a couple of days, different vss snapshots, different dirs, different rclone flags, etc..
not just me, happens to others fellow rcloners as posted about in the forum.
fastcopy, fc, fchash, 7zip, certutil, double commander, windows explorer, dos cmd, powershell, all, always copies the files and return the correct md5 hash.

this does NOT affect md5sum in the same way, i tested three ways and get correct md5 hash.

rclone md5sum t:\filename
rclone md5sum t:\ 
rclone md5sum t:\ --include=filename

so far, only issue, rclone copy, when the source is a dir
--include=filename has nothing to do with it. i did that to keep the log short.
so both of these will generate the same exact error(s).

rclone copy t:\
rclone copy t:\ --include=filename

i am no expert at the source code but i did notice that rclone copy behave differently, if the source is a dir or filename, maybe some plumbing with sync.CopyDir

https://github.com/rclone/rclone/blob/f56ea2bee2e7552e273ff679c6f0eb429a0969d3/cmd/copy/copy.go#L94

if srcFileName == "" {
		return sync.CopyDir(context.Background(), fdst, fsrc, createEmptySrcDirs)
	}
	return operations.CopyFile(context.Background(), fdst, fsrc, srcFileName, srcFileName)

or maybe this

ncw · October 10, 2023, 9:25am

Hmm ok, so this is something to do with VSS.

If you do this do you get the correct file size or not?

rclone lsl t:\ --include=/filename

So when rclone read the file via the directory listing it was the correct size 1147752 - I'd expect the lsl above to give the correct size 1147752 in that case.

It is when rclone read the size through an open file descriptor that the size is wrong, so this comment

Is probably correct.

os.Lstat good, os.File.Stat bad.

That's definitely a bug, but in what? Most likely the Go runtime, though it could be a bug in Windows VSS also. The go runtime is calling the GetFileInformationByHandle system call to read the info about the file from the file handle

Can you make a reproduction for me? So what I need is a sequence of commands to run to make the VSS and replicate the problem. I've never made a VSS before so I need help with that bit! I have a Windows 10 VM I can try things on.

asdffdsa · October 10, 2023, 1:11pm

sure, i can make a reproduction for you.

in the mean time, to understand the basics, might do a quick scan of my wiki.
https://github.com/rclone/rclone/wiki/How-to-enable-VSS-for-rclone

not sure the issue is vss, as all the other programs work as expected. that is why i did not want to mention it at first.

clearly, rclone can compute the hash and obtain file size.
i guess you have some logic to using two different os calls. os.Lstat and os.File.Stat?

another, maybe simpler option, to add additional debug output, see which functions are returning what values.
and i can run the tests.

so let me know what you need from me.

ncw · October 10, 2023, 6:41pm

The os.Lstat is the file that rclone listed from the directory. The os.File.Stat is for the file that rclone has open right now so it should definitely be correct.

Download this https://pub.rclone.org/windows-shadow-bug.zip and unzip it. It has a Go program which will do a simple test. You'll find a windows binary and a linux binary. Give it the path to the troublesome file and run it. It should print something like this

> windows-shadow-bug.exe main.go
Testing "main.go"
Size read with os.Stat      : 814
Size read with os.File.Stat : 814

If that gives a different answer then that's enough for me to make report a Go bug, though I should try to replicate it first! If I can't replicate it then you could join me on the Go bug report and answer questions.

system · November 9, 2023, 6:42pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.