Rclone crashes during upload

What is the problem you are having with rclone?

Rclone keeps crashing during uploading same folder/files to google drive.
Interestingly, these files (some learning videos from TTC) are the only ones crashing rclone.
was able to upload them through the google drive web interface, so it must be rclone related?

What is your rclone version (output from rclone version)

rclone v1.53.1

  • os/arch: windows/amd64
  • go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

copy file to mounted drive

The rclone config contents with secrets removed.

'''
[remote]
type = drive
scope = drive
token = {"access_token":"masked","token_type":"Bearer","refresh_token":"masked","expiry":"2020-10-04T07:44:19.0527006+03:00"}
'''

A log from the command with the -vv flag

Can you share your rclone mount command since it's a clip of the debug log and what version of WinFSP?

rclone mount remote: x:

winFSP 1.7.20172

That seems to be winfsp related. Can you try v1.6 as I think that one is a beta isn't it?

Can you reliably reproduce this problem?

that is not beta.
i still use v1.6 and works perfectly.

My bad, I mis-read the downloads section on the WinFSP web site.

This is the relevant part of the traceback

unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x4c360e]
 
goroutine 66 [running, locked to thread]:
runtime.throw(0x1bd0b9e, 0x5)
        runtime/panic.go:1116 +0x79 fp=0xc0003ffd20 sp=0xc0003ffcf0 pc=0x4fc259
runtime.sigpanic()
        runtime/signal_windows.go:249 +0x24f fp=0xc0003ffd50 sp=0xc0003ffd20 pc=0x510f4f
indexbytebody(0x500000000000501, 0xaff, 0x52cf00, 0xc000085380, 0xaff, 0x0, 0x500000000000501, 0x500000000000501, 0xaff, 0xc0003ffdf8, ...)
        internal/bytealg/indexbyte_amd64.s:124 +0xce fp=0xc0003ffd58 sp=0xc0003ffd50 pc=0x4c360e
runtime.findnull(0x500000000000501, 0xc0003ffdf8)
        runtime/string.go:454 +0x87 fp=0xc0003ffdb0 sp=0xc0003ffd58 pc=0x516627
runtime.gostring(0x500000000000501, 0xc000185040, 0xeb627ffa10)
        runtime/string.go:320 +0x36 fp=0xc0003ffe08 sp=0xc0003ffdb0 pc=0x52e616
github.com/billziss-gh/cgofuse/fuse._Cfunc_GoString(...)
        _cgo_gotypes.go:322
github.com/billziss-gh/cgofuse/fuse.c_GoString(...)
        github.com/billziss-gh/cgofuse@v1.4.0/fuse/host_cgo.go:686
github.com/billziss-gh/cgofuse/fuse.hostRelease(0x500000000000501, 0xeb627ffb40, 0x21000000000)
        github.com/billziss-gh/cgofuse@v1.4.0/fuse/host.go:284 +0x94 fp=0xc0003ffe68 sp=0xc0003ffe08 pc=0x15d0c54
github.com/billziss-gh/cgofuse/fuse.go_hostRelease(...)
        github.com/billziss-gh/cgofuse@v1.4.0/fuse/host_cgo.go:901
github.com/billziss-gh/cgofuse/fuse._cgoexpwrap_e4f666a8d790_go_hostRelease(0x500000000000501, 0xeb627ffb40, 0x0)
        _cgo_gotypes.go:802 +0x3c fp=0xc0003ffe90 sp=0xc0003ffe68 pc=0x15d641c
runtime.call32(0x0, 0xeb627ff930, 0xeb627ffaa0, 0x18)
        runtime/asm_amd64.s:540 +0x45 fp=0xc0003ffec0 sp=0xc0003ffe90 pc=0x5311e5
runtime.cgocallbackg1(0x0)
        runtime/cgocall.go:332 +0x1b8 fp=0xc0003fff58 sp=0xc0003ffec0 pc=0x4c51b8
runtime.cgocallbackg(0x0)
        runtime/cgocall.go:207 +0xe5 fp=0xc0003fffc0 sp=0xc0003fff58 pc=0x4c4f45
runtime.cgocallback_gofunc(0x0, 0x20, 0x2030001, 0x2030001)
        runtime/asm_amd64.s:794 +0xb2 fp=0xc0003fffe0 sp=0xc0003fffc0 pc=0x532992
runtime.goexit()
        runtime/asm_amd64.s:1374 +0x1 fp=0xc0003fffe8 sp=0xc0003fffe0 pc=0x532c61

And here is part of the call stack

Ultimately it looks like the kernel passed an invalid address to rclone for a path
0x500000000000501 - I'm not familiar with Windows user space addressing but that address looks very high to be valid...

@BogdanS can you replicate this easily?

Can you try disabling all anti-virus checkers and see if you can still replicate it?

Yes, I can replicate it any time. It ALWAYS crashes when I try to copy those files
There was a nfo file that was quite small in one of the folder with which I tested. I tried to rename the file, trim the file, put it in a different filepath, move it to another disk, no-matter what I tried when I copied it it crashes rclone. All files that were in the same initial path (avi,pdf etc) act the same. It's like they're "marked" somehow.

If it matters, I'm using "Total Commander" to copy these files.
I don't see the mounted drive under Windows Explorer but I do see it in Total Commander.
I did copy 10TB of files to Google Drive until now without issues however files from this folder cannot be copied at all.

I turned off the anti-virus to test but it acts the same.

I made a video of the whole process. The "Warranty.pdf" file that I copy at first was not originally in that folder. As you can see it copies fine. Then I try to copy antoher PDF that crashes rclone. The result would be the same if I would try and copy one of the AVI files.

recording is on streamable dot com slash eiodd8

hi,
can you rclone copy the files?

can you copy the files locally from one folder to another folder

  • using tc?
  • using rclone?

the e: drive, is that usb?
i get that sometimes, for the local file, if you go into properties, you might find a checkbox, with text 'unblock'

image

  • rclone copy sourcefile remote: works fine !
  • I can copy the sourcefiles wherever with TC (network,local-disk,USB)
  • e: drive is USB
  • sourcefile can be on network or on localdisk , same crash happens
  • file is not "blocked", I checked.

So the only scenario where I can copy the file is when I use rclone copy.
When the remote is mounted using rclone files cannot be copied.
I tried copying with Windows Explorer (copy/paste) and I get "Invalid MS-DOS function"

2020-10-05 20_59_08-Window

can you try the version i use, never had a problem
https://github.com/billziss-gh/winfsp/releases/tag/v1.6

Just tried with 1.6, still crashes.
I will accept it and move on, not sure what else I can do...

oh no, we cannot move on, we are on the hunt for a bug.

would you be willing to email me the file as an attachment.
i could private message you with an email address?

Sure, but first I'll try to replicate the setup on my other machine and see if it acts the same way. I'll get back to you tomorrow

It must be some bug related to WinFSP combined with my Windows system. I also tried ExpanDrive and that too crashes with these files. So most likely not rclone related but WinFSP.

Interesting.

WinFSP problems are usually caused by incompatibility with something, usually anti virus, but it could be something else that hooks into the file system - eg a file system monitor.

How about if you try from safe mode?

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

This turned out to be a bug in WinFSP which should be fixed in the beta - please can you try

And see if you can get it to go wrong again?

Thanks

Hi, I just found this issue opened here and I actually have experienced this problem for a period of time.

So the situation is that when I tried to move a set of large-sized files (mostly over 5g in total) to the mounted drive, the system notified me that the error with invalid MS DOS operation, just like what the guy who opened this issue said. Consequently there will be a set of files with same name but 0kb size created on the mounted driver. And latterly when I tried to move those files again, and no matter I manually deleted those empty files or click replace when moving, the system still said MS DOS invalid.

I tried to unmount the drive, and I found that those empty files buffered in the cache folder would not be removed. (If I manually remove them and mount again, the same issue still happens)

I tried update the WinFSP to the 2020.1beta version, and the problem still happens.

sorry, the WinFSP version I tried was actually 2021.1beta