Local copy operation fails with "handle is invalid"

jared · March 17, 2021, 5:03pm

What is the problem you are having with rclone?

a local rclone copy operation results in "fatal error. the handle is invalid"

What is your rclone version (output from `rclone version`)

rclone version 1.54.1

Which OS you are using and how many bits

Windows 10 pro, 64 bit

Which cloud storage system are you using?

local

The command you were trying to run

rclone copy

rclone copy --verbose --log-file=localerror.txt -vv --transfers=15 --exclude /Cornelius/ "\\192.168.1.20\ATB Server\Server" "\\192.168.1.20\Archive 2\Server 3-16-21"

The rclone config contents with secrets removed.

local remotes not listed in config. is it still needed?

A log from the command with the `-vv` flag

2021/03/17 11:40:07 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "copy" "--verbose" "--log-file=localerror.txt" "-vv" "--transfers=15" "--exclude" "/Cornelius/" "\\\\192.168.1.20\\TBA Server\\Server" "\\\\192.168.1.20\\Archive 2\\Server 3-16-21"]
2021/03/17 11:40:07 DEBUG : Creating backend with remote "\\\\192.168.1.20\\TBA Server\\Server"
2021/03/17 11:40:07 DEBUG : Using config file from "C:\\Users\\Jared\\.config\\rclone\\rclone.conf"
2021/03/17 11:40:07 DEBUG : fs cache: renaming cache item "\\\\192.168.1.20\\TBA Server\\Server" to be canonical "//?/UNC/192.168.1.20/TBA Server/Server"
2021/03/17 11:40:07 DEBUG : Creating backend with remote "\\\\192.168.1.20\\Archive 2\\Server 3-16-21"
2021/03/17 11:40:07 DEBUG : fs cache: renaming cache item "\\\\192.168.1.20\\Archive 2\\Server 3-16-21" to be canonical "//?/UNC/192.168.1.20/Archive 2/Server 3-16-21"
2021/03/17 11:40:07 DEBUG : Cornelius: Excluded
2021/03/17 11:40:24 NOTICE: Network/PBX Server Backups/PBX Backup 6-26 Full.tgz: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Network\PBX Server Backups\PBX Backup 6-26 Full.tgz: The handle is invalid.
2021/03/17 11:40:24 DEBUG : Gina/Customer Tax Exempt Forms/2020 Tax Forms/WAPA (Huron, PA)/PAtaxexemptform.pdf: Size and modification time the same (differ by 0s, within tolerance 100ns)
2021/03/17 11:40:24 DEBUG : Gina/Customer Tax Exempt Forms/2020 Tax Forms/WAPA (Huron, PA)/PAtaxexemptform.pdf: Unchanged skipping
2021/03/17 11:40:24 NOTICE: Network/PBX Server Backups/PBX Backup 6-29 Full.tgz: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Network\PBX Server Backups\PBX Backup 6-29 Full.tgz: The handle is invalid.
2021/03/17 11:40:24 NOTICE: Network/PBX Server Backups/PBX Backup 11-10 Full.tgz: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Network\PBX Server Backups\PBX Backup 11-10 Full.tgz: The handle is invalid.
2021/03/17 11:40:24 NOTICE: Network/PBX Server Backups/PBX Backup 8-17 Full.tgz: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Network\PBX Server Backups\PBX Backup 8-17 Full.tgz: The handle is invalid.
2021/03/17 11:40:24 NOTICE: Network/PBX Server Backups/PBX Backup 10-18 Full.tgz: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Network\PBX Server Backups\PBX Backup 10-18 Full.tgz: The handle is invalid.
2021/03/17 11:40:24 NOTICE: TBAccount/ToolBox.mde: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\TBAccount\ToolBox.mde: The handle is invalid.
2021/03/17 11:40:24 NOTICE: Network/PBX Server Backups/PBX Backup 2-9 Full.tgz: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Network\PBX Server Backups\PBX Backup 2-9 Full.tgz: The handle is invalid.
2021/03/17 11:40:24 NOTICE: Network/PBX Server Backups/PBX Backup 7-16 Full.tgz: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Network\PBX Server Backups\PBX Backup 7-16 Full.tgz: The handle is invalid.
2021/03/17 11:40:24 DEBUG : Gina/Customer Tax Exempt Forms/2020 Tax Forms/WAPA (Huron, PA)/Tax exempt letter.pdf: Size and modification time the same (differ by 0s, within tolerance 100ns)
2021/03/17 11:40:24 DEBUG : Gina/Customer Tax Exempt Forms/2020 Tax Forms/WAPA (Huron, PA)/Tax exempt letter.pdf: Unchanged skipping
2021/03/17 11:40:24 NOTICE: Marketing/Accounts and Logins/SprucelaneHB2018.mp4: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Marketing\Accounts and Logins\SprucelaneHB2018.mp4: The handle is invalid.
2021/03/17 11:40:24 NOTICE: VisualPrime Backup/TBAOXES2011: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\VisualPrime Backup\TBAOXES2011: The handle is invalid.
2021/03/17 11:40:24 NOTICE: TBAccount/Cortech Update 08-07-2013/VisualPrime 5.5.621.12.exe: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\TBAccount\Cortech Update 08-07-2013\VisualPrime 5.5.621.12.exe: The handle is invalid.
2021/03/17 11:40:24 NOTICE: VisualPrime Backup/ACTBA2017: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\VisualPrime Backup\ACTBA2017: The handle is invalid.
2021/03/17 11:40:24 NOTICE: Network/PBX Server Backups/PBX Backup 10-8 Full.tgz: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\Network\PBX Server Backups\PBX Backup 10-8 Full.tgz: The handle is invalid.
2021/03/17 11:40:24 NOTICE: VisualPrime Backup/TBAPICKUP2015: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\VisualPrime Backup\TBAPICKUP2015: The handle is invalid.
2021/03/17 11:40:24 NOTICE: VisualPrime Backup/TFPabrication: Removing partially written file on error: write \\?\UNC\192.168.1.20\Archive 2\Server 3-16-21\VisualPrime Backup\TFPabrication: The handle is invalid.
2021/03/17 11:42:13 DEBUG : Design Data/Ford Models/F250-350 Front Structure/HC3C-5005-CB_77-FR ASY A_P (F250_350 4X2 SRW S_C L_BOX 164.1WB)/HC3C-5005-CB_77-FR ASY A_P (F250_350 4X2 SRW S_C L_BOX 164.1WB).stp: Reopening on read failure after 29356032 bytes: retry 1/10: can't read status of source file while transferring: GetFileInformationByHandle \\?\UNC\192.168.1.20\TBA Server\Server\Design Data\Ford Models\F250-350 Front Structure\HC3C-5005-CB_77-FR ASY A_P (F250_350 4X2 SRW S_C L_BOX 164.1WB)\HC3C-5005-CB_77-FR ASY A_P (F250_350 4X2 SRW S_C L_BOX 164.1WB).stp: The handle is invalid.
2021/03/17 11:43:07 INFO  : 
Transferred:   	    3.009G / 20.273 GBytes, 15%, 17.126 MBytes/s, ETA 17m12s
Errors:               112 (retrying may help)
Checks:            112658 / 112658, 100%
Transferred:            0 / 57, 0%
Elapsed time:       3m0.1s
Transferring:
 * Design Data/Ford Model…/P552 SC SHORT BOX.stp:  4% /271.834M, 39.571k/s, 1h51m41s
 * Design Data/Ford Model…C 157WB TAILLIGHTS.stp:  5% /226.768M, 39.568k/s, 1h32m16s
 * Design Data/Ford Model…MY20 164WB 8FT BOX.stp:  2% /425.579M, 40.193k/s, 2h56m8s
 * Design Data/Ford Model…S_C L_BOX 164.1WB).stp:  2% /414.054M, 40.660k/s, 2h49m13s
 * Design Data/Ford Model…S_C L_BOX 164.1WB).stp:  4% /267.107M, 40.146k/s, 1h48m0s
 * Gina/My Documents/Own … 2012 ppt/Winter  .m4v:  2% /510.082M, 39.575k/s, 3h34m25s
 * Manufacturing/Backup V…-21-2020/ACTBA2017.bak:  7% /181.017M, 39.664k/s, 1h12m20s
 * Manufacturing/Backup V…1-2020/TBAOXES2011.bak:  1% /1.093G, 39.855k/s, 7h53m44s
 * Manufacturing/Backup V…20/TBAPICKUP2015.bak:  1% /1.139G, 39.380k/s, 8h19m52s
 * Manufacturing/Backup V…2020/TFPabrication.bak:  2% /504.230M, 39.567k/s, 3h31m56s
 * Design Data/Ford Model…164WB 8FT TAILGATE.stp:  1% /693.983M, 38.907k/s, 4h59m50s
 * Design Data/Ford Model…S_C L_BOX 164.1WB).stp:  2% /421.388M, 43.116k/s, 2h42m13s
 * Design Data/Ford Model…S_C S_BOX 147.9WB).stp:  2% /404.474M, 38.789k/s, 2h53m23s
 * Design Data/Ford Model…S_C S_BOX 147.9WB).stp:  5% /254.704M, 39.760k/s, 1h43m47s
 * Gina/My Documents/Own …e Come Girls Choir.mp4:  5% /223.717M, 40.047k/s, 1h29m48s

there's not much to be had from the log file other than what the verbose logging would say in the CLI. is there anything that commonly causes "handle invalid" for a local copy?

albertony · March 17, 2021, 5:49pm

No, it is not needed.

Not that I know of. What happens if you add argument --local-nounc=true?

For copying "local" paths like that, on Windows, I would probably use robocopy instead. Still, rclone should also be able to do it..

asdffdsa · March 17, 2021, 6:11pm

that is good software due to the the many and unique flags, but its fatal flaw is not able to verify file transfers.

fastcopy is open source, checksum files, good command line and a GUI.
i have used fastcopy for 8+ years, never a problem yet

ncw · March 17, 2021, 6:22pm

Can you try with the latest beta?

If you can still replicate then please make a new issue on github and that looks like a bug of some kind.

jared · March 17, 2021, 7:03pm

no this is still the same result.

robocopy has been discontinued and is only still being used by a small village in Africa..

if possible, i would like one software that can wing everything from local to the cloud

using the latest beta lands me in the same place brother. i'm not sure my setup isn't causing this. i'm using rclone on windows 10, as mentioned, to access an smb share hosted on a truenas installation on an old server i recycled. are there any known issues that ring a bell off the top of your head for this use case? though i'm constantly doing sync operations to this server using rclone..

Animosity022 · March 17, 2021, 7:13pm

Robocopy is used quite a bit and is part of every platform of Windows these days.

ncw · March 17, 2021, 7:24pm

There have been quite a few issues reported with SMB and Linux, but not with Windows as far as I know.

One thing you cloud try is disabling preallocate: Local Filesystem this is only in the beta at the moment.

asdffdsa · March 17, 2021, 7:39pm

a little off topic but since you posted a reply to me.

i am just the opposite - never use one tool to rule them all.

as for robocopy, after 25 years, it has been super stable and bug free.
in a windows server environment, i cannot think of any tool close to it.
after robocopy, when the source and dest are not the same computer, i can use a checksum verify tool.

Robocopy - Wikipedia
"10.0.19041.1 2020 Bundled with Windows 10 20H2"

on the other hand, many rcloners use rclone for local and based on forum posts, it does work well.

jared · March 17, 2021, 8:01pm

no joy. same result.

never is a long time

i'm a little new to CLI tools. most of my cases are file transfer operations over networks. i do hope rclone does a checksum for each file i transter..? i know it does when i use crypt remotes (i'm sure the sharepoint servers are getting tired of me already) but i'm hoping it does this locally without the encryption option.

asdffdsa · March 17, 2021, 8:10pm

good point, i deleted that from my post

well, for a rclone crypt remote, rclone does not perform a checksum check on upload.
the only way to know if a local file matches its corresponding crypted cloud file is rclone cryptcheck.

albertony · March 17, 2021, 8:40pm

I think it does:

asdffdsa · March 17, 2021, 8:58pm

i guess i had forgotten that.
after ncw stated that there is a version of cryptcheck for new uploads, i replied to that topic...

tho this is not shown in the debug logs
for crypt remote
INFO : 01.kdbx: Copied (new)

for non-crypt remote

DEBUG : zip/kdbx.20210314.185330.7z: MD5 = 814e77ae9ae377c3201193c858da46d7 OK
INFO  : zip/kdbx.20210314.185330.7z: Copied (new)

albertony · March 17, 2021, 9:26pm

Perhaps in next release it will be:

jared · March 18, 2021, 2:36pm

what about rsync? have you used that before? your thoughts?

asdffdsa · March 18, 2021, 2:46pm

not a linux expert.

for a one time copy/sync, not sure the practical difference.
but after that first copy/sync and a file in the source has changed and it already exists in the dest.

if the source file has just one byte changed in a 100GB file, rclone copy/sync has to copy the entire 100GB file.
rsync can figure out what parts have changed and copy only that part.

so it really depends on what your use-case is.

jared · March 18, 2021, 3:04pm

oh. right. forgot it's natively a linux application. the differential sync would be nice though. i'm guessing rclone will have it in the future as well

jared · March 18, 2021, 11:11pm

jojo. when doing a copy between two cloud remotes, no encryption/decryption involved, how come i'm not seeing the debug line that shows the checksum? the one with "OK" at the end?

asdffdsa · March 18, 2021, 11:12pm

what are the cloud remotes?
what is the command?

jared · March 18, 2021, 11:15pm

the cloud remotes are OneDrivePersonal to OneDriveBusiness. so the exact command would be:

rclone copy --verbose -vv --transfers=15 "JOfficeHome:TBA Server\Human Resource" "OneDriveTina:Server 3-12-21\Human Resource"

asdffdsa · March 18, 2021, 11:17pm

copy one file, and post the debug log