Rclone sees imaginary files that I cannot

What is the problem you are having with rclone?

rclone sees a bunch of imaginary files so transfers are not working.

When I run ls on my Desktop folder I see a few files, if i use *rclone.exe ls ..\Desktop* it shows a bunch of stuff i do not recognize.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

  • os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
  • os/kernel: 10.0.19045.4170 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

none yet, planning on onedrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

.\rclone.exe ls ..\Desktop\

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[SLAM]
type = onedrive
token = XXX
drive_id = XXX
drive_type = business
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

Output

welcome to the forum,

please post that

you posted v1.66.0 but your command has v1.65.2

Thanks!

here you go:

PS C:\Users\slam\rclone-v1.65.2-windows-amd64> .\rclone.exe ls ..\Desktop\ -vv
2024/04/11 08:15:26 DEBUG : rclone: Version "v1.66.0" starting with parameters ["C:\\Users\\slam\\rclone-v1.65.2-windows-amd64\\rclone.exe" "ls" "..\\Desktop\\" "-vv"]
2024/04/11 08:15:26 DEBUG : Creating backend with remote "..\\Desktop\\"
2024/04/11 08:15:26 DEBUG : Using config file from "C:\\Users\\slam\\AppData\\Roaming\\rclone\\rclone.conf"
2024/04/11 08:15:26 DEBUG : fs cache: renaming cache item "..\\Desktop\\" to be canonical "//?/C:/Users/slam/Desktop"

ok, so are running latest rclone inside a folder named after an old version of rclone?
that was/is confusing.

hmm, this is strange, not sure what is going on.
i would try to narrow down the cause

--- did you have the issue when using v1.65.2 ?
--- run rclone under cmd.exe and see what the output is.
--- test on another folder, not inside your homedir.

And what you see when you run:

ls -al ..\Desktop\

??

Possibly there are many hidden files saved in your Desktop directory.

i couldn't make the al flags work in powershell, but I found a link that says -Force should work.

PS C:\Users\slam\Desktop> ls -Force
    Directory: C:\Users\slam\Desktop
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         3/12/2024   8:04 AM         111090 0236_001.pdf
-a----          4/2/2024   8:21 AM        4705453 4-2-24 Inventory.rtf
-a----         4/21/2023   1:35 PM           2444 Brave.lnk
-a-hs-         4/20/2023   7:33 AM            282 desktop.ini
-a----         3/12/2024   8:06 AM         130943 Hospital Bill and Note.pdf
-a----         3/12/2024   8:04 AM          21967 Hospital Bill Note.pdf
-a----          5/8/2023  11:07 AM           2264 Tabby Terminal.lnk
-a----         2/14/2024   3:34 AM           2084 Webex.lnk

The only thing that shows that didn't before was desktop.ini.

ok - my wrong. I thought it is Linux shell.

Yes that's from when I installed it, I updated it before posting. I can go ahead and copy the exe to system32 to make it less confusing.

The problem exists in both versions.

Using command it does the same thing.

It looks okay until I go into a c:\Users\ folder.
All User folders have gibberish in them, according to rclone.

those files look like dummy files used by anti-virus/anti-ransomware software

what anti-virus and such protection software is running on that machine?

Maybe!

cortex xdr

Anyway to filter them out?

have you searched the internet, it seems that those filenames are not gibberish.

I have, but not effectively.

I assume you mean search for those filenames exactly, which I will now do.

Run your command with debug "-vv" and share the full log.

for each file, what are the file attributes?
Get-ItemProperty -Path ..\Desktop\!!!!!1613157082.pem | Format-list -Property * -Force

read about rclone filters
a very simple example,
rclone ls ..\Desktop\ --exclude="!!!!*"

PSPath            : Microsoft.PowerShell.Core\FileSystem::C:\Users\slam\Desktop\!!!!!1613157082.pem
PSParentPath      : Microsoft.PowerShell.Core\FileSystem::C:\Users\slam\Desktop
PSChildName       : !!!!!1613157082.pem
PSDrive           : C
PSProvider        : Microsoft.PowerShell.Core\FileSystem
Mode              : ---h--
VersionInfo       : File:             C:\Users\slam\Desktop\!!!!!1613157082.pem
                    InternalName:
                    OriginalFilename:
                    FileVersion:
                    FileDescription:
                    Product:
                    ProductVersion:
                    Debug:            False
                    Patched:          False
                    PreRelease:       False
                    PrivateBuild:     False
                    SpecialBuild:     False
                    Language:

BaseName          : !!!!!1613157082
Target            : {C:\ProgramData\Cyvera\Ransomware\4899759477436019851\!!!!!2740303738.pem}
LinkType          :
Name              : !!!!!1613157082.pem
Length            : 4048
DirectoryName     : C:\Users\slam\Desktop
Directory         : C:\Users\slam\Desktop
IsReadOnly        : False
Exists            : True
FullName          : C:\Users\slam\Desktop\!!!!!1613157082.pem
Extension         : .pem
CreationTime      : 3/26/2024 9:28:29 AM
CreationTimeUtc   : 3/26/2024 2:28:29 PM
LastAccessTime    : 3/26/2024 9:28:29 AM
LastAccessTimeUtc : 3/26/2024 2:28:29 PM
LastWriteTime     : 3/26/2024 9:28:29 AM
LastWriteTimeUtc  : 3/26/2024 2:28:29 PM
Attributes        : Hidden

filtering !!! is one thing, but then there is also

XORXOR1038814426.doc
ZZZZZ1171102099.pdf
fnRhxanX1717037742.pptx

I guess I need to run a recursive tree on the entire user directory and work out all the filters to ignore all these annoying files.

It's at least a path forward!

Also clearly these are not "imaginary" files. They seem to be hidden from Windows shell but not from rclone. Something on your computer creates these files for some reason.

I am not Windows user but above line sort of points into what @asdffdsa suggested - some anti-virus/ant-ransomware program at work.

So not really rclone problem... It lists files your OS + anti-virus software serve.

yeah, as @asdffdsa hinted

apparently they are honeypot files.

so rclone can see past whatever kinda magic they are using to keep them from PS and CMD

the same as any other program this software tries to protect you from:) Files are there - but some Windows wizardry hides them from your shell and probably explorer.

I think the best what you could do is to talk to this anti-virus authors/community. What is their solution? As this potentially affects many other backup software etc.

On the other hand if obvious exclusion list exist than all honypot idea is pointless:) As can be used by clever bad software too to not touch these files.

Attributes : Hidden

not sure it is --magic, as by default windows os hides hidden and system files.
whereas rclone does not.
so for now, my suggestion of using rclone filters should be a kludge workaround.

up above, i shared the weblink for the rclone docs about filters.
so take a few minutes to read the rclone docs, about --exclude-from

Yeah, but he did an ls -force in powershell which will show you all files, even hidden/system/etc.

Very interesting. I have not seen antivirus software place honeypot files before. Maybe it uses some sort of filesystem filter driver that is process aware. If you use cmd, powershell, explorer (etc), it doesn't add the fake files, but another process gets the fake entries.