Rclone mount as nssm service and permissions

Thanks for these clues. I will downgrade the packages on the RGW and try again. This is the only change I can see for now besides any Windows updates that might have broken WinFSP. As I said previously, creating empty folders was working fine until recently and Ceph RGWs handles this well whatever the client.

First of all, let me agree with this statement : "TLDR: ncw is awesome!" :smiley:
For a moment I thought I was hitting the same one as you hit as I had seen this messages ERROR : : Entry doesn't belong in directory "" many times in the past using the linux version of rlcone. But... Trying the latest rclone-beta-latest-windows-amd64.zip did not help with the 4 folders being created. I'm left with downgrading yum packages on one of our RGWs.
You guys have a nice day.
Frédéric.

Well well well... I could isolate one of the RGWs and downgrade the packages to the version that I know was working fine. Then I forced Windows network traffic to go through this RGWs (entry in hosts file) and it appears that the Ceph version is not the cause. Creating a folder still results in showing 4 folders, with actually none created.

What I don't understand is why this is now happening as I don't remember ever encountering this issue before, when I made all my tests of the rclone mount in September (Rclone download) and the windows user attached to the rclone mount service was already the .\SYSTEM user.
When mounting from the CLI with no user privileges elevation (Windows+R + 'cmd') I can create a folder with no issues (virtually as the folder is not actually created until a file is dropped in it) and it doesn't turn into 4 folders being created (or not created actually).
When mounting from the CLI with Administrator privileges elevation ('cmd' run as Administrator) then it doesn't work and a folder creation turns into 4 folders... Same as if the mount was started by the service under the .\SYSTEM account.

I'm not sure how to solve this. Should I use a specific user to run the windows service?

Frédéric.

This is what Bill Ziss (the WinFSP developer) said about the 4 directory problem

This is almost always a permission issue. Windows Explorer has this weird behavior where if it gets "Access Denied" during directory creation it retries it 4 times! In order for this problem to be resolved you will have to fix the permissions that the file system reports.

This could perhaps be a permission problem of the underlying s3 system, but I still suspect some problem with the actual WinFSP config.

Note that s3 doesn't really support empty directories without making directory markers and rclone doesn't make directory markers.

This means you can create empty directories in the mount but they will disappear after the directory cache time expires.

Some clients do create directory markers (which are 0 length files ending in /).

BTW I run the integration tests with rclone against CEPH+RGQ both swift and s3 interfaces.

As an experiment open a cmd window as a normal user and run the rclone mount command. See if it works like that.

Hi Nick,

Thank you for your time.

When I open a cmd window and mount as normal user (I'm logged in as Administrator actually), everything works as expected. Creating a folder results in 1 folder being created. When the mount is done as normal user, here are the rights on S:
user_mount

When the mount is done by the service (run as SYSTEM), creating a folder fails and triggers the 4 folders retry scenario described by Bill. When the mount is done by the service, here are the rights on S:
service_mount
It's clearly a right issue. My question is how do I work around this?
It looks like changing the rights on S:\ is not allowed (buttons are greyed out).

Can you tell me more about the WinFSP config and how it could help fixing the permission issue?

I'd like the mount done as a service (by local SYSTEM user) and Administrators able to read/write files in the mount, eventually any local user allowed to read/write in the mount.

PS: I believe I know now the reason why it doesn't work anymore: this server has joined the Active Directory infrastructure.

Frédéric.

OK, we've nailed down the problem which is good :slight_smile:

Now we are talking about rights on Windows we are way beyond my knowledge of Windows alas (I'm a unix person).

BillZ always recommends using the WInFSP launcher at this point - have you given that a go?

Or alternatively you could use nssm which I know users have had success with.

Nick,

Thank you.

I've used nssm.exe to create the service, actually. I don't think it helps by itself, but maybe this is because the Windows Service was created before the computer joined the Active Directory infrastructure. Maybe this is something SID related and I need to uninstall/reinstall WinFSP and/or remove the Windows Service and recreate it using nssm.exe. That's the next 2 things I'll try.

Apparently, the troubles came with joining this computer to the Active Directory. Maybe @billziss-gh will know anything about this.

I didn't try WinFSP launcher yet. I need to read WinFSP documentation to understand what option does what. Looks like @billziss-gh has got something here : https://github.com/rclone/rclone/issues/3216#issuecomment-503880038 but I don't know how to use it.

Also this could help : https://github.com/billziss-gh/winfsp/wiki/WinFsp-Tutorial#running-the-file-system-as-a-service

If I come up with something, I'll let you know. It would be nice to have rlcone handle this by itself, if not in the way it mounts the resource, in the way it configures the essentials regarding the rights (WinFSP registry entries or so) so when the resource is mounted, it's accessible to all authenticated users.

Bests,
Frédéric.

Great!

Yes rclone should really create the registry entries for the WinFSP.Launcher. However I haven't had the time to do that yet. If a Windows user figured out what needed to be done I'd be happy to do the programming :slight_smile:

Hi Nick,

I finally made some progress.

First, I just want to recall that this permissions issue started to happen after the computer was joined to the Active Directory domain. It don't know if this help any understandings but this is here...

What I tried was :

  • remove the Windows Services that was created with nssm.exe prior to the computer joining the AD, and recreate the service. Didn't help.
  • remove WinFSP, reboot and reinstall WinFSP. Didn't help either.

What helped me was this post from @ghost (@pbm3 I believe before his Github account was deleted):

The problem was indeed the missing uid.
'rclone.exe mount gdrive: X: --config rclone.conf -o uid=197609' did fix the folder duplication for me.

And this WinFSP FAQ related to FUSE mounts:

Alternatives include -o uid=-1,gid=-1 , which presents files as owned by the user and group that launched the file system and -o uid=-1,gid=11 , which presents files as owned by the user that launched the file system and the group "Authenticated Users". (The fsptool utility in the bin subdirectory of the WinFsp installation directory can be used to convert Windows accounts/SID’s to UID’s and vice versa.)

So I ran the 'fsptool' and noted the Administrator 'uid' which was 197108 on my side, edited the Windows Service and added -o uid=197108,gid=11 at the end of the rclone mount command.

Now Administrator user and Authenticated Users group have write permissions and everything is fine.
Well... At least Administrator can write into the mount.

Still, it's not clear to me whether or not rclone has a -o command line option.

What bothers me is that when using uid=<any_user>,gid=544, the Administrator cannot write into the mount despite being a member of group 544 which is the local admin group. Looks like the WinFSP-FUSE permission set for gid is read-only.

[EDIT] Actually...

  • when using uid=<any_user>,gid=544, the Administratror account can create / delete files but creating a folder results in 4 different folders (the retry/fail loop)
  • when using uid=197108 or uid=65792 (ID of user Everyone), the Administrator account has no issues creating / deleting any files and creating a folder actually creates a folder as anyone would expect (not 4 folders).

Best regards,
Frédéric.

Rclone mount does and it will pass those options straight on to WinFSP

It sounds like you are making great progress!

Rclone should probably document or automate some of these tweaks to be a bit more user friendly on Windows.

From the WinFSP FAQ

The WinFsp-FUSE layer includes a built-in command line option that can help: -o uid=-1 . This instructs the WinFsp-FUSE layer to present all file system files as if they are owned by the user that launched the file system.

Alternatives include -o uid=-1,gid=-1 , which presents files as owned by the user and group that launched the file system and -o uid=-1,gid=11 , which presents files as owned by the user that launched the file system and the group "Authenticated Users". (The fsptool utility in the bin subdirectory of the WinFsp installation directory can be used to convert Windows accounts/SID’s to UID’s and vice versa.)

Note that Rclone passes -o uid=-1,gid=-1 by default to WinFSP.

What would be your recommendation for updated docs for rclone? Do you think rclone should be doing something different here?

Hello Nick,

I narrowed it down a bit. Since this issue clearly happened after the Windows 2019 Standard Version 1809 server joined the Active Directory domain, I decided to check whether this permissions issue / behaviour (4 folders false creations) was due to the fact that computer was part of an Active Directory domain or not. So I took another server, but this time a Windows 2012 R2 server.

I installed WinFSP and rclone and created the very same nssm.exe service with the exact same rclone mount options (with no -o uid... option), so with the service run by the SYSTEM account. No issues.
I then joined the computer to the very same Active Directory domain. No issue! I was expecting the opposite actually.

On the Windows 2019 server, the fsptool shows these permissions:

C:\Program Files (x86)\WinFsp\bin>fsptool-x64.exe perm s:
O:SYG:SYD:P(A;;FA;;;SY)(A;;0x1201ef;;;SY)(A;;0x1201ef;;;WD) (perm=18:18:0777)

On the Windows 2012 R2 server, the fsptool shows theses permissions:

C:\Program Files (x86)\WinFsp\bin>fsptool-x64.exe perm t:
O:SYG:SYD:P(A;;FA;;;SY)(A;;0x1201ef;;;SY)(A;;0x1201ef;;;WD) (perm=18:18:0777)

Exact same permissions on both computers.

Still, creating a folder on the Windows 2019 server results in 4 folders showing up in Windows Explorer.
Apart from that, writing a file in the root of the volume or in one of the 4 folders succeeds afterwards.
It's just that 4 folders get (virtually) created instead of one which looks like a bug to the user.

This looks to me like permissions got tighten since recent Windows versions. It's not clear to me whether this should be handled by loosening these windows restrictions or by using the -o option to set the file property. But maybe a clever way to fix this would be for WinFSP (I doubt rclone can do that) to tweak the NTFS special permissions. @billziss-gh might help us here.

Regarding your question about the documentation, it currently reads:

The easiest way around this is to start the drive from a normal command prompt. It is also possible to start a drive from the SYSTEM account (using the WinFsp.Launcher infrastructure) which creates drives accessible for everyone on the system or alternatively using the nssm service manager.

Until we find a global fix, I think we could add something like this:


There may be some situations were the drive might not be accessible to everyone from Windows Explorer, even to the Administrator account. In these situations using the -o option to set files property would help, for example:

rclone mount .... -o uid=65792,gid=544

uid=65972 is for user "Everyone" and gid=544 is for group "Administrators". fsptool that comes with WinFSP can be used to get user/group uid and gid.

C:\Program Files (x86)\WinFsp\bin>fsptool-x64.exe id "Administrator"
S-1-5-21-2790659633-1931783022-730602116-500(PC\Administrator) (uid=197609)

I'll let you rephrase it as English is not my mother tongue. :wink:

Hope @billziss-gh can see this post and come help. :slight_smile:

Bests,
Frédéric.

Hi Nick,

Ii turns out that the Active Directory had nothing to do with it as it also happens on a Windows 2019 Server that has never joined the AD domain.

I will try to follow the FileSpy route suggested by Bill and report to him then: https://github.com/rclone/rclone/issues/3216#issuecomment-508567098

We'll see if he can advise on how to set permissions right to avoid this bad 4 folders behaviour with recent Windows OSs.

Take care,
Frédéric.

i have access to many windows personal and server operating systems using rclone mount and winfsp.
i have run rclone as standard user, admin user and as system user.
in all my time using winfsp, i have never seen this issue of four folder behavior.

can you state in a concise manner how i might replicate it?

Hi asdffdsa,

Sure. Set up a Windows Service to "rclone mount" a Ceph S3 bucket in Windows 2019 Standard (French) as S:\ using nssm.exe. The Windows service will run as user .\SYSTEM by default. Start the Windows service. When logged in as user .\Administrator, open Windows Explorer and browse to S:.
Create a folder and see what happens.

Frédéric.

thanks,

the server is the free windows server 2019 hyper-v edition.
it is command line version, no gui, so i am blessed not having windows explorer.
i am using task scheduler, not nssm

i tried in the past to replicate this "4folderproblem" and now again, i am not able to replicate.
the mount is pointing to s3 clone provider wasabi

t:\>mkdir test4folderproblem
t:\>dir
 Volume in drive T is wasabieast2 thefourfolderproblem
 Volume Serial Number is 60D4-1C7C

 Directory of t:\

05/27/2020  06:09 PM    <DIR>          test4folderproblem
               0 File(s)              0 bytes
               1 Dir(s)  1,125,899,906,842,624 bytes free

t:\>whoami
vserver03\administrator

t:\>ver
Microsoft Windows [Version 10.0.17763.348]

edit:
on that server, i use the app double commander, and with that, still not able to replicate the 4folderproblem.

Thanks for testing. I doesn't happen either on my side want I create the folder from the Windows shell. The issue happens when trying to create a folder from Windows Explorer as .\Administrator.

I'm now pretty sure that this is a permission issue, even more when considering that NTFS permissions are set as special permissions instead of classic permissions on the mount point.

I created 2 new Virtual Machines. A Windows Server 2016 Standard and a Windows Server 2019 Standard. The 4 folders issue happens on both of them. Nos issues creating a folder from CLI but creating a folder from Windows Explorer turns into 4 folders being created. Didn't happen with Windows Server 2012 R2.

100% reproducible.

This is clearly a permission issue. There are many topics on the Internet regarding this issue. You can try searching for "new folder creates 4 folders"... My next move is to ping @billziss-gh.

Frédéric.

hi,

i was able to replicate the 4 folder problem
on ws.2019.essentials, using windows explorer.

then when i ran the same mount command on:

  • w10.1909. i was able to see those 4 folders
  • w2019.hyperv. i was able to see those 4 folders.

i was not able to replicate the problem using:

  • w10.1909 using windows explorer
  • on ws.2019.hyperv, as it does not run windows explorer.
  • on ws.2019.essentials, using double commander and command line.

so it would seem the problem happens.

  • later version of windows server
  • using windows explorer.

Thanks again for your tests Jojo :slight_smile:

I created this issue here: https://github.com/billziss-gh/winfsp/issues/308

Best regards,
Frédéric.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.