Error mounting s3 bucket in docker container on Windows 2022 on AWS EC2

Help and Support
1

What is the problem you are having with rclone?

With the same configuration (dockerfile), the S3 bucket mounts OK in the docker host and container on my Windows 11 laptop, and in the docker host on EC2, but not in a container on EC2.
Same results with/without the --isolation=process paramter in the docker run command.

022/01/25 22:01:32 ERROR : S3 bucket rclonetest20220225: Mount failed
2022/01/25 22:01:32 Fatal error: failed to mount FUSE fs: mount stopped before calling Init: mount failed

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: Microsoft Windows Server 2022 Datacenter 2009 (64 bit)
  • os/kernel: 10.0.20348.473 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This fails:

"c:\ominer\bin\rclone\rclone.exe" mount s3clone:rclonetest20220225 c:\rclonetest20220225 --vfs-cache-mode full --config="c:\ominer\config\rclone.conf"

This works:

"c:\ominer\bin\rclone\rclone.exe" ls s3clone:rclonetest20220225 --config="c:\ominer\config\rclone.conf"

The rclone config contents with secrets removed.

[s3clone]
type = s3
provider = AWS
access_key_id = XXXXXXXXXXXXXXXXXXXX
secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
region = eu-west-2
location_constraint = eu-west-2
storage_class = STANDARD

A log from the command with the -vv flag

2022/01/25 22:10:03 DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\\ominer\\bin\\rclone\\rclone.exe" "mount" "-vv" "s3clone:rclonetest20220225" "c:\\rclonetest20220225" "--vfs-cache-mode" "full" "--config=c:\\ominer\\config\\rclone.conf"]
2022/01/25 22:10:03 DEBUG : Creating backend with remote "s3clone:rclonetest20220225"
2022/01/25 22:10:03 DEBUG : Using config file from "c:\\ominer\\config\\rclone.conf"
2022/01/25 22:10:03 INFO  : S3 bucket rclonetest20220225: poll-interval is not supported by this remote
2022/01/25 22:10:03 DEBUG : vfs cache: root is "C:\\Users\\ContainerAdministrator\\AppData\\Local\\rclone"
2022/01/25 22:10:03 DEBUG : vfs cache: data root is "\\\\?\\C:\\Users\\ContainerAdministrator\\AppData\\Local\\rclone\\vfs\\s3clone\\rclonetest20220225"
2022/01/25 22:10:03 DEBUG : vfs cache: metadata root is "\\\\?\\C:\\Users\\ContainerAdministrator\\AppData\\Local\\rclone\\vfsMeta\\s3clone\\rclonetest20220225"
2022/01/25 22:10:03 DEBUG : Creating backend with remote "C:/Users/ContainerAdministrator/AppData/Local/rclone/vfs/s3clone/rclonetest20220225"
2022/01/25 22:10:03 DEBUG : fs cache: renaming cache item "C:/Users/ContainerAdministrator/AppData/Local/rclone/vfs/s3clone/rclonetest20220225" to be canonical "//?/C:/Users/ContainerAdministrator/AppData/Local/rclone/vfs/s3clone/rclonetest20220225"
2022/01/25 22:10:03 DEBUG : Creating backend with remote "C:/Users/ContainerAdministrator/AppData/Local/rclone/vfsMeta/s3clone/rclonetest20220225"
2022/01/25 22:10:03 DEBUG : fs cache: renaming cache item "C:/Users/ContainerAdministrator/AppData/Local/rclone/vfsMeta/s3clone/rclonetest20220225" to be canonical "//?/C:/Users/ContainerAdministrator/AppData/Local/rclone/vfsMeta/s3clone/rclonetest20220225"
2022/01/25 22:10:03 DEBUG : Network mode mounting is disabled
2022/01/25 22:10:03 DEBUG : Mounting on "c:\\rclonetest20220225" ("s3clone rclonetest20220225")
2022/01/25 22:10:03 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/01/25 22:10:03 DEBUG : S3 bucket rclonetest20220225: Mounting with options: ["-o" "attr_timeout=1" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=s3clone rclonetest20220225"]
2022/01/25 22:10:03 ERROR : S3 bucket rclonetest20220225: Mount failed
2022/01/25 22:10:03 Fatal error: failed to mount FUSE fs: mount stopped before calling Init: mount failed
2

The dockerfile, with the application specific parts removed is:

FROM mcr.microsoft.com/windows/servercore:ltsc2022

# Install winfsp
COPY winfsp-1.10.22006.msi c:\\ominer\\installs\\winfsp.msi
RUN msiexec /q /i "C:\ominer\installs\winfsp.msi"

# Install rclone
COPY rclone-v1.57.0-windows-amd64.zip  c:\\ominer\\installs\\rclone.zip
RUN powershell -Command "expand-archive -Path 'c:\ominer\installs\rclone.zip' -DestinationPath 'c:\ominer\temp'"
RUN xcopy /iery C:\ominer\temp\rclone-v1.57.0-windows-amd64 C:\ominer\bin\rclone

COPY admin  c:\\ominer\\admin
RUN c:\ominer\admin\aws-mount-config.bat
ENTRYPOINT ["c:\\ominer\\admin\\aws-mount.bat","c:\\"]

systeminfo in the host outputs:

Host Name:                 EC2AMAZ-1234567
OS Name:                   Microsoft Windows Server 2022 Datacenter
OS Version:                10.0.20348 N/A Build 20348
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          EC2
Registered Organization:   Amazon.com
Product ID:                00454-60000-00001-XX999
Original Install Date:     1/20/2022, 9:08:39 AM
System Boot Time:          1/25/2022, 7:27:09 PM
System Manufacturer:       Amazon EC2
System Model:              t3.xlarge
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 85 Stepping 4 GenuineIntel ~2500 Mhz
BIOS Version:              Amazon EC2 1.0, 10/16/2017
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC) Coordinated Universal Time
Total Physical Memory:     16,196 MB
Available Physical Memory: 12,228 MB
Virtual Memory: Max Size:  19,140 MB
Virtual Memory: Available: 15,033 MB
Virtual Memory: In Use:    4,107 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    WORKGROUP
Logon Server:              \\EC2AMAZ-1234567
Hotfix(s):                 3 Hotfix(s) Installed.
                           [01]: KB5008882
                           [02]: KB5010796
                           [03]: KB5008995
Network Card(s):           1 NIC(s) Installed.
                           [01]: Amazon Elastic Network Adapter
                                 Connection Name: Ethernet
                                 DHCP Enabled:    Yes
                                 DHCP Server:     192.168.0.1
                                 IP address(es)
                                 [01]: 192.168.0.45
                                 [02]: fe80::d8f7:48b9:5678:1234
Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.
3

hello and welcome to the forum,

no idea why this fails in that one case, container on EC2.
do not think i have seen a docker on windows post here in the forum.

fwiw, i always test in a new situation, using the simplest command possible, no remotes, no vfs file cache, no config file.
rclone.exe mount c:\source c:\mount

4

Thank you for your reply, and your suggestion. This also fails:

c:\ominer\bin\rclone>rclone.exe mount c:\ominer c:\ominermount
2022/01/25 23:12:08 NOTICE: Config file "C:\\Users\\ContainerAdministrator\\AppData\\Roaming\\rclone\\rclone.conf" not found - using defaults
2022/01/25 23:12:08 ERROR : Local file system at //?/c:/ominer: Mount failed
2022/01/25 23:12:08 Fatal error: failed to mount FUSE fs: mount stopped before calling Init: mount failed
5

if rclone could not find winfsp, there would be a different error.

try mounting to a drive letter, not a folder.

i would look up how to enable logging for winfsp.

6

So it is unlikely that the problem is with winfsp?
What would cause the error?

I couldn't get mounting to a drive letter to work in a Docker container, on Windows Server 2022 or Windows 11.

BTW This works fine in a Docker container on Windows 11:

rclone.exe mount c:\ominer c:\ominermount

I have looked it up, but all I can find is source code. Any ideas?

7

my point is that,
if rclone could not find winfsp then would be a very specific error about that.
your error message is generic.

no idea. could be rclone, could be winfsp, could be something else.

8

Same problem with rclone-v1.56.2-windows-amd64.zip

I'd be surprised if I was the only person to try mounting S3 as a volume in a Windows docker container on AWS. Seems like a pretty obvious thing to want to do. But how?

9

well, then should be able to find posts about docker on windows here in the forum.
if you find any, let me know.

10

I couldn't find anything, so maybe it just works for everyone else and I'm doing something wrong which has not been spotted.

But maybe it's a bug triggered by the specifics of my setup or I'm just the first to hit it, so I reposted as a Suspected Bug: Mount fails in a docker container on Windows Server 2022 Datacenter

11

The 'Suspected bug' post appears to have been deleted. Why is that?

12

Because it's a duplicate of this one as we don't need 2 posts for the same thing.

13

well, i did not delete it but why open a second topic?
best to use this one.

imho, so far, there is no evidence this is a rclone bug.
rclone on docker is designed to run on standard linux only.
rclone on docker on windows is a very narrow use-case.

could be an issue with
--- docker on windows
--- ws2022
--- ec2
--- winfsp

14

Bingo, eureka, you were right, the issue was with winfsp not being installed in the host.

For rclone to work in a Windows docker container, WinFsp must be installed in the host AND the container.
And the container run with the --isolation=process parameter.

As stated in black and white here WinFsp Container Support · WinFsp

The reason it worked on Windows 11 is I'd been playing with rclone before trying it in a Docker container.

Thank you for your help and pointing me in the right direction.

15

good to know. and that explains everything.

sure, winfsp needs to be installed where rclone is executed. i just assumed that from that start.

what threw me was the generic error you got,
Fatal error: failed to mount FUSE fs: mount stopped before calling Init: mount failed

that had me thinking that winfsp was installed in the container, and the issue was a config issue, permissions issue, or something like that.

as normally, if rclone cannot find winfsp, it outputs a very specific error.
Fatal error: failed to mount FUSE fs: mount stopped before calling Init: mount failed: cgofuse: cannot find winfsp

16

winfsp was installed in the container, where rclone was executing, which is why rclone found winfsp.

But winfsp also needs to be installed in the host, for its File System Driver.

I can see this being a common mistake, because the mentality of the docker administrator is to install all the software in the container and minimise configuration in the host. But drivers are about hardware, so belong in the host.

17

Windows+docker+s3 might be an unusual use-case at the moment, but I think it'll grow rapidly when people catch on - you can migrate your legacy applications to fully managed cloud with effectively unlimited disk space that's automatically backed up, all without rewriting a line of code. That's gold.

18

i thought winfsp was installed on the host, as per this

19

Those tests were not on the same machine. I created/terminated multiple server instances while testing different Windows versions, application versions, 3rd party tools, configurations etc over several days, so once I confirmed winsft+rclone worked in the host, I didn't install it in subsequent instances, didn't realise I needed to, until now.

20

ok, that was not clear in the opening post.
in any event, now it is all working.