S3 Bucket mount in Windows

Help and Support
1

What is the problem you are having with rclone?

Hello, I'm a new user of rclone and I'm trying to create a script to mount S3 buckets in Windows.
The mount itself is working, but I'm experiencing a delay when uploading files to the S3 buckets. The delay is that I see the file or files in the mounted bucket in File Explorer. But other users do not see the file directly, depending on the size of the file, it takes about 10 minutes to an hour. Is there a way to remove this delay? The last command that I use, is pasted below, but that one cannot upload files larger than 1 GB it seems.
If someone has a idea about that, I would really appreciate it.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: Microsoft Windows 10 Enterprise 21H2 (64 bit)
  • os/kernel: 10.0.19044.1288 Build 19044.1288.1288 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount flux.s3:flux-smallprojects/ * --volname \\cloud\remote\flux-smallprojects --log-file C:\rclone\logs\rclone_mount_flux-smallprojects.log

The rclone config contents with secrets removed.

[flux.s3]
type = s3
provider = AWS
access_key_id = ********
secret_access_key = ********
endpoint = ********

A log from the command with the -vv flag

2023/03/23 06:35:57 NOTICE: Assigning drive letter "Y:"
The service rclone has been started.
The service rclone has been stopped.
2023/03/23 06:51:53 ERROR : *: Unmounted rclone mount
2023/03/23 08:24:20 NOTICE: Assigning drive letter "Y:"
The service rclone has been started.
The service rclone has been stopped.
2023/03/23 08:26:59 ERROR : *: Unmounted rclone mount
2023/03/23 08:27:09 NOTICE: Assigning drive letter "Y:"
The service rclone has been started.
The service rclone has been stopped.
2023/03/23 08:32:31 ERROR : *: Unmounted rclone mount
2023/03/23 08:33:17 NOTICE: Assigning drive letter "Y:"
The service rclone has been started.
2023/03/23 08:33:40 ERROR : rclone.conf: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/03/23 08:34:27 ERROR : FileZillaProject_FileZillaClient_3.63.2.1_EN_x64_R01.zip: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/03/23 08:34:27 NOTICE: S3 bucket flux-smallprojects: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
2023/03/23 08:35:47 ERROR : Cardpresso.zip: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/03/23 08:38:54 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/03/23 08:39:57 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:39:58 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:39:59 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:00 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:01 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:02 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:03 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:04 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:04 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/03/23 08:40:05 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:06 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:07 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:08 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:09 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:10 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:11 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:40:12 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:45:54 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/03/23 08:47:34 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:35 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:36 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:37 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:38 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:39 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:40 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:41 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:41 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/03/23 08:47:42 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:43 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:44 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:45 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:47 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:48 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:49 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/03/23 08:47:50 ERROR : SAS_Add-inStandalone_8.3_ENG_x64_R03.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
The service rclone has been stopped.
2023/03/23 10:20:26 ERROR : *: Unmounted rclone mount
2

On S3 which doesn't support change notify, you will only see changed things on the remote when the directory cache has expired

--dir-cache-time duration   Time to cache directory entries for (default 5m0s)

So set this to the time that you are happy with. Lower means more transactions with S3 and more cost, but more immediate views of new files.

3

hello and welcome to the forum,

ERROR : Cardpresso.zip: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
to remove that error, need to use --vfs-cache-mode=writes or --vfs-cache-mode=full

next time, can you post the output a full debug log, or at least the top 30 lines?
--- add --log-level=DEBUG

for instant views of new files: force rclone to refresh the vfs dir cache.
--- to the mount command, add --rc --rc-no-auth
--- run rclone mount command.
--- wait until the rclone mount command is running.
--- run rclone rc vfs/refresh recursive=true -vv

4

Hi ncw,

thank you for your quick response. I've tried this, but it does not give the solution I was hoping for.
I used --dir-cache-time 10s in the command line.
Than I copied a zip file to the mount. File Explorer gave the confirmation that the file was copied, but when I look on the S3 storage via WinSCP, the file was not visible till about 30 minutes after. Also, on the other test account I use for this, the file was not visible within the mount.

What I want to achieve, is about 9 users with a mounted S3 bucket and that they can work simultaneously in the bucket and that all files shared are visible with the least amount of delay(preferably no delay).

5

Hi,

thank you for the quick response!

When I used the --vfs-cache-mode=writes , than there is a delay in the sync for all files, not just the big ones.

for instant views of new files: force rclone to refresh the vfs dir cache.
--- to the mount command, add --rc --rc-no-auth
--- run rclone mount command.
--- wait until the rclone mount command is running.
--- run rclone rc vfs/refresh recursive=true -vv

When I try to use --rc --rc-no-auth with my mount command line, I just get a empty cmd windows waiting on a command.
When I use rclone rc, i need to use more parameters i think, because i just get a unauthorized message.

Is that correct?

6

S3 is not a polling remote. perhaps use gdrive and dropbox, as they support polling.

the best you can have is on each machine running rclone mount

  • set --dir-cache-time, setting this to a low value can use lots of api calls and that costs money.
    note: wasabi, s3 clone, does not charge for api calls.
  • use rclone vfs/refresh

how do you plan to implement file locking with multiple mounts?

i suggest that:

  • run a single instance of rclone mount and share that using over network using smb/samba/nfs/etc....
    or
  • run a single instance of rclone serve webdav and on each client, map a drive letter to the webdav server.
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.