AWS S3 Mount Hanging/Crashing

Help and Support
1

What is the problem you are having with rclone?

AWS S3 mount

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1

  • os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
  • os/kernel: 10.0.17763.3650 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.19.3
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

.\rclone.exe mount aws_s3:igea J: --read-only --no-modtime --s3-no-system-metadata --cache-dir P:\rclone_uew_aws_cache --vfs-cache-mode full --vfs-cache-max-size 10000M -o --log-file=P:\rclonelogging\uewawslog.txt --log-level DEBUG

The rclone config contents with secrets removed.

[aws_s3]
type = s3
provider = AWS
env_auth = true
access_key_id = <hidden>
secret_access_key = <hidden>
region = us-west-2

Basically, the S3 bucket has 2 directories containing over 1.6 million files in each directory and if I try to open either, explorer freezes and sometimes even crashes. I have tried the --no-modtime flag to try and speed it up but the result is the same.

As a secondary issue, logging isn't working either and I'm not sure why

2

hello and welcome to the forum,

i would remove -o

P:\
that is a local path?

3

Can explorer cope with directories with 1.6M files?

4

That's a fair point, but is there possibly a way to have rclone only show the files in chunks of 1000 or so?

5

It's not rclone as Explorer wants to display all the files in a folder so you are hitting that I'd imagine as the best advice is don't use explorer.

6

hi,
add these flags to the mount command, should help with windows explorer
--network-mode --buffer-size=0

that should not be needed.
given your mount is read-only, might want to pre-load the vfs dir cache.
after that, rclone will not have to re-read list of files/dirs

rclone has two types of vfs cache, can read about it
https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34

  1. make sure to add --network-mode --buffer-size=0
    now to pre-load the vfs dir cache.
  2. add --rc --rc-no-auth
  3. start the mount, but do NOT access J:
  4. run rclone rc vfs/refresh recursive=true -vv - can take a few minutes to complete
  5. now navigate J: - hopefully will be more performant.
7

another issue could be memory.
could need 4GiB of memory for the vfs dir cache.

8

I will give this a try!

9

i did some quick testing, and got an interesting result, tho not sure exactly what is implies.

on local laptop, approx 1,000,000 files in one folder.
on s3, using wasabi, have the same set of files.

on local, windows explorer takes approx. 12 seconds to view the 1,000,000 files.
so that shows windows explorer can handle 1,000,000 files in a single folder

  1. rclone mount wasabi01:source.1000000 b:\rclone\mount\source.1000000 --read-only --dir-cache-time=5000m --rc --rc-no-auth --log-level=DEBUG --log-file=.\log.mount.source.1000000.txt
  2. run rc vfs/refresh recursive=true, took approx. 70 seconds
  3. now, rclone mount is now using over 4GiB memory.
  4. using windows explorer to view the folder, progress bar moves at a steady state, 50% complete in 30 seconds and then the progress bar slows down and seems to stop at 95%
  5. now, rclone mount is using 80% cpu, tho memory is less at 3GiB

seems that rclone is stuck or what?

and here is the debug log,
notice 2022/12/13 18:01:22 DEBUG : /: Readdir: ofst=0, fh=0x2
that has not changed in 30+ minutes.

2022/12/13 17:58:57 DEBUG : rclone: Version "v1.60.0" starting with parameters ["c:\\data\\rclone\\rclone.exe" "mount" "wasabi01:source.1000000" "b:\\rclone\\mount\\source.1000000" "--read-only" "--dir-cache-time=5000m" "--rc" "--rc-no-auth" "--log-level=DEBUG" "--log-file=.\\log.mount.source.1000000.txt"]
2022/12/13 17:58:57 NOTICE: Serving remote control on http://localhost:5572/
2022/12/13 17:58:57 DEBUG : Creating backend with remote "wasabi01:source.1000000"
2022/12/13 17:58:57 DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
2022/12/13 17:58:57 INFO  : S3 bucket source.1000000: poll-interval is not supported by this remote
2022/12/13 17:58:57 DEBUG : Network mode mounting is disabled
2022/12/13 17:58:57 DEBUG : Mounting on "b:\\rclone\\mount\\source.1000000" ("wasabi01 source.1000000")
2022/12/13 17:58:57 DEBUG : S3 bucket source.1000000: Mounting with options: ["-o" "attr_timeout=1" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=wasabi01 source.1000000"]
2022/12/13 17:58:57 DEBUG : S3 bucket source.1000000: Init: 
2022/12/13 17:58:57 DEBUG : S3 bucket source.1000000: >Init: 
2022/12/13 17:58:57 DEBUG : /: Statfs: 
2022/12/13 17:58:57 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2022/12/13 17:58:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 17:58:57 DEBUG : /: >Getattr: errc=0
2022/12/13 17:58:57 DEBUG : /: Readlink: 
2022/12/13 17:58:57 DEBUG : /: >Readlink: linkPath="", errc=-40
2022/12/13 17:58:57 DEBUG : /: Getxattr: name="non-existant-a11ec902d22f4ec49003af15282d3b00"
2022/12/13 17:58:57 DEBUG : /: >Getxattr: errc=-40, value=""
The service rclone has been started.
2022/12/13 17:58:57 DEBUG : /: Statfs: 
2022/12/13 17:58:57 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2022/12/13 17:58:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 17:58:57 DEBUG : /: >Getattr: errc=0
2022/12/13 17:58:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 17:58:57 DEBUG : /: >Getattr: errc=0
2022/12/13 17:58:57 DEBUG : /: Opendir: 
2022/12/13 17:58:57 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/13 17:58:57 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/13 17:58:57 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/12/13 17:58:57 DEBUG : /: Releasedir: fh=0x0
2022/12/13 17:58:57 DEBUG : /: >Releasedir: errc=0
2022/12/13 17:59:05 DEBUG : rc: "vfs/refresh": with parameters map[recursive:true]
2022/12/13 17:59:05 DEBUG : : Reading directory tree
2022/12/13 18:00:17 DEBUG : : Reading directory tree done in 1m11.9103984s
2022/12/13 18:00:17 DEBUG : rc: "vfs/refresh": reply map[result:map[:OK]]: <nil>
2022/12/13 18:01:21 DEBUG : /.sync/FolderType: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:21 DEBUG : /.sync/FolderType: >Getattr: errc=-2
2022/12/13 18:01:21 DEBUG : /.sync/ID: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:21 DEBUG : /.sync/ID: >Getattr: errc=-2
2022/12/13 18:01:21 DEBUG : /.sync/FolderType: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /desktop.ini: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /.sync/FolderType: >Getattr: errc=-2
2022/12/13 18:01:22 DEBUG : /.sync/ID: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /desktop.ini: >Getattr: errc=-2
2022/12/13 18:01:22 DEBUG : /.sync/ID: >Getattr: errc=-2
2022/12/13 18:01:22 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /: >Getattr: errc=0
2022/12/13 18:01:22 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /: >Getattr: errc=0
2022/12/13 18:01:22 DEBUG : /: Opendir: 
2022/12/13 18:01:22 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/13 18:01:22 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/13 18:01:22 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/12/13 18:01:22 DEBUG : /: Releasedir: fh=0x0
2022/12/13 18:01:22 DEBUG : /: >Releasedir: errc=0
2022/12/13 18:01:22 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /: >Getattr: errc=0
2022/12/13 18:01:22 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /: >Getattr: errc=0
2022/12/13 18:01:22 DEBUG : /: Opendir: 
2022/12/13 18:01:22 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/13 18:01:22 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/13 18:01:22 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/12/13 18:01:22 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /: >Getattr: errc=0
2022/12/13 18:01:22 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /: >Getattr: errc=0
2022/12/13 18:01:22 DEBUG : /: Opendir: 
2022/12/13 18:01:22 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/13 18:01:22 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/13 18:01:22 DEBUG : /: >Opendir: errc=0, fh=0x1
2022/12/13 18:01:22 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /: >Getattr: errc=0
2022/12/13 18:01:22 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/12/13 18:01:22 DEBUG : /: >Getattr: errc=0
2022/12/13 18:01:22 DEBUG : /: Opendir: 
2022/12/13 18:01:22 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/12/13 18:01:22 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/12/13 18:01:22 DEBUG : /: >Opendir: errc=0, fh=0x2
2022/12/13 18:01:22 DEBUG : /: Readdir: ofst=0, fh=0x2
10

Try adding --use-server-modtime here - that should stop rclone running a HEAD request for each object.

11

I only need to be able to access the files from a local path, would a sync/copy rather than a mount be more stable/usable?

12

yes, that would be more stable and usable.
https://forum.rclone.org/t/fastest-way-to-check-for-changes-in-2-5-million-files/25957/11

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.