Everything works great. I can browse the files very quickly through the terminal. However, when it comes down to Finder, it takes forever to load the files.
I know it's trying to get metadata (something like that) from every file and folder, which may take a while. I can't remember the option to disable it, so I can browse the files faster through Finder.
Add logging to the output and watch them as you browse in Finder vs Terminal. Finder often tries to make nice thumbnails and, for that, it needs to read the files. At least some of them. That can be really slowing things down.
What Finder view are you using? I general prefer the Miller Columns but when I am on a slower connection (e.g. network share, especially over a VPN), I use list-view.
What is your backend for files:. I am assuming it is bucket-based since you have --fast-list. Is it S3? I seem to recall that S3 needs an additional call for mod-times. Does anyone know if that is per-file? If so, that could really be costing you speed if Finder needs them as per someone else's comment.
I have a vague memory that you can tell macOS that a given file system is a network file system which might make it behave more sensibly. I can't remember how you do that though!