Add logging to the output and watch them as you browse in Finder vs Terminal. Finder often tries to make nice thumbnails and, for that, it needs to read the files. At least some of them. That can be really slowing things down.
What Finder view are you using? I general prefer the Miller Columns but when I am on a slower connection (e.g. network share, especially over a VPN), I use list-view.
What is your backend for files:. I am assuming it is bucket-based since you have --fast-list. Is it S3? I seem to recall that S3 needs an additional call for mod-times. Does anyone know if that is per-file? If so, that could really be costing you speed if Finder needs them as per someone else's comment.