So it's possible to listen on a port (e.g. 8081) on all IPv6 addresses, but it's not possible to listen on a port on all IPv4 addresses.
I think that is correct.
It could be made configurable - if there are enough good reasons to warrant it, a feature request is made, and someone picks it up...
The network must be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
For TCP networks, if the host in the address parameter is empty or a literal unspecified IP address, Listen listens on all available unicast and anycast IP addresses of the local system. To only use IPv4, use network "tcp4"
(net package - net - Go Packages)