We run private ipv6 addresses (fe00 space)
FE00::/9 is reserved
. You are supposed to be using RFC 4193 addresses. For example you could use fd6d:6259:b5e8::/48.
If you use proper addresses, then browsers will have a chance of realizing that IPv6 can only be used to connect to hosts on the LAN and not for hosts on the Internet. In other words, if the only (non-link-local) address a host has is an RFC 4193 address, then a sensible implementation would only default to IPv6 if the server has an address within the same /48.
There is no guarantee that switching to proper RFC 4193 addresses will help, but it is certainly better than using reserved addresses (which the browser probably treats the same way as public addresses from 2000::/3).
Using more recent browsers may also help. Supposedly recent browsers are better at dealing with broken networks, such as yours. A common strategy is to only wait a few hundred milliseconds from starting the first connection before trying the other protocol. Then the actual request is sent over the connection which first completes a handshake.