Fix an issue that invalid QUIC listener setting could casue segfault.