Try setting it to 0 but vbulletin support might be best to help you with that.
setting max connections your not doing in PP you do that to your server. You can ask your host to raise levels.
My suggestion is set the nix thing in
vb to 0. Then you dont get any messages. If your server becomes slow etc then you go to your host and say fix this.
You can contact your host and say your message came from vbulletin because you have a setting to give that message if your NIX level is 5 or over. Ask them what they can do to lighten the load on the server.