From owner-freebsd-hackers Sun Jan 26 13:41:27 2003 Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DEE3437B401 for ; Sun, 26 Jan 2003 13:41:25 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4C77543EB2 for ; Sun, 26 Jan 2003 13:41:25 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.12.6/8.12.5) with SMTP id h0QLe7P4003907; Sun, 26 Jan 2003 16:40:08 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Sun, 26 Jan 2003 16:40:07 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Sam Tannous Cc: freebsd-hackers@freebsd.org Subject: Re: max simultaneous TCP connections (32,763)? In-Reply-To: <20030126195657.GA14704@cisco.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, 26 Jan 2003, Sam Tannous wrote: > I have two freebsd boxes (back to back) and I've been playing with a > simple server on one machine and client on the other machine (this was > simply an exercise with playing with kqueue). Both the server and the > client are single processes and the client seems to stop at 32,763 > connections. > > I've modified the port range, tcp keepalive, kern.ipc.somaxconn, > maxfiles, maxsockets, nmbclusters. I even tried > net.inet.tcp.tcbhashsize (up to 1024). > > Is there some other parameter I'm missing? Or is this a known > limitation/bug? Some of this has to do with limits on the available ancillary ports for out-going connections. Try adding additional IP addresses to the client machine, and forcing your client software to use specific IP addresses. TCP uniquely identifies connections by the pair of port numbers and IP addresses, so assuming unconstrained use of the outgoing port space on a particular IP, that TCP/IP can in theory support up to (approx) 64k outgoing connections. In practice, we only allocate out of specific ranges. By adding additional IP addresses for outgoing connections, you increase the number of potential connections to a particular remote IP/port tuple. However, if you're not specifying a local IP address, the stack will pick "the most appropriate" local address for the route, which is probably the first IP address on the interface associated with the route to the other endpoint. Hard-coding local addreses in your application overrides that. I've never tried this (i.e., using multiple IPs to get around the TCP/IP limit), so if it doesn't work, let me know. In theory, it should. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message