From owner-freebsd-net@FreeBSD.ORG Thu Nov 5 08:30:04 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE1781065670 for ; Thu, 5 Nov 2009 08:30:03 +0000 (UTC) (envelope-from is@rambler-co.ru) Received: from mailrelay1.rambler.ru (mailrelay1.rambler.ru [81.19.66.239]) by mx1.freebsd.org (Postfix) with ESMTP id 61AFC8FC12 for ; Thu, 5 Nov 2009 08:30:03 +0000 (UTC) Received: from kas30pipe.localhost (localhost [127.0.0.1]) by mailrelay1.rambler.ru (Postfix) with ESMTP id 1EA93130CB7 for ; Thu, 5 Nov 2009 11:30:02 +0300 (MSK) Received: from localhost (sysoev.ru [81.19.68.137]) by mailrelay1.rambler.ru (Postfix) with ESMTP id 7F990130C16 for ; Thu, 5 Nov 2009 11:30:01 +0300 (MSK) Date: Thu, 5 Nov 2009 11:30:01 +0300 From: Igor Sysoev To: freebsd-net@freebsd.org Message-ID: <20091105083001.GC87654@rambler-co.ru> References: <20070820151142.GA20183@rambler-co.ru> <46C9BF02.5050007@tomjudge.com> <20070820163443.GE20183@rambler-co.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <20070820163443.GE20183@rambler-co.ru> User-Agent: Mutt/1.5.20 (2009-06-14) X-Anti-Virus: Kaspersky Anti-Virus for MailServers 5.5.33/RELEASE, bases: 02092009 #2738642, status: clean X-SpamTest-Envelope-From: is@rambler-co.ru X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 9536 [Sen 02 2009] X-SpamTest-Info: {HEADERS: header Content-Type found without required header Content-Transfer-Encoding} X-SpamTest-Method: none X-SpamTest-Rate: 10 X-SpamTest-SPF: pass X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0284], KAS30/Release Subject: Re: maximum number of outgoing connections X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2009 08:30:04 -0000 On Mon, Aug 20, 2007 at 08:34:43PM +0400, Igor Sysoev wrote: > On Mon, Aug 20, 2007 at 05:19:14PM +0100, Tom Judge wrote: > > > Igor Sysoev wrote: > > >It seems that FreeBSD can not make more than > > > > > >net.inet.ip.portrange.last - net.inet.ip.portrange.first > > > > > >simultaneous outgoing connections, i.e., no more than about 64k. > > > > > >If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then > > >connect() to an external address returns EADDRNOTAVAIL. > > > > > >net.inet.ip.portrange.randomized is 0. > > > > > >sockets, etc. are enough: > > > > > >ITEM SIZE LIMIT USED FREE REQUESTS FAILURES > > >socket: 356, 204809, 13915, 146443, 148189452, 0 > > >inpcb: 180, 204820, 20375, 137277, 147631805, 0 > > >tcpcb: 464, 204800, 13882, 142102, 147631805, 0 > > >tcptw: 48, 41028, 6493, 11213, 29804665, 0 > > > > > >I saw it on 6.2-STABLE. > > > > > > > > > > In an ideal world (Not sure if this is quite correct for FreeBSD) TCP > > connections are tracked with a pair of tupels source-addr:src-port -> > > dst-addr:dst-port > > > > As your always connecting to the same destination service 127.0.0.1:80 > > and always from the same source IP 127.0.0.1 then you only have one > > variable left to change, the source port. If you where to use the hole > > of the whole of the port range minus the reserved ports you would only > > ever be able to make 64512 simultaneous connections. In order to make > > more connections the first thing that you may want to start changing is > > the source IP. If you added a second IP to you lo0 interface (say > > 127.0.0.2) and used a round robin approach to making your out bound > > connections then you could make around 129k outbound connections. > > Connections to 127.0.0.1 were via lo0, external connections are via bge0. > > > I am not sure if there are any other constraints that need to be taken > > into account such as the maximum number of sockets, RAM etc.... > > No, there are no constraints in memory, sockets, mbufs, clusters, etc. > If there's contraint in memory, then FreeBSD simply panics. > If there's contraint in mbuf clusters, then process stucks in zonelimit > state forever. > > I suspect that local address in in_pcbbind_setup() is 0.0.0.0 so there > is 64K limit. Recently I looked the issue again and find (with Ruslan Ermilov's help) that if I set the SO_REUSEADDR option, FreeBSD allows to use the same local port for different connections: 192.168.1.1:5000 > 10.0.0.1:80 192.168.1.1:5000 > 10.0.0.2:80 Linux allows this by default. I believe FreeBSD should set SO_REUSEADDR internally while connect(). BTW, bind()ing socket to a local address does not help. -- Igor Sysoev http://sysoev.ru/en/