From owner-freebsd-stable@FreeBSD.ORG Thu May 3 21:47:27 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3326B16A40A; Thu, 3 May 2007 21:47:27 +0000 (UTC) (envelope-from scrappy@freebsd.org) Received: from hub.org (hub.org [200.46.204.220]) by mx1.freebsd.org (Postfix) with ESMTP id BFA8913C483; Thu, 3 May 2007 21:47:26 +0000 (UTC) (envelope-from scrappy@freebsd.org) Received: from localhost (unknown [200.46.204.191]) by hub.org (Postfix) with ESMTP id 2EA5248A322; Thu, 3 May 2007 18:47:16 -0300 (ADT) Received: from hub.org ([200.46.204.220]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 90077-03; Thu, 3 May 2007 18:47:25 -0300 (ADT) Received: from ganymede.hub.org (blk-89-241-126.eastlink.ca [24.89.241.126]) by hub.org (Postfix) with ESMTP id 8D93948A31F; Thu, 3 May 2007 18:47:15 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by ganymede.hub.org (Postfix) with ESMTP id 7D6E56038A; Thu, 3 May 2007 18:47:35 -0300 (ADT) Date: Thu, 03 May 2007 18:47:35 -0300 From: "Marc G. Fournier" To: Robert Watson Message-ID: <5C07C592A1A1F97E055965A6@ganymede.hub.org> X-Mailer: Mulberry/4.0.7 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: Adrian Chadd , freebsd-stable@freebsd.org Subject: Re: What triggers "No Buffer Space Available"? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 May 2007 21:47:27 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - --On Thursday, May 03, 2007 19:28:56 +0100 Robert Watson wrote: > I generally recommend using a combination of netstat and sockstat. Sockets > represent, loosely, IPC endpoints. There are actually two "layers" > associated with each socket -- the IPC object (socket) and the protocol > control block (PCB). Both are resource limited to pevent run-away processes > from swamping the system, so exhaustion of either can lead to ENOBUFS. > > The behaviors of netstat and sockstat are quite different, even though the > output is similar: netstat walks the protocol-layer connection lists and > prints information about them. sockstat walks the process file descriptor > table and prints information on reachable sockets. As sockets can exist > without PCBs, and PCBs can exist without sockets, you need to look at both to > get a full picture. This can occur if a proces exits, closes the socket, and > the connection remains in, for example, the TIME_WAIT state. > > There are some other differences -- the same socket can appear more than once > in sockstat output, as more than one process can reference the same socket. > Sockets can also exist without any referencing process (if the application > closes, but there is still data draining on an open socket). > > I would suggest starting with sockstat, as that will allow you to link socket > use to applications, and provide a fairly neat summary. When using netstat, > use "netstat -na", which will list all sockets and avoid name lookups. 'k, all I'm looking at right now is the Unix Domain Sockets, and the output of netstat -> sockstat is growing since I first started counting both .. This was shortly after reboot: mars# netstat -A | grep stream | wc -l ; sockstat -u | wc -l 2705 2981 - From your explanation above, I'm guessing that the higher sockstat #s is where you were talking about one socket being used by multiple processes? But, right now: mars# netstat -nA | grep stream | wc -l ; sockstat -u | wc -l 5025 2905 sockstat -u #s are *down*, but netstat -na is almost double ... Again, based on what you state above: "Sockets can also exist without any referencing process (if the application closes, but there is still data draining on an open socket)." Now, looking at another 6-STABLE server, but one that has been running for 2 months now, I'm seeing numbers more consistent with what mars looks like shortly after all the jails start up: venus# netstat -nA | grep stream | wc -l ; sockstat -u | wc -l 2126 2209 So, if those sockets on mars are 'still draining on an open socket', is there some way of finding out where? If I'm understanding what you've said above, these 'draining sockets' don't have any processes associated with them anymore? So, its not like I can just kill off a process, correct? - ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQFGOlh34QvfyHIvDvMRApSUAJ9jPszXBw83hXPRLbczimNWFtn6WwCgpijT nDWi/kW4Gt8/J2a4U3n2prk= =IQCW -----END PGP SIGNATURE-----