From owner-freebsd-stable@FreeBSD.ORG Fri May 4 09:49:20 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D8F5C16A400; Fri, 4 May 2007 09:49:20 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id AD73313C43E; Fri, 4 May 2007 09:49:20 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 36FDD4729A; Fri, 4 May 2007 05:49:20 -0400 (EDT) Date: Fri, 4 May 2007 10:49:20 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "Marc G. Fournier" In-Reply-To: <5C07C592A1A1F97E055965A6@ganymede.hub.org> Message-ID: <20070504104426.E30345@fledge.watson.org> References: <5C07C592A1A1F97E055965A6@ganymede.hub.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Adrian Chadd , freebsd-stable@freebsd.org Subject: Re: What triggers "No Buffer Space Available"? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 May 2007 09:49:20 -0000 On Thu, 3 May 2007, Marc G. Fournier wrote: > 'k, all I'm looking at right now is the Unix Domain Sockets, and the output > of netstat -> sockstat is growing since I first started counting both .. > > This was shortly after reboot: > > mars# netstat -A | grep stream | wc -l ; sockstat -u | wc -l > 2705 > 2981 > > - From your explanation above, I'm guessing that the higher sockstat #s is where > you were talking about one socket being used by multiple processes? But, right > now: > > mars# netstat -nA | grep stream | wc -l ; sockstat -u | wc -l > 5025 > 2905 > > sockstat -u #s are *down*, but netstat -na is almost double ... > > Again, based on what you state above: "Sockets can also exist without any > referencing process (if the application closes, but there is still data > draining on an open socket)." > > Now, looking at another 6-STABLE server, but one that has been running for 2 > months now, I'm seeing numbers more consistent with what mars looks like > shortly after all the jails start up: > > venus# netstat -nA | grep stream | wc -l ; sockstat -u | wc -l > 2126 > 2209 > > So, if those sockets on mars are 'still draining on an open socket', is > there some way of finding out where? If I'm understanding what you've said > above, these 'draining sockets' don't have any processes associated with > them anymore? So, its not like I can just kill off a process, correct? The draining state cannot occur for UNIX domain sockets. The only cases I know of in which UNIX domain sockets can have PCBs without a process connection is if the UNIX domain socket is attached to a socket that is being passed over another socket where the original socket has released all other references to it, and in using FIFOs. The former is a relatively rare occurence with almost all applications, since very few use explicit file descriptor passing. Is there any chance that any of your applications is using a large number of POSIX FIFOs? BTW, when using sockstat as above, you need to sockstat -u | grep -c stream, for the same reason you do it with netstat. Datagram UNIX domain sockets are quite frequently used -- for example, with syslog, so need to be omitted from the count if you are comparing only stream sockets. Robert N M Watson Computer Laboratory University of Cambridge