From owner-freebsd-stable@FreeBSD.ORG Tue Oct 30 13:29:52 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5E3CDB5A for ; Tue, 30 Oct 2012 13:29:52 +0000 (UTC) (envelope-from adams-freebsd@ateamsystems.com) Received: from fss.sandiego.ateamservers.com (fss.sandiego.ateamservers.com [69.55.229.149]) by mx1.freebsd.org (Postfix) with ESMTP id 41E768FC20 for ; Tue, 30 Oct 2012 13:29:51 +0000 (UTC) Received: from [192.168.15.220] (gw.digitalspark.net [118.175.84.92]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by fss.sandiego.ateamservers.com (Postfix) with ESMTPSA id 81B9CB9F24 for ; Tue, 30 Oct 2012 09:21:50 -0400 (EDT) Message-ID: <508FD46D.3040608@ateamsystems.com> Date: Tue, 30 Oct 2012 20:21:49 +0700 From: Adam Strohl User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: FreeBSD-Stable ML Subject: No buffer space available / tcp_inpcb value Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Oct 2012 13:29:52 -0000 Hey -STABLE, I've got a client who we've setup a FreeBSD cluster for with about a dozens servers, all behind two front end proxies/LBs/firewalls which also act as NAT gateways for the internal servers. On the active front end proxy we've started seeing "fatal: socket: No buffer space available" errors during high-peak times. I can see in vmstat -z that this is what is getting denied: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP tcp_inpcb: 392, 32770, 19398, 13372,1449734621,6312858, 0 We've got a lot of the other values bumped, and it appears to be this input limit that is getting hit. There are no other non-zero FAILed counters except 64 and 128 buckets which I believe are normal. I cannot seem to find the sysctl (or equiv) that controls this limit though, or even what it is. Anyone know? I'm obviously in need of this specific answer, but overall is there a codex of vmstat -z's items that explains this that I have just not found in my searches? This isn't the first time I've had to dig into a value like this to increase it's limit, but this time I'm not turning anything up. Any thoughts/ideas appreciated! -- Adam Strohl http://www.ateamsystems.com/