From owner-freebsd-hackers@FreeBSD.ORG  Mon Apr  7 15:57:33 2014
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 4A3FF91F
 for <freebsd-hackers@freebsd.org>; Mon,  7 Apr 2014 15:57:33 +0000 (UTC)
Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1])
 (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 23A2470
 for <freebsd-hackers@freebsd.org>; Mon,  7 Apr 2014 15:57:33 +0000 (UTC)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
 by bigwig.baldwin.cx (Postfix) with ESMTPSA id 149ADB946;
 Mon,  7 Apr 2014 11:57:32 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: Karl Pielorz <kpielorz_lst@tdx.co.uk>
Subject: Re: Stuck CLOSED sockets / sshd / zombies...
Date: Mon, 7 Apr 2014 11:48:10 -0400
User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; )
References: <3FE645E9723756F22EF901AE@Mail-PC.tdx.co.uk>
 <201404041613.09808.jhb@freebsd.org>
 <2C5B099DE2229F0E8D82D8C8@Mail-PC.tdx.co.uk>
In-Reply-To: <2C5B099DE2229F0E8D82D8C8@Mail-PC.tdx.co.uk>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <201404071148.10157.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
 (bigwig.baldwin.cx); Mon, 07 Apr 2014 11:57:32 -0400 (EDT)
Cc: freebsd-hackers@freebsd.org
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 07 Apr 2014 15:57:33 -0000

On Monday, April 07, 2014 7:12:03 am Karl Pielorz wrote:
> 
> --On 04 April 2014 16:13 -0400 John Baldwin <jhb@freebsd.org> wrote:
> 
> > Ugh, ok.  Is this easy to reproduce?
> 
> Ok, yes - I can reproduce this now. I scanned the new host I setup with our 
> security scanning software.
> 
> This generated a number of sshd caught in 'urdlck' - and a large number of 
> sockets that end up as 'CLOSE_WAIT' I'm guessing given time these will 
> finally move to 'CLOSED' (it was scanned hours ago and there's still 50+ in 
> CLOSE_WAIT state).
> 
> As I said originally this can't be the only cause - but it is a cause.
> 
> So now I can reproduce it - what next?

Ok, do you have a matching /usr/src on the boxes in question?  If so, please 
do this:

cd /usr/src/lib/libc
make DEBUG_FLAGS=-g all install
cd /usr/src/lib/libthr
make DEBUG_FLAGS=-g all install
cd /usr/src/secure/lib/libssh
make DEBUG_FLAGS=-g all install
cd /usr/src/secure/usr.sbin/sshd
make DEBUG_FLAGS=-g all install

sh /etc/rc.d/sshd restart

Then re-run the scan to get a stuck sshd.  Once that happens, please
attach to the top-most stock sshd (the one in "urdlck") with gdb
(gdb /usr/sbin/sshd <pid>) and run 'bt' and reply with the output.

-- 
John Baldwin