From owner-freebsd-hackers@FreeBSD.ORG Fri Apr 11 13:50:25 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4CCF7309; Fri, 11 Apr 2014 13:50:25 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id DAF8C1C67; Fri, 11 Apr 2014 13:50:24 +0000 (UTC) Received: from Mail-PC.tdx.co.uk (storm.tdx.co.uk [62.13.130.251]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s3BDoNqs034431 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 11 Apr 2014 14:50:23 +0100 (BST) Date: Fri, 11 Apr 2014 14:50:22 +0100 From: Karl Pielorz To: Konstantin Belousov Subject: Re: Stuck CLOSED sockets / sshd / zombies... Message-ID: <652B8CA4866C0B9E4650430B@Mail-PC.tdx.co.uk> In-Reply-To: <20140411131649.GR21331@kib.kiev.ua> References: <201404081533.53990.jhb@freebsd.org> <92366925229B4C5B21B04D81@study64.tdx.co.uk> <20140408212319.GC21331@kib.kiev.ua> <20140409084951.GE21331@kib.kiev.ua> <2A722BB3B12E0D80CA9FF075@Mail-PC.tdx.co.uk> <20140409111917.GH21331@kib.kiev.ua> <851413886E3982D2CCFEA9D9@Mail-PC.tdx.co.uk> <20140410184855.GP21331@kib.kiev.ua> <211BD03C086DDB1A07FDF036@Mail-PC.tdx.co.uk> <20140411131649.GR21331@kib.kiev.ua> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2014 13:50:25 -0000 --On 11 April 2014 16:16 +0300 Konstantin Belousov wrote: > On Fri, Apr 11, 2014 at 01:39:54PM +0100, Karl Pielorz wrote: >> >> Ok, rebuilt a debug world (with your rtld-elf patch), installed it - >> reproduced the issue, and ran up gdb on a 'urdlck' stuck sshd, and got >> the trace below. > The trace looks reasonable. Great :) > I vaguelly remember that you already answered this, but I want to start > investigating from the different angle. Please show me the output > of 'ldd /usr/sbin/sshd' on your machine. This happens on stable/10, > right ? " ldd /usr/sbin/sshd /usr/sbin/sshd: libssh.so.5 => /usr/lib/private/libssh.so.5 (0x800860000) libutil.so.9 => /lib/libutil.so.9 (0x800abb000) libwrap.so.6 => /usr/lib/libwrap.so.6 (0x800ccd000) libpam.so.5 => /usr/lib/libpam.so.5 (0x800ed6000) libbsm.so.3 => /usr/lib/libbsm.so.3 (0x8010e2000) libgssapi_krb5.so.10 => /usr/lib/libgssapi_krb5.so.10 (0x8012fc000) libgssapi.so.10 => /usr/lib/libgssapi.so.10 (0x80151a000) libkrb5.so.11 => /usr/lib/libkrb5.so.11 (0x801723000) libhx509.so.11 => /usr/lib/libhx509.so.11 (0x801999000) libasn1.so.11 => /usr/lib/libasn1.so.11 (0x801be1000) libcom_err.so.5 => /usr/lib/libcom_err.so.5 (0x801e7a000) libroken.so.11 => /usr/lib/libroken.so.11 (0x80207c000) libwind.so.11 => /usr/lib/libwind.so.11 (0x80228d000) libheimbase.so.11 => /usr/lib/libheimbase.so.11 (0x8024b5000) libheimipcc.so.11 => /usr/lib/private/libheimipcc.so.11 (0x8026b9000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x8028bb000) libcrypto.so.7 => /lib/libcrypto.so.7 (0x802adb000) libz.so.6 => /lib/libz.so.6 (0x802ec6000) libc.so.7 => /lib/libc.so.7 (0x8030db000) libldns.so.5 => /usr/lib/private/libldns.so.5 (0x803474000) libmd.so.6 => /lib/libmd.so.6 (0x8036c8000) libthr.so.3 => /lib/libthr.so.3 (0x8038d8000) " The box is stable/10 - quite an old stable 10 now, but afaik other people have hit a similar issue on newer stable 10's - I've not updated this box, as I've seen nothing to say it's "fixed" in newer versions [and it's obviously been under investigation for weeks now on this machine as well, long before I posted to -hackers]. I can update to a newer version (e.g. today) if you want. > I do not see any linking with libpthread in the sshd Makefile. Could it > be that libthr is loaded as dependency of some pam module ? Possibly - I don't know. This is stock FreeBSD #10 Stable - i.e. I've not configured anything differently on SSH than what you get 'out the box'. I've never done anything with PAM - so I don't know where I'd go checking that kind of thing (but can if you point me in the right direction). -Karl