From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 2 16:55:45 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9760E616; Wed, 2 Apr 2014 16:55:45 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 23D26B1F; Wed, 2 Apr 2014 16:55:44 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s32GthqA008751 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 2 Apr 2014 17:55:43 +0100 (BST) Date: Wed, 02 Apr 2014 17:55:43 +0100 From: Karl Pielorz To: John Baldwin , freebsd-hackers@freebsd.org Subject: Re: Stuck CLOSED sockets / sshd / zombies... Message-ID: In-Reply-To: <201404021130.39478.jhb@freebsd.org> References: <3FE645E9723756F22EF901AE@Mail-PC.tdx.co.uk> <201404021130.39478.jhb@freebsd.org> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Apr 2014 16:55:45 -0000 --On 2 April 2014 11:30:39 -0400 John Baldwin wrote: >> # ps ax | grep 4344 >> ps axl | grep 4344 >> 0 4344 895 0 20 0 84868 6944 urdlck Is - 0:00.01 sshd: >> unknown [priv] (sshd) > > Can you get 'procstat -k 4344' to see where this process is stuck? Sure, " # procstat -k 4344 PID TID COMM TDNAME KSTACK 4344 100068 sshd - mi_switch sleepq_catch_signals sleepq_wait_sig _sleep umtxq_sleep do_rw_rdlock __umtx_op_rw_rdlock amd64_syscall Xfast_syscall " >> 22 4345 4344 0 20 0 0 0 - Z - 0:00.00 >> 0 4346 4344 0 21 0 84868 6952 sbwait I - 0:00.00 sshd: >> unknown [pam] (sshd) > > 'procstat -f' and 'procstat -k' for this process might also be useful. Ok, think you mean PID 4346? " # procstat -f 4346 PID COMM FD T V FLAGS REF OFFSET PRO NAME 4346 sshd text v r r------- - - - /usr/sbin/sshd 4346 sshd cwd v d r------- - - - / 4346 sshd root v d r------- - - - / 4346 sshd 0 v c rw------ 6 0 - /dev/null 4346 sshd 1 v c rw------ 6 0 - /dev/null 4346 sshd 2 v c rw------ 6 0 - /dev/null 4346 sshd 3 s - rw---n-- 2 0 TCP 192.168.0.138:22 192.168.0.45:54588 4346 sshd 5 p - rw------ 2 0 - - 4346 sshd 6 s - rw------ 2 0 UDS - 4346 sshd 7 p - rw------ 1 0 - - 4346 sshd 8 s - rw------ 2 0 UDS - " " # procstat -k 4346 PID TID COMM TDNAME KSTACK 4346 100100 sshd - mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait soreceive_generic dofileread kern_readv sys_read amd64_syscall Xfast_syscall " In case it's relevant, one of the -xen guys originally said "It seems like the process is stuck while trying to acquire a rw mutex in read mode." [from when I thought it might be a -xen issue, which it's obviously not] If you want / need any more stuff running - let me know, -Karl