From owner-freebsd-xen@FreeBSD.ORG Tue Mar 18 08:44:33 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 89048ECA for ; Tue, 18 Mar 2014 08:44:33 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BF56ECB5 for ; Tue, 18 Mar 2014 08:44:32 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.97,676,1389744000"; d="scan'208";a="112389170" Received: from accessns.citrite.net (HELO FTLPEX01CL02.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 18 Mar 2014 08:44:29 +0000 Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.79) with Microsoft SMTP Server id 14.2.342.4; Tue, 18 Mar 2014 04:44:28 -0400 Message-ID: <5328076C.9000304@citrix.com> Date: Tue, 18 Mar 2014 09:44:28 +0100 From: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Karl Pielorz , Subject: Re: FBSD 10.0-S (r261289M) under XenServer 6.2 - Stuck sshd in urdlck? References: <4B8380EBE379080FAD3271FA@Mail-PC.tdx.co.uk> <53272E41.3050409@citrix.com> <6534613783544E26A15CF44F@study64.tdx.co.uk> In-Reply-To: <6534613783544E26A15CF44F@study64.tdx.co.uk> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2014 08:44:33 -0000 On 17/03/14 19:44, Karl Pielorz wrote: > > > --On 17 March 2014 18:17:53 +0100 Roger Pau Monné > wrote: > >>> Anyone know what 'urdlck' is? >> >> It seems like the process is stuck while trying to acquire a rw mutex in >> read mode. Could you obtain a backtrace of the process with gdb? > > Ok, I think I did this right - let me know if I've not... > > # gdb /usr/sbin/sshd 5325 > ... > Attaching to program: /usr/sbin/sshd, process 5325 > > warning: current_sos: Can't read pathname for load map: Bad address > [repeated several times] > [lots of reading symbols from - 'no debugging symbols found' output] > ... > [New Thread 804006400 (LWP 100184/sshd)] > [a few reading symbols - 'no debugging symbols found' output] > Loaded symbols for /libexec/ld-elf.so.1 > [Switching to Thread 804006400 (LWP 100184/sshd)] > 0x00000008038eb89c in __error () from /lib/libthr.so.3 > (gdb) bt > #0 0x00000008038eb89c in __error () from /lib/libthr.so.3 > #1 0x00000008038e921c in pthread_timedjoin_np () from /lib/libthr.so.3 > #2 0x000000080064f9a2 in _rtld_get_stack_prot () from /libexec/ld-elf.so.1 > #3 0x00000008006498c9 in r_debug_state () from /libexec/ld-elf.so.1 > #4 0x00000008006470cd in .text () from /libexec/ld-elf.so.1 > #5 0x0000000000000246 in ?? () > #6 0x0000000000000000 in ?? () > " > >> Also a >> kernel-space dump might be useful, could you also run procstat -k ? > > procstat output is: > > " > # procstat -k 5334 > PID TID COMM TDNAME KSTACK > 5334 100183 sshd - mi_switch > sleepq_catch_signals sleepq_wait_sig _sleep umtxq_sleep do_rw_rdlock > __umtx_op_rw_rdlock amd64_syscall Xfast_syscall > " > > If you can briefly tell me how to do the kernel-space dump? Do I panic > the machine (i.e. cause a crash-dump?) somehow? The output of vmstat -ai might also be helpful to assure that event timers are working correctly. Also, does this VM have some PCI-passthrough? Did you migrate it? The xl configuration file used to create the domain would also be interesting. Also, could you try the same workload with a pristine GENERIC kernel? Roger.