From owner-freebsd-stable@FreeBSD.ORG Thu Dec 25 08:22:07 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D197048D; Thu, 25 Dec 2014 08:22:07 +0000 (UTC) Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by mx1.freebsd.org (Postfix) with ESMTP id B573712E0; Thu, 25 Dec 2014 08:22:06 +0000 (UTC) Received: from ppp14-2-30-215.lns21.adl2.internode.on.net (HELO leader.local) ([14.2.30.215]) by ipmail06.adl2.internode.on.net with ESMTP; 25 Dec 2014 18:51:58 +1030 Message-ID: <549BC924.3050402@ShaneWare.Biz> Date: Thu, 25 Dec 2014 18:51:56 +1030 From: Shane Ambler User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: John Baldwin , freebsd-stable@freebsd.org Subject: Re: Help debugging stable/10 References: <5488F58D.7060708@ShaneWare.Biz> <201412161129.57704.jhb@freebsd.org> In-Reply-To: <201412161129.57704.jhb@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Alexander Motin , hselasky@freebsd.org, mjg@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Dec 2014 08:22:07 -0000 On 17/12/2014 02:59, John Baldwin wrote: > On Wednesday, December 10, 2014 8:38:21 pm Shane Ambler wrote: >> Since upgrading to 10.1 (RC2) I have had trouble getting uptimes greater >> than 1 day. I have little experience debugging the OS so could use some >> help. >> > It looks like your processes are hanging on a global lock used by sysctl(3). > That would explain hangs in top/ps/procstat as they all use sysctls. For > example: > Hi again, I just had a usb failure while still able to start processes so have a bit more info. It took approximately 45 minutes before other things locked up and I was unable to create new processes, I couldn't gather any more info at that time. http://shaneware.biz/freebsddebugdata/procstat-2014-12-25-15-52 http://shaneware.biz/freebsddebugdata/kgdb.output-2014-12-25-15-52 FreeBSD leader.local 10.1-STABLE FreeBSD 10.1-STABLE #1 r276101: Tue Dec 23 16:19:53 ACDT 2014 root@leader.local:/usr/obj/usr/src/sys/GENERIC amd64 Shortly before the time of the procstat and kgdb output I inserted a usb memstick and it failed to create the device entries, I removed and reinserted the stick and the detach event showed in log/messages but not the re-insertion. I tried inserting two different branded memsticks in two physical slots and both failed to show insert events. I was running poudriere at the time but stopped that before logging the data. I see 3 entries in procstat that have _sx_xlock_hard - 6 100058 g_journal switcher g_journal switch mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d g_journal_switcher+0x1ca fork_exit+0x9a fork_trampoline+0xe 13 100015 geom g_event mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a g_run_events+0x82 fork_exit+0x9a fork_trampoline+0xe 16 100073 pagedaemon - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d g_journal_lowmem+0xa3 vm_pageout+0x2bf fork_exit+0x9a fork_trampoline+0xe -- FreeBSD - the place to B...Software Developing Shane Ambler