From owner-freebsd-hackers@FreeBSD.ORG Thu Jul 10 19:31:09 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D17B99FD for ; Thu, 10 Jul 2014 19:31:09 +0000 (UTC) Received: from secure.xzibition.com (secure.xzibition.com [173.160.118.92]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6ADF12C4E for ; Thu, 10 Jul 2014 19:31:08 +0000 (UTC) DomainKey-Signature: a=rsa-sha1; c=nofws; d=shatow.net; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=sweb; b=NPWsCU 0JH2SuFnTLBxSH9jJJhzYK9xxYCaXiAdHV+fMipxiFfIIKB7nHPeWlLKPFT57Dz7 IlDPyClSXfVgjpC2By9NYXGVeCb2Xz0JYCp88ZT4kMhJGLrPD4M9itcyC1VVmvGj 9M/QWeu8mHgtz7suLB17QQMlpFi8pSFCKZhjk= DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=shatow.net; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=sweb; bh=QsAGgrRAd3sF GC/J9NmN8YA+T1fIDg/taOP7PcG8IhI=; b=qIdxtr1gGca32oAGjGBwC4kau9t6 H1wFSipz1Bgh4ZAeV/oa3h1mf+4tDr16r3BodK5kkzIXZlz52GogTRYYtiYU8ks7 4kB67oYfz5m6oIX7hN/xDge0cZgB1U0fOVHI5UvQJlR1F7fOg/KFKzYcjZsJdgT9 U/gb76fw7MwDlTs= Received: (qmail 4700 invoked from network); 10 Jul 2014 14:24:25 -0500 Received: from unknown (HELO ?10.10.0.24?) (bryan@shatow.net@10.10.0.24) by sweb.xzibition.com with ESMTPA; 10 Jul 2014 14:24:25 -0500 Message-ID: <53BEE861.1020603@shatow.net> Date: Thu, 10 Jul 2014 14:24:17 -0500 From: Bryan Drewery User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: kldstat / kernel linker deadlock References: <50AED0B9.7040108@shatow.net> <201301111613.38676.jhb@freebsd.org> In-Reply-To: <201301111613.38676.jhb@freebsd.org> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 19:31:10 -0000 On 1/11/2013 3:13 PM, John Baldwin wrote: > On Thursday, November 22, 2012 08:26:17 PM Bryan Drewery wrote: >> On 8.3-RELEASE I've hit a deadlock with kldstat. >> >> I can't provide much information as procstat(1) locks up and I have >> already rebooted the servers due to it breaking quite a bit in my setup. >> >>> # kldstat >>> Id Refs Address Size Name >>> load: 0.91 cmd: kldstat 9936 [kernel linker] 51.21r 0.00u 0.00s 0% 768k >>> ^C >>> load: 0.72 cmd: kldstat 9936 [kernel linker] 225.23r 0.00u 0.00s 0% 704k >>> load: 0.72 cmd: kldstat 9936 [kernel linker] 225.39r 0.00u 0.00s 0% 704k >>> load: 0.42 cmd: kldstat 9936 [kernel linker] 1837.24r 0.00u 0.00s 0% >>> 692k >> >> Short list of affected processes (74 in all): >>> root 3685 0.0 0.0 3264 700 ?? D 7:27PM 0:00.00 >>> kldstat root 67061 0.0 0.0 3380 892 ?? D 7:27PM >>> 0:00.00 /usr/bin/netstat -nrf inet root 5579 0.0 0.0 3380 >>> 892 ?? D 7:37PM 0:00.00 /usr/bin/netstat -nrf inet root >>> 6393 0.0 0.0 3264 704 ?? D 7:32PM 0:00.00 /sbin/kldstat -v >>> root 99635 0.0 0.1 3324 1244 13 D+ 7:52PM 0:00.01 >>> procstat -ka >> >> [... 69 more removed ...] >> >> I had 2 minutely cron entries that were running kldstat(1)/netstat(1). >> >> Guessing the kldstat(1) and netstat(1) deadlocked initially. > > Next time get a dump if at all possible. > Pretty sure this was fixed in r224546 which did not get MFC'd to releng/8.3 before release. > r224546 | glebius | 2011-07-31 08:49:15 -0500 (Sun, 31 Jul 2011) | 4 lines > Changed paths: > M /head/sys/kern/kern_linker.c > > Don't leak kld_sx lock in kldunloadf(). > @@ -1108,12 +1108,13 @@ kern_kldunload(struct thread *td, int fi > #ifdef HWPMC_HOOKS > if (error == 0) { > KLD_DOWNGRADE(); > PMC_CALL_HOOK(td, PMC_FN_KLD_UNLOAD, (void *) &pkm); > KLD_UNLOCK_READ(); > } else > + KLD_UNLOCK(); > #else > KLD_UNLOCK(); > #endif I reviewed my logs from the day and found that the initial command I ran was 'kldunload linux' due to SA-12:08.linux.asc. The module was in use due to running processes using the linuxelf brand so the module had returned EBUSY. I also had HWPMC_HOOKS since it was in GENERIC. When EBUSY was returned the kld lock was kept xlocked. -- Regards, Bryan Drewery