From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 16 03:43:54 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AF2DA1065672; Thu, 16 Feb 2012 03:43:54 +0000 (UTC) (envelope-from vijju.singh@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4CFCD8FC18; Thu, 16 Feb 2012 03:43:54 +0000 (UTC) Received: by vbbfa15 with SMTP id fa15so1781504vbb.13 for ; Wed, 15 Feb 2012 19:43:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=xTIMqZ5Kybd4h9wRK2S16jDh2VdYlzRsh8yrJTX2Qfg=; b=UCQKsxe054jat1AMbm7gRE5OWcdB/+S7wviLFm1Ky+dRFRt4bL9d1B1uZXNF/GzTLl PuvSXtGuDZI3aAr2K3yJsbljulWs6IdgOvq4O89AIB4zjszs10a2/0f8hWYJSAQJ0b8z x3eOdPyIv+Snj95FHY+Kh7g5kJHa9Zz6cUNr8= MIME-Version: 1.0 Received: by 10.52.34.4 with SMTP id v4mr283921vdi.85.1329362435240; Wed, 15 Feb 2012 19:20:35 -0800 (PST) Received: by 10.220.179.131 with HTTP; Wed, 15 Feb 2012 19:20:35 -0800 (PST) In-Reply-To: <201111081005.43851.jhb@freebsd.org> References: <201111081005.43851.jhb@freebsd.org> Date: Wed, 15 Feb 2012 19:20:35 -0800 Message-ID: From: Vijay Singh To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Panic in 8.1 in softclock(): mutex Giant not owned at ../../../../sys/kern/kern_mutex.c:185 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2012 03:43:54 -0000 Hi John, >> (kgdb-amd64-6.8-71) f 5 >> #5 =A00xffffffff803b73e6 in softclock (arg=3D0xffffffff80c90f40) at >> ../../../../sys/kern/kern_timeout.c:461 >> 461 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 class->lc_unlock(c_lock); >> (kgdb-amd64-6.8-71) l >> 456 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 lastfunc =3D c_func; >> 457 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = } >> 458 =A0 #endif >> 459 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = CTR1(KTR_CALLOUT, "callout %p finished", c); >> 460 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = if ((c_flags & CALLOUT_RETURNUNLOCKED) =3D=3D 0) >> 461 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 class->lc_unlock(c_lock); >> 462 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 skip: >> 463 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = CC_LOCK(cc); >> 464 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = /* >> 465 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0* If the current callout is locally >> (kgdb-amd64-6.8-71) i local >> c_func =3D (void (*)(void *)) 0xffffffff8060d550 > > You'll need to see if there is a way that this routine can drop the lock.= =A0If > you can reproduce this panic easily, then perhaps try building with KTR a= nd > KTR_LOCK traces enabled and use 'show ktr' in ddb to see when this thread > dropped Giant. An update. I haven't been able to enable KTR in my environment. However I found some interesting. In the code below: (kgdb-amd64-6.8-48) l 402 if (c_lock !=3D NULL) { 403 if (c_lock =3D=3D &Giant.lock_object) { 404 mtx_lock(&Giant); 405 mtx_assert(&Giant, MA_OWNED); 406 } else { 407 class->lc_lock(c_lock, sharedlock); 408 } I hit the assert in line 405. So right after locking Giant, it is found unlocked!!! Everytime this hits, I see that kldload is running on the other CPU: (kgdb-amd64-6.8-48) show_ps_running CPU ticks pri thread tid pid state VSZ RSS cmd 0 2 52 0xffffff000424c800 100006 11 TDS_RUNNING 0K 0K [intr] 1 3 76 0xffffff0004618400 100065 1481 TDS_RUNNING 3808K 760K kldload Reporting it here to see if someone has seen this before. -vijay