From owner-freebsd-stable@FreeBSD.ORG Thu Nov 16 10:24:39 2006 Return-Path: X-Original-To: stable@FreeBSD.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2BFAD16A403 for ; Thu, 16 Nov 2006 10:24:39 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.sick.ru (cell.sick.ru [217.72.144.68]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7F2DA43D58 for ; Thu, 16 Nov 2006 10:24:38 +0000 (GMT) (envelope-from glebius@FreeBSD.org) Received: from cell.sick.ru (glebius@localhost [127.0.0.1]) by cell.sick.ru (8.13.4/8.13.3) with ESMTP id kAGAOaBO093724 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 16 Nov 2006 13:24:37 +0300 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.sick.ru (8.13.4/8.13.1/Submit) id kAGAOaYj093723 for stable@freebsd.org; Thu, 16 Nov 2006 13:24:36 +0300 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.sick.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Thu, 16 Nov 2006 13:24:36 +0300 From: Gleb Smirnoff To: stable@FreeBSD.org Message-ID: <20061116102436.GN32700@FreeBSD.org> References: <20061113084430.GE59604@dimma.mow.oilspace.com> Mime-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <20061113084430.GE59604@dimma.mow.oilspace.com> User-Agent: Mutt/1.5.6i Cc: Subject: Re: RELENG_6 panic under heavy load X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Nov 2006 10:24:39 -0000 I wonder why UMA was suspected to be the problem. Dima gave me access to the core. Here are more details from the trace: Unread portion of the kernel message buffer: panic: thread 100147(nagios):1 holds process lock but isn't blocked on a lock #9 0xd060038e in panic (fmt=0xd08094d9 "thread %d(%s):%d holds %s but isn't blocked on a lock\n") at /usr/src/sys/kern/kern_shutdown.c:549 #10 0xd0629228 in propagate_priority (td=0xd745c900) at /usr/src/sys/kern/subr_turnstile.c:239 #11 0xd0629f32 in turnstile_wait (lock=0xd5dd5498, owner=0xd745c900) at /usr/src/sys/kern/subr_turnstile.c:643 #12 0xd05f4fc1 in _mtx_lock_sleep (m=0xd5dd5498, tid=3583683968, opts=0, file=0x12
, line=18) at /usr/src/sys/kern/kern_mutex.c:579 #13 0xd05f4992 in _mtx_lock_flags (m=0xd5dd5498, opts=0, file=0xd0806c3d "/usr/src/sys/kern/kern_thread.c", line=824) at /usr/src/sys/kern/kern_mutex.c:288 #14 0xd060d340 in thread_single (mode=0) at /usr/src/sys/kern/kern_thread.c:824 #15 0xd05e38b9 in fork1 (td=0xd59aad80, flags=20, pages=0, procp=0xf5cacccc) at /usr/src/sys/kern/kern_fork.c:274 #16 0xd05e3509 in fork (td=0xd59aad80, uap=0xf5cacd04) at /usr/src/sys/kern/kern_fork.c:98 #17 0xd07a6d10 in syscall (frame= {tf_fs = 134938683, tf_es = 59, tf_ds = -809566149, tf_edi = 134953856, tf_esi = 673312612, tf_ebp = -809526568, tf_isp = -171258524, tf_ebx = 672261300, tf_edx = 0, tf_ecx = 134963456, tf_eax = 2, tf_trapno = 12, tf_err = 2, tf_eip = 672684403, tf_cs = 51, tf_eflags = 642, tf_esp = -809526660, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 #18 0xd078f38f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) frame 10 #10 0xd0629228 in propagate_priority (td=0xd745c900) at /usr/src/sys/kern/subr_turnstile.c:239 239 KASSERT(TD_ON_LOCK(td), ( (kgdb) list 234 #endif 235 236 /* 237 * If we aren't blocked on a lock, we should be. 238 */ 239 KASSERT(TD_ON_LOCK(td), ( 240 "thread %d(%s):%d holds %s but isn't blocked on a lock\n", 241 td->td_tid, td->td_proc->p_comm, td->td_state, 242 ts->ts_lockobj->lo_name)); 243 (kgdb) frame 14 #14 0xd060d340 in thread_single (mode=0) at /usr/src/sys/kern/kern_thread.c:824 824 PROC_LOCK(p); (kgdb) list 819 thread_stopped(p); 820 thread_suspend_one(td); 821 PROC_UNLOCK(p); 822 mi_switch(SW_VOL, NULL); 823 mtx_unlock_spin(&sched_lock); 824 PROC_LOCK(p); 825 mtx_lock_spin(&sched_lock); 826 if (mode == SINGLE_EXIT) 827 remaining = p->p_numthreads; 828 else if (mode == SINGLE_BOUNDARY) -- Totus tuus, Glebius. GLEBIUS-RIPN GLEB-RIPE