From owner-freebsd-current Sun Nov 17 11:54:17 2002 Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D574537B401 for ; Sun, 17 Nov 2002 11:54:13 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 65A1843E4A for ; Sun, 17 Nov 2002 11:54:12 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.12.6/8.12.5) with SMTP id gAHJs4BF010901; Sun, 17 Nov 2002 14:54:04 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Sun, 17 Nov 2002 14:54:04 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: "Joel M. Baldwin" Cc: current@freebsd.org Subject: Re: more info from panic from running dnet on SMP kernel ( lock order reversal, recursed on non-recursive lock ) In-Reply-To: <211086306.1037497825@[192.168.1.20]> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hmm. It looks like there is indeed a lock leak in the RFTHREAD code. Maybe a change like the following might help: PROC_LOCK(p2); psignal(p2, SIGKILL); PROC_UNLOCK(p2); } Change the } to: =09=09} else =09=09=09PROC_UNLOCK(p1->p_leader); And see if that gets rid of the problem. Any chance this is highly reproduceable, btw? :-) And what app are you running that's using RFTHREAD -- linux thread stuff? Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories On Sun, 17 Nov 2002, Joel M. Baldwin wrote: >=20 > running dnet on a SMP kernel causes the kernel to panic. >=20 >=20 > lock order reversal > 1st 0xc2c803e8 process lock (process lock) @=20 > ../../../kern/kern_fork.c:571 > 2nd 0xc03cfce0 proctree (proctree) @ ../../../kern/kern_fork.c:596 > recursed on non-recursive lock (sleep mutex) process lock @=20 > ../../../kern/kern_fork.c:599 > first acquired @ ../../../kern/kern_fork.c:571 > panic: recurse > cpuid =3D 1; lapic.id =3D 01000000 > Debugger("panic") > Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0 > db> t > Debugger(c03926fa,1000000,c0395ada,d26f5c08,1) at Debugger+0x55 > panic(c0395ada,c038feab,23b,c038feab,257) at panic+0x11f > witness_lock(c2c803e8,8,c038feab,257,0) at witness_lock+0x3e6 > _mtx_lock_flags(c2c803e8,0,c038feab,257,d26f5cb8) at=20 > _mtx_lock_flags+0xb2 > fork1(c2773d00,6050,0,d26f5cd4,c2c803e8) at fork1+0xbfc > rfork(c2773d00,d26f5d10,c03b07a2,407,1) at rfork+0x65 > syscall(2f,2f,2f,0,80ddf10) at syscall+0x28e > Xint0x80_syscall() at Xint0x80_syscall+0x1d > --- syscall (251, FreeBSD ELF32, rfork), eip =3D 0x8087d14, esp =3D=20 > 0xbfbff4a8, ebp =3D 0xbfbff524 --- > db> ps > pid proc addr uid ppid pgrp flag stat wmesg wchan=20 > cmd > 6217 c2b98e00 d28a7000 0 6215 6216 0000000 newpanic: unknown=20 > thread state > cpuid =3D 1; lapic.id =3D 01000000 > boot() called on cpu#1 > Uptime: 1h43m39s > pfs_vncache_unload(): 1 entries remaining > Dumping 255 MB > 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 > Dump complete > Automatic reboot in 15 seconds - press a key on the console to abort > Rebooting... > cpu_reset called on cpu#1 > cpu_reset: Restarting BSP > cpu_reset_proxy: Stopped CPU 1 >=20 >=20 >=20 > And then when the system came back up and I took a closer > look at the core dump. >=20 >=20 > (kgdb) where > #0 doadump () at ../../../kern/kern_shutdown.c:232 > #1 0xc02114ad in boot (howto=3D260) at ../../../kern/kern_shutdown.c:364 > #2 0xc0211767 in panic () at ../../../kern/kern_shutdown.c:517 > #3 0xc014f2bc in db_ps (dummy1=3D-1070342907, dummy2=3D0, dummy3=3D-1,= =20 > dummy4=3D0xd26f5a24 "") > at ../../../ddb/db_ps.c:169 > #4 0xc014d142 in db_command (last_cmdp=3D0xc03be920, cmd_table=3D0x0,=20 > aux_cmd_tablep=3D0xc03b5540, > aux_cmd_tablep_end=3D0xc03b5558) at ../../../ddb/db_command.c:346 > #5 0xc014d256 in db_command_loop () at ../../../ddb/db_command.c:472 > #6 0xc014feea in db_trap (type=3D3, code=3D0) at ../../../ddb/db_trap.c:= 72 > #7 0xc033da10 in kdb_trap (type=3D3, code=3D0, regs=3D0xd26f5b80) > at ../../../i386/i386/db_interface.c:166 > #8 0xc0356a3f in trap (frame=3D > {tf_fs =3D -1069481960, tf_es =3D 16, tf_ds =3D 16, tf_edi =3D=20 > -1032372992, tf_esi =3D 256, tf_ebp =3D -764453940, tf_isp =3D -764453972= ,=20 > tf_ebx =3D 0, tf_edx =3D 0, tf_ecx =3D 1, tf_eax =3D 18, tf_trapno =3D 3,= tf_err=20 > =3D 0, tf_eip =3D -1070342907, tf_cs =3D 8, tf_eflags =3D 662, tf_esp =3D= =20 > -1069883258, tf_ss =3D -1069996294}) at ../../../i386/i386/trap.c:603 > #9 0xc033f238 in calltrap () at {standard input}:99 > #10 0xc021174f in panic (fmt=3D0x0) at ../../../kern/kern_shutdown.c:503 > #11 0xc02333d6 in witness_lock (lock=3D0xc2c803e8, flags=3D8, > file=3D0xc038feab "../../../kern/kern_fork.c", line=3D599) at=20 > ../../../kern/subr_witness.c:609 > #12 0xc02079c2 in _mtx_lock_flags (m=3D0xc03cf4c0, opts=3D0,=20 > file=3D0xc042cfd4 "=E8\003=C8=C2=AB=FE8=C0;\002", > line=3D-1027079192) at ../../../kern/kern_mutex.c:328 > #13 0xc01fd3ec in fork1 (td=3D0xc2773d00, flags=3D24656, pages=3D0,=20 > procp=3D0xd26f5cd4) > at ../../../kern/kern_fork.c:599 > #14 0xc01fc6c5 in rfork (td=3D0xc2773d00, uap=3D0xd26f5d10) at=20 > ../../../kern/kern_fork.c:168 > #15 0xc035739e in syscall (frame=3D > {tf_fs =3D 47, tf_es =3D 47, tf_ds =3D 47, tf_edi =3D 0, tf_esi =3D= =20 > 135126800, tf_ebp =3D -1077938908, tf_isp =3D -764453516, tf_ebx =3D 2,= =20 > tf_edx =3D 135381248, tf_ecx =3D 135381248, tf_eax =3D 251, tf_trapno =3D= 0,=20 > tf_err =3D 2, tf_eip =3D 134774036, tf_cs =3D 31, tf_eflags =3D 659, tf_e= sp =3D=20 > -1077939032, tf_ss =3D 47}) > at ../../../i386/i386/trap.c:1033 > #16 0xc033f28d in Xint0x80_syscall () at {standard input}:141 > ---Can't read userspace from dump, or kernel process--- >=20 >=20 >=20 >=20 >=20 >=20 > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-current" in the body of the message >=20 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message