Date: Mon, 28 Jun 2010 19:45:29 +0200 From: Attilio Rao <attilio@freebsd.org> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-current@freebsd.org, pluknet <pluknet@gmail.com>, Anton Yuzhaninov <citrin@citrin.ru> Subject: Re: panic in deadlkres Message-ID: <AANLkTikcSxBoULJod8_wAkAdzS4eagBq5sdRiJ4jsR_W@mail.gmail.com> In-Reply-To: <201006281132.57541.jhb@freebsd.org> References: <i01u5g$39n$1@dough.gmane.org> <AANLkTinam4rwsVtFPSAidVOzdzrlx-whNBId8cMg_ySQ@mail.gmail.com> <201006281132.57541.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
2010/6/28 John Baldwin <jhb@freebsd.org>:
> On Friday 25 June 2010 4:52:22 pm pluknet wrote:
>> On 25 June 2010 13:50, Anton Yuzhaninov <citrin@citrin.ru> wrote:
>> > I've got panic on 9-current from Jun 25 2010
>> >
>> > May be this is bug in deadlock resolver
>> >
>> > panic: blockable sleep lock (sleep mutex) process lock @
>> > /usr/src/sys/kern/kern_clock.c:203
>> >
>> > db> show alllocks
>> > Process 0 (kernel) thread 0xc4dcd270 (100047)
>> > shared sx allproc (allproc) r =3D 0 (0xc0885ebc) locked @
>> > /usr/src/sys/kern/kern_clock.c:193
>> >
>> > db> show lock 0xc4dcd270
>> > =C2=A0class: spin mutex
>> > =C2=A0name: D
>> > =C2=A0flags: {SPIN, RECURSE}
>> > =C2=A0state: {OWNED}
>> >
>> > (kgdb) bt
>> > #0 =C2=A0doadump () at pcpu.h:248
>> > #1 =C2=A00xc05ae59f in boot (howto=3D260) at
> /usr/src/sys/kern/kern_shutdown.c:416
>> > #2 =C2=A00xc05ae825 in panic (fmt=3DVariable "fmt" is not available.
>> > ) at /usr/src/sys/kern/kern_shutdown.c:590
>> > #3 =C2=A00xc048ff45 in db_panic (addr=3DCould not find the frame base =
for
> "db_panic".
>> > ) at /usr/src/sys/ddb/db_command.c:478
>> > #4 =C2=A00xc0490533 in db_command (last_cmdp=3D0xc086ef1c, cmd_table=
=3D0x0,
> dopager=3D1) at /usr/src/sys/ddb/db_command.c:445
>> > #5 =C2=A00xc0490662 in db_command_loop () at /usr/src/sys/ddb/db_comma=
nd.c:498
>> > #6 =C2=A00xc04923ef in db_trap (type=3D3, code=3D0) at
> /usr/src/sys/ddb/db_main.c:229
>> > #7 =C2=A00xc05dade6 in kdb_trap (type=3D3, code=3D0, tf=3D0xc4b31bd0) =
at
> /usr/src/sys/kern/subr_kdb.c:535
>> > #8 =C2=A00xc078696b in trap (frame=3D0xc4b31bd0) at
> /usr/src/sys/i386/i386/trap.c:692
>> > #9 =C2=A00xc076ca0b in calltrap () at /usr/src/sys/i386/i386/exception=
.s:165
>> > #10 0xc05daf30 in kdb_enter (why=3D0xc07ea02d "panic", msg=3D0xc07ea02=
d
> "panic") at cpufunc.h:71
>> > #11 0xc05ae806 in panic (fmt=3D0xc07efd94 "blockable sleep lock (%s) %=
s @
> %s:%d") at /usr/src/sys/kern/kern_shutdown.c:573
>> > #12 0xc05ee30b in witness_checkorder (lock=3D0xc5148088, flags=3D9,
> file=3D0xc07e3b20 "/usr/src/sys/kern/kern_clock.c", line=3D203, interlock=
=3D0x0)
>> > =C2=A0 =C2=A0at /usr/src/sys/kern/subr_witness.c:1067
>> > #13 0xc05a093c in _mtx_lock_flags (m=3D0xc5148088, opts=3D0, file=3D0x=
c07e3b20
> "/usr/src/sys/kern/kern_clock.c", line=3D203)
>> > =C2=A0 =C2=A0at /usr/src/sys/kern/kern_mutex.c:200
>> > #14 0xc05706a9 in deadlkres () at /usr/src/sys/kern/kern_clock.c:203
>> > #15 0xc0588721 in fork_exit (callout=3D0xc05705ea <deadlkres>, arg=3D0=
x0,
> frame=3D0xc4b31d38) at /usr/src/sys/kern/kern_fork.c:843
>> > #16 0xc076ca80 in fork_trampoline () at
> /usr/src/sys/i386/i386/exception.s:270
>>
>> Hi!
>>
>> [throw in ideas (just ignore them if they're dumb, thinking badly atm).]
>>
>> AFAIK, that indicates that some thread already has
>> a spin mutex and then it tries to acquire a sleep mutex.
>>
>> Looks like kern/kern_clock.c v1.213 (SVN rev 206482)
>> has a regression in handling ticks wrap-up
>> w.r.t. it doesn't release a thread mutex, does it?
>
> This looks like a correct analysis to me.
>
>> >From subr_witness.c:
>> 1062: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 * Since sp=
in locks include a critical section, this
> check
>> 1063: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 * implicit=
ly enforces a lock order of all sleep
>> locks before
>> 1064: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 * all spin=
locks.
>> 1065: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 */
>> 1066: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (td->td_=
critnest !=3D 0 && !kdb_active)
>> 1067: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0panic("blockable sleep lock (%s) %s @ %s:%d",
>> 1068: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0class->lc_name, lock->lo_name, file, line);
>>
>> >From kern_clock.c, v1.213 (in several places, while holding a thread lo=
ck):
>> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* Handle ticks wra=
p-up. */
>> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (ticks < td->td_=
blktick)
>> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 continue;
>>
>> Should not it be like the next:
>> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* Handle ticks wra=
p-up. */
>> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (ticks < td->td_=
blktick) {
>> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 thread_unlock(td);
>> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 continue;
>> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
>>
>> The precondition idea to reproduce it is to lock a subject thread
>> in some deadlkres callout, handle re-wrap condition, then try
>> to lock a process to witch the thread belongs in (n+m)'th deadlkres
>> callout, or in different context.
Thanks, that may be fixed in r209577.
Attilio
--=20
Peace can only be achieved by understanding - A. Einstein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTikcSxBoULJod8_wAkAdzS4eagBq5sdRiJ4jsR_W>