Date: Mon, 23 Nov 2009 18:26:57 +0300 From: pluknet <pluknet@gmail.com> To: Scott Long <scottl@samsco.org> Cc: freebsd-scsi <freebsd-scsi@freebsd.org>, freebsd-stable@freebsd.org, John Baldwin <jhb@freebsd.org> Subject: Re: mfi(4) endless loop kernel output on attach Message-ID: <a31046fc0911230726j794aeb4r3fb08898723d91fb@mail.gmail.com> In-Reply-To: <6B4DCB23-179D-4163-923F-24FC66303086@samsco.org> References: <a31046fc0910150251q727ea426jca72be60ed29d513@mail.gmail.com> <200910150853.49850.jhb@freebsd.org> <a31046fc0910220730x786d1fdcm68dee86db48ce81b@mail.gmail.com> <6B4DCB23-179D-4163-923F-24FC66303086@samsco.org>
next in thread | previous in thread | raw e-mail | index | archive | help
2009/11/23 Scott Long <scottl@samsco.org>:
> Did you ever get a resolution for this? =A0The 6.x bugs definitely need t=
o be
> fixed. =A0The reported timeouts on 7.x might be due to the adapter taking=
a
> log time to do the command.
>
> Scott
An "endless loop kernel output" on boot always solved with clearing logs
with MegaCli -AdpEventLog -GetEventLogInfo -aAll.
As for BBULearn issue, I just tested it again on one of my boxes:
# ./MegaCli -AdpBbuCmd -BbuLearn -aall
Adapter 0: BBU Learn Succeeded.
Exit Code: 0x00
So it seems to work. I see no problems.
mfi0: 3748 (312279437s/0x0008/info) - Battery relearn started
mfi0: 3749 (312279437s/0x0008/WARN) - BBU disabled; changing WB
virtual disks to WT
mfi0: 3750 (312279437s/0x0001/info) - Policy change on VD 00/0 to
[ID=3D00,dcp=3D6d,ccp=3D6c,ap=3D0,dc=3D0,dbgi=3D0] from
[ID=3D00,dcp=3D6d,ccp=3D6d,ap=3D0,dc=3D0,dbgi=3D0]
mfi0: 3751 (312279442s/0x0008/info) - Battery is discharging
mfi0: 3752 (312279442s/0x0008/info) - Battery relearn in progress
>
> On Oct 22, 2009, at 8:30 AM, pluknet wrote:
>
>> 2009/10/15 John Baldwin <jhb@freebsd.org>:
>>>
>>> On Thursday 15 October 2009 5:51:19 am pluknet wrote:
>>>>
>>>> Hi.
>>>>
>>>> This is 7.2-R. Seen on IBM x3650M2.
>>>>
>>>> During the boot I get those endless looping kernel messages while on
>>>> mfi(4) attach phase.
>>>> It's getting more odd since 7.2 booted and worked fine on exactly this
>>>> server model
>>>> months ago (on different box though).. Any hints?
>>>
>>> We just had some boxes die like this (but spewing a different loop of
>>> messages
>>> on boot related to continuously scheduling patrol reads and consistency
>>> checks that finished immediately) at work. =A0We fixed them by swapping=
out
>>> the
>>> controller. =A0We might try stick them in a different box and reflashin=
g
>>> them
>>> using mfiutil(8) to see if it's some sort of corrupted state that
>>> flashing
>>> the adapter fixes.
>>>
>>> In your case it looks lik the firmware keeps crashing and restarting.
>>>
>>
>> Some more thoughts..
>>
>> There was a problem I got with 'MegaCli -AdpBbuCmd -BbuLearn -aall'
>> command.
>> On 6.2-R process slept on mfiwait wchan:
>>
>> db> bt 14734
>> Tracing pid 14734 tid 100135 td 0xc93f8190
>> sched_switch(c93f8190,0,1) at sched_switch+0x143
>> mi_switch(1,0,c93f8190,f9a32acc,c06a43a4,...) at mi_switch+0x1ba
>> sleepq_switch(c8c6b0d0) at sleepq_switch+0x87
>> sleepq_wait(c8c6b0d0,0,c93f8190,c8c6b0d0,c8c25800,...) at sleepq_wait+0x=
5c
>> msleep(c8c6b0d0,c8c25954,4c,c090acbc,0) at msleep+0x269
>> mfi_wait_command(c8c25800,c8c6b0d0,0,0,cc382460,...) at
>> mfi_wait_command+0xa8
>> mfi_ioctl(c8c31300,c1144d01,cc870a00,1,c93f8190,...) at mfi_ioctl+0x485
>> devfs_ioctl_f(c90a2750,c1144d01,cc870a00,c9048000,c93f8190) at
>> devfs_ioctl_f+0xaf
>> ioctl(c93f8190,f9a32d04) at ioctl+0x445
>> syscall(3b,3b,3b,0,bfbfedc0,...) at syscall+0x2bf
>> Xint0x80_syscall() at Xint0x80_syscall+0x1f
>> --- syscall (54, FreeBSD ELF32, ioctl), eip =3D 0x8177207, esp =3D
>> 0xbfbfe88c, ebp =3D 0xbfbfe8b8 ---
>>
>> Then:
>> mfi0: COMMAND 0xc8c6b0d0 TIMEOUT AFTER 51 SECONDS
>> mfi0: COMMAND 0xc8c61d50 TIMEOUT AFTER 49 SECONDS
>> mfi0: COMMAND 0xc8c61850 TIMEOUT AFTER 49 SECONDS
>>
>>
>> On 6.4-R MegaCli throws a page fault due to NULL deref
>> in mfi_data_cb():cm->cm_sg (see below).
>>
>> There was past 6.4 backport mentioning
>> "fix some bugs in the API for the management ioctl."
>> With this patch I have no longer panic and/or locks.
>>
>> Thanks to LSI now on 7.2-R (and on patched 6.4-R) it returns an error:
>> # ./MegaCli -AdpBbuCmd -BbuLearn -aall
>>
>> Adapter 0: BBU Learn Failed
>>
>> Exit Code: 0x32
>>
>>
>> db> bt
>> Tracing pid 43059 tid 101363 td 0xcf46e680
>> mfi_data_cb(c9cfae00,c9cc3e00,1,0) at mfi_data_cb+0x5e
>> bus_dmamap_load(c9cd7c80,0,caf86270,0,c0597240,c9cfae00,0) at
>> bus_dmamap_load+0x4a1
>> mfi_mapcmd(c9cc3800,c9cfae00) at mfi_mapcmd+0x31
>> mfi_startio(c9cc3800) at mfi_startio+0x9b
>> mfi_wait_command(c9cc3800,c9cfae00,0,0,caf86270,...) at
>> mfi_wait_command+0x89
>> mfi_ioctl(c9cf7200,c1144d01,d3fb6200,1,cf46e680,...) at mfi_ioctl+0x52a
>> devfs_ioctl_f(d1a551b0,c1144d01,d3fb6200,cbf52c80,cf46e680) at
>> devfs_ioctl_f+0xaf
>> ioctl(cf46e680,fbd91d04) at ioctl+0x445
>> syscall(3b,3b,3b,0,bfbfedc0,...) at syscall+0x2bf
>> Xint0x80_syscall() at Xint0x80_syscall+0x1f
>> --- syscall (54, FreeBSD ELF32, ioctl), eip =3D 0x8177207, esp =3D
>> 0xbfbfe88c, ebp =3D 0xbfbfe8b8
>>
>> #9 =A00xc08cbb1a in calltrap () at /usr/src/sys/i386/i386/exception.s:13=
9
>> #10 0xc059729e in mfi_data_cb (arg=3D0xc8a744b0, segs=3D0xc8a49e00, nseg=
s=3D1,
>> ---Type <return> to continue, or q <return> to quit---
>> =A0 error=3D0) at /usr/src/sys/dev/mfi/mfi.c:1488
>> #11 0xc08c7afd in bus_dmamap_load (dmat=3D0xc8a6f100, map=3D0xac89e000,
>> =A0 buf=3D0xc8a5ac60, buflen=3D0, callback=3D0xc0597240 <mfi_data_cb>,
>> =A0 callback_arg=3D0xc8a744b0, flags=3D0)
>> =A0 at /usr/src/sys/i386/i386/busdma_machdep.c:733
>> #12 0xc059721d in mfi_mapcmd (sc=3D0xc8a49800, cm=3D0xc8a49e00)
>> =A0 at /usr/src/sys/dev/mfi/mfi.c:1452
>> #13 0xc0597177 in mfi_startio (sc=3D0xc8a49800)
>> =A0 at /usr/src/sys/dev/mfi/mfi.c:1436
>> #14 0xc0595f09 in mfi_wait_command (sc=3D0xc8a49800, cm=3D0xc8a744b0)
>> =A0 at /usr/src/sys/dev/mfi/mfi.c:822
>> #15 0xc059840a in mfi_ioctl (dev=3D0xac89e000, cmd=3D0, arg=3D0xc8de8800=
"",
>> flag=3D1,
>> =A0 td=3D0xc8a5ac60) at /usr/src/sys/dev/mfi/mfi.c:2061
>> #16 0xc06598b7 in devfs_ioctl_f (fp=3D0xc902dc18, com=3D3239333121,
>> =A0 data=3D0xc8de8800, cred=3D0xc9052980, td=3D0xc8e2dd00)
>> =A0 at /usr/src/sys/fs/devfs/devfs_vnops.c:480
>> #17 0xc06d3a11 in ioctl (td=3D0xc8e2dd00, uap=3D0xeb37bd04) at file.h:26=
5
>>
>> (kgdb) f 10
>> #10 0xc059729e in mfi_data_cb (arg=3D0xc8a744b0, segs=3D0xc8a49e00, nseg=
s=3D1,
>> =A0 error=3D0) at /usr/src/sys/dev/mfi/mfi.c:1488
>> 1488 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sgl->sg32[i]=
.addr =3D segs[i].ds_addr;
>> (kgdb) list
>> 1483 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return;
>> 1484 =A0 =A0 =A0 =A0 =A0 =A0}
>> 1485
>> 1486 =A0 =A0 =A0 =A0 =A0 =A0if ((sc->mfi_flags & MFI_FLAGS_SG64) =3D=3D =
0) {
>> 1487 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for (i =3D 0; i < nsegs; i++=
) {
>> 1488 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sgl->sg32[i]=
.addr =3D segs[i].ds_addr;
>> 1489 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sgl->sg32[i]=
.len =3D segs[i].ds_len;
>> 1490 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
>> 1491 =A0 =A0 =A0 =A0 =A0 =A0} else {
>> 1492 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for (i =3D 0; i < nsegs; i++=
) {
>> (kgdb) p i
>> $1 =3D 0
>> (kgdb) p *segs
>> $3 =3D {ds_addr =3D 2457600, ds_len =3D 65536}
>> (kgdb) p sgl
>> $4 =3D (union mfi_sgl *) 0x0
>> (kgdb) p *cm
>> $6 =3D {cm_link =3D {tqe_next =3D 0x0, tqe_prev =3D 0xc8a49814}, cm_time=
stamp =3D 0,
>> =A0cm_sc =3D 0xc8a49800, cm_frame =3D 0xe8fee680, cm_frame_busaddr =3D 3=
748513408,
>> =A0cm_sense =3D 0xe904c780, cm_sense_busaddr =3D 3749103488, cm_dmamap =
=3D 0x0,
>> =A0cm_sg =3D 0x0, cm_data =3D 0xc8a5ac60, cm_len =3D 0, cm_total_frame_s=
ize =3D 0,
>> =A0cm_extra_frames =3D 0, cm_flags =3D 6, cm_aen_abort =3D 0, cm_complet=
e =3D 0,
>> =A0cm_private =3D 0x0, cm_index =3D 15, cm_error =3D 0}
>>
>>
>> --
>> wbr,
>> pluknet
>> _______________________________________________
>> freebsd-stable@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org=
"
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
>
--=20
wbr,
pluknet
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a31046fc0911230726j794aeb4r3fb08898723d91fb>