From owner-freebsd-stable@FreeBSD.ORG Mon Nov 23 15:27:00 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A3DD1065694; Mon, 23 Nov 2009 15:27:00 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-bw0-f220.google.com (mail-bw0-f220.google.com [209.85.218.220]) by mx1.freebsd.org (Postfix) with ESMTP id 8E3718FC08; Mon, 23 Nov 2009 15:26:59 +0000 (UTC) Received: by bwz20 with SMTP id 20so4040872bwz.14 for ; Mon, 23 Nov 2009 07:26:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=NnmkRq+Ypjb37eZ29UXAixdSssytkqc4ScuPXwhryq8=; b=eY35PnH8EglwaEGjodi9f+kCY7sAqVcbTLG97vT70Iy0qlgcmA62bjjMB6kzQDH72T WJqZn/BHzEfRO3UThPwLS806cIwBsdnYx05cY5FdcN8+Q5ecQgD/z/mifOsMi/GC3GOc os11IAURsX90GaQgCoprVr1FLALHCaWf9N6C0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=G+Cs3CvXAFFDoANSyY4epG+Sqg/rw47ELHGM5k2H5+smY3F+s8PnprX0aG/SnH2iUt Giauo8Tz/8mNbh66LfKIpMmxatBiEWGka+vCRLpRLVVc/v9ixfcc/I8bQSbX7cOlrell e0gbwg0kQY38GOZxePMFQ4s7BymC85QCPz2O4= MIME-Version: 1.0 Received: by 10.204.7.88 with SMTP id c24mr4887647bkc.17.1258990018421; Mon, 23 Nov 2009 07:26:58 -0800 (PST) In-Reply-To: <6B4DCB23-179D-4163-923F-24FC66303086@samsco.org> References: <200910150853.49850.jhb@freebsd.org> <6B4DCB23-179D-4163-923F-24FC66303086@samsco.org> Date: Mon, 23 Nov 2009 18:26:57 +0300 Message-ID: From: pluknet To: Scott Long Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-scsi , freebsd-stable@freebsd.org, John Baldwin Subject: Re: mfi(4) endless loop kernel output on attach X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2009 15:27:00 -0000 2009/11/23 Scott Long : > Did you ever get a resolution for this? =A0The 6.x bugs definitely need t= o be > fixed. =A0The reported timeouts on 7.x might be due to the adapter taking= a > log time to do the command. > > Scott An "endless loop kernel output" on boot always solved with clearing logs with MegaCli -AdpEventLog -GetEventLogInfo -aAll. As for BBULearn issue, I just tested it again on one of my boxes: # ./MegaCli -AdpBbuCmd -BbuLearn -aall Adapter 0: BBU Learn Succeeded. Exit Code: 0x00 So it seems to work. I see no problems. mfi0: 3748 (312279437s/0x0008/info) - Battery relearn started mfi0: 3749 (312279437s/0x0008/WARN) - BBU disabled; changing WB virtual disks to WT mfi0: 3750 (312279437s/0x0001/info) - Policy change on VD 00/0 to [ID=3D00,dcp=3D6d,ccp=3D6c,ap=3D0,dc=3D0,dbgi=3D0] from [ID=3D00,dcp=3D6d,ccp=3D6d,ap=3D0,dc=3D0,dbgi=3D0] mfi0: 3751 (312279442s/0x0008/info) - Battery is discharging mfi0: 3752 (312279442s/0x0008/info) - Battery relearn in progress > > On Oct 22, 2009, at 8:30 AM, pluknet wrote: > >> 2009/10/15 John Baldwin : >>> >>> On Thursday 15 October 2009 5:51:19 am pluknet wrote: >>>> >>>> Hi. >>>> >>>> This is 7.2-R. Seen on IBM x3650M2. >>>> >>>> During the boot I get those endless looping kernel messages while on >>>> mfi(4) attach phase. >>>> It's getting more odd since 7.2 booted and worked fine on exactly this >>>> server model >>>> months ago (on different box though).. Any hints? >>> >>> We just had some boxes die like this (but spewing a different loop of >>> messages >>> on boot related to continuously scheduling patrol reads and consistency >>> checks that finished immediately) at work. =A0We fixed them by swapping= out >>> the >>> controller. =A0We might try stick them in a different box and reflashin= g >>> them >>> using mfiutil(8) to see if it's some sort of corrupted state that >>> flashing >>> the adapter fixes. >>> >>> In your case it looks lik the firmware keeps crashing and restarting. >>> >> >> Some more thoughts.. >> >> There was a problem I got with 'MegaCli -AdpBbuCmd -BbuLearn -aall' >> command. >> On 6.2-R process slept on mfiwait wchan: >> >> db> bt 14734 >> Tracing pid 14734 tid 100135 td 0xc93f8190 >> sched_switch(c93f8190,0,1) at sched_switch+0x143 >> mi_switch(1,0,c93f8190,f9a32acc,c06a43a4,...) at mi_switch+0x1ba >> sleepq_switch(c8c6b0d0) at sleepq_switch+0x87 >> sleepq_wait(c8c6b0d0,0,c93f8190,c8c6b0d0,c8c25800,...) at sleepq_wait+0x= 5c >> msleep(c8c6b0d0,c8c25954,4c,c090acbc,0) at msleep+0x269 >> mfi_wait_command(c8c25800,c8c6b0d0,0,0,cc382460,...) at >> mfi_wait_command+0xa8 >> mfi_ioctl(c8c31300,c1144d01,cc870a00,1,c93f8190,...) at mfi_ioctl+0x485 >> devfs_ioctl_f(c90a2750,c1144d01,cc870a00,c9048000,c93f8190) at >> devfs_ioctl_f+0xaf >> ioctl(c93f8190,f9a32d04) at ioctl+0x445 >> syscall(3b,3b,3b,0,bfbfedc0,...) at syscall+0x2bf >> Xint0x80_syscall() at Xint0x80_syscall+0x1f >> --- syscall (54, FreeBSD ELF32, ioctl), eip =3D 0x8177207, esp =3D >> 0xbfbfe88c, ebp =3D 0xbfbfe8b8 --- >> >> Then: >> mfi0: COMMAND 0xc8c6b0d0 TIMEOUT AFTER 51 SECONDS >> mfi0: COMMAND 0xc8c61d50 TIMEOUT AFTER 49 SECONDS >> mfi0: COMMAND 0xc8c61850 TIMEOUT AFTER 49 SECONDS >> >> >> On 6.4-R MegaCli throws a page fault due to NULL deref >> in mfi_data_cb():cm->cm_sg (see below). >> >> There was past 6.4 backport mentioning >> "fix some bugs in the API for the management ioctl." >> With this patch I have no longer panic and/or locks. >> >> Thanks to LSI now on 7.2-R (and on patched 6.4-R) it returns an error: >> # ./MegaCli -AdpBbuCmd -BbuLearn -aall >> >> Adapter 0: BBU Learn Failed >> >> Exit Code: 0x32 >> >> >> db> bt >> Tracing pid 43059 tid 101363 td 0xcf46e680 >> mfi_data_cb(c9cfae00,c9cc3e00,1,0) at mfi_data_cb+0x5e >> bus_dmamap_load(c9cd7c80,0,caf86270,0,c0597240,c9cfae00,0) at >> bus_dmamap_load+0x4a1 >> mfi_mapcmd(c9cc3800,c9cfae00) at mfi_mapcmd+0x31 >> mfi_startio(c9cc3800) at mfi_startio+0x9b >> mfi_wait_command(c9cc3800,c9cfae00,0,0,caf86270,...) at >> mfi_wait_command+0x89 >> mfi_ioctl(c9cf7200,c1144d01,d3fb6200,1,cf46e680,...) at mfi_ioctl+0x52a >> devfs_ioctl_f(d1a551b0,c1144d01,d3fb6200,cbf52c80,cf46e680) at >> devfs_ioctl_f+0xaf >> ioctl(cf46e680,fbd91d04) at ioctl+0x445 >> syscall(3b,3b,3b,0,bfbfedc0,...) at syscall+0x2bf >> Xint0x80_syscall() at Xint0x80_syscall+0x1f >> --- syscall (54, FreeBSD ELF32, ioctl), eip =3D 0x8177207, esp =3D >> 0xbfbfe88c, ebp =3D 0xbfbfe8b8 >> >> #9 =A00xc08cbb1a in calltrap () at /usr/src/sys/i386/i386/exception.s:13= 9 >> #10 0xc059729e in mfi_data_cb (arg=3D0xc8a744b0, segs=3D0xc8a49e00, nseg= s=3D1, >> ---Type to continue, or q to quit--- >> =A0 error=3D0) at /usr/src/sys/dev/mfi/mfi.c:1488 >> #11 0xc08c7afd in bus_dmamap_load (dmat=3D0xc8a6f100, map=3D0xac89e000, >> =A0 buf=3D0xc8a5ac60, buflen=3D0, callback=3D0xc0597240 , >> =A0 callback_arg=3D0xc8a744b0, flags=3D0) >> =A0 at /usr/src/sys/i386/i386/busdma_machdep.c:733 >> #12 0xc059721d in mfi_mapcmd (sc=3D0xc8a49800, cm=3D0xc8a49e00) >> =A0 at /usr/src/sys/dev/mfi/mfi.c:1452 >> #13 0xc0597177 in mfi_startio (sc=3D0xc8a49800) >> =A0 at /usr/src/sys/dev/mfi/mfi.c:1436 >> #14 0xc0595f09 in mfi_wait_command (sc=3D0xc8a49800, cm=3D0xc8a744b0) >> =A0 at /usr/src/sys/dev/mfi/mfi.c:822 >> #15 0xc059840a in mfi_ioctl (dev=3D0xac89e000, cmd=3D0, arg=3D0xc8de8800= "", >> flag=3D1, >> =A0 td=3D0xc8a5ac60) at /usr/src/sys/dev/mfi/mfi.c:2061 >> #16 0xc06598b7 in devfs_ioctl_f (fp=3D0xc902dc18, com=3D3239333121, >> =A0 data=3D0xc8de8800, cred=3D0xc9052980, td=3D0xc8e2dd00) >> =A0 at /usr/src/sys/fs/devfs/devfs_vnops.c:480 >> #17 0xc06d3a11 in ioctl (td=3D0xc8e2dd00, uap=3D0xeb37bd04) at file.h:26= 5 >> >> (kgdb) f 10 >> #10 0xc059729e in mfi_data_cb (arg=3D0xc8a744b0, segs=3D0xc8a49e00, nseg= s=3D1, >> =A0 error=3D0) at /usr/src/sys/dev/mfi/mfi.c:1488 >> 1488 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sgl->sg32[i]= .addr =3D segs[i].ds_addr; >> (kgdb) list >> 1483 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return; >> 1484 =A0 =A0 =A0 =A0 =A0 =A0} >> 1485 >> 1486 =A0 =A0 =A0 =A0 =A0 =A0if ((sc->mfi_flags & MFI_FLAGS_SG64) =3D=3D = 0) { >> 1487 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for (i =3D 0; i < nsegs; i++= ) { >> 1488 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sgl->sg32[i]= .addr =3D segs[i].ds_addr; >> 1489 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sgl->sg32[i]= .len =3D segs[i].ds_len; >> 1490 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> 1491 =A0 =A0 =A0 =A0 =A0 =A0} else { >> 1492 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for (i =3D 0; i < nsegs; i++= ) { >> (kgdb) p i >> $1 =3D 0 >> (kgdb) p *segs >> $3 =3D {ds_addr =3D 2457600, ds_len =3D 65536} >> (kgdb) p sgl >> $4 =3D (union mfi_sgl *) 0x0 >> (kgdb) p *cm >> $6 =3D {cm_link =3D {tqe_next =3D 0x0, tqe_prev =3D 0xc8a49814}, cm_time= stamp =3D 0, >> =A0cm_sc =3D 0xc8a49800, cm_frame =3D 0xe8fee680, cm_frame_busaddr =3D 3= 748513408, >> =A0cm_sense =3D 0xe904c780, cm_sense_busaddr =3D 3749103488, cm_dmamap = =3D 0x0, >> =A0cm_sg =3D 0x0, cm_data =3D 0xc8a5ac60, cm_len =3D 0, cm_total_frame_s= ize =3D 0, >> =A0cm_extra_frames =3D 0, cm_flags =3D 6, cm_aen_abort =3D 0, cm_complet= e =3D 0, >> =A0cm_private =3D 0x0, cm_index =3D 15, cm_error =3D 0} >> >> >> -- >> wbr, >> pluknet >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org= " > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > --=20 wbr, pluknet