Date: Thu, 20 Oct 2022 12:29:12 -0400 From: Ken Merry <ken@freebsd.org> To: Alan Somers <asomers@freebsd.org> Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 11778fca4a83 - main - Fix mpr(4) panic during a firmware update. Message-ID: <F4FC4172-7C8A-4F9D-A194-AEE1C7BD734A@freebsd.org> In-Reply-To: <CAOtMX2jLP1icxmw-Q-wry01StSLFt6n7tOVJOJfeUDivS9aEEg@mail.gmail.com> References: <202210171652.29HGqtSf007611@gitrepo.freebsd.org> <CAOtMX2jLP1icxmw-Q-wry01StSLFt6n7tOVJOJfeUDivS9aEEg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_2608340F-DAC8-41FE-A3D5-9D8DA3CFCA3C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Oct 20, 2022, at 12:23, Alan Somers <asomers@freebsd.org> wrote: >=20 > On Mon, Oct 17, 2022 at 10:53 AM Kenneth D. Merry <ken@freebsd.org = <mailto:ken@freebsd.org>> wrote: >>=20 >> The branch main has been updated by ken: >>=20 >> URL: = https://cgit.FreeBSD.org/src/commit/?id=3D11778fca4a83f5e3b597c75785aa5c0e= e0dc518e >>=20 >> commit 11778fca4a83f5e3b597c75785aa5c0ee0dc518e >> Author: Kenneth D. Merry <ken@FreeBSD.org> >> AuthorDate: 2022-10-17 16:48:34 +0000 >> Commit: Kenneth D. Merry <ken@FreeBSD.org> >> CommitDate: 2022-10-17 16:48:34 +0000 >>=20 >> Fix mpr(4) panic during a firmware update. >>=20 >> Issue Description: >> The RequestCredits field of IOCFacts got changed between the = Phase23 >> firmware to Phase24 firmware. So as part of firmware update = operation, >> driver has to free the resources & pools which are created with = the Phase23 >> Firmware's IOCFacts data (i.e. during driver load time) and has to >> reallocate the resources and pools using Phase24's IOCFacts data. = Here >> driver has freed the interrupts but missed to reallocate the = interrupts and >> hence config page read operation is getting timed out and = controller is >> going for recursive reinit (controller reset) operations and = leading to >> kernel panic. >>=20 >> Fix: >> Reallocate the interrupts if the interrupts are disabled as part = of >> firmware update/downgrade operation. >>=20 >> Submitted by: Sreekanth Ready <sreekanth.reddy@broadcom.com> >> Tested by: ken >> MFC after: 3 days >=20 > Would this commit fix the panic in bug 252575? >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252575 = <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252575>; Unfortunately, no. This is the panic I ran into on a firmware upgrade using storcli64 on a = Broadcom 9600-16i card going from Phase 23 to Phase 24: mpr0: Reinitializing controller mpr0: Firmware: 24.00.00.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: = 2fa84c<ScsiTaskFull,DiagTrace,EEDP,TransRetry,EventReplay ,MSIXIndex,HostDisc,FastPath,RDPQArray,AtomicReqDesc> mpr0: Calling Reinit from mpr_wait_command, timeout=3D30, elapsed=3D30 mpr0: Reinitializing controller mpr_config_get_ioc_pg8: request for header completed with error 0 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr0: Unfreezing SIM queue mpr0: fault_state(0x40001500)! mpr0: Timeout while writing doorbell panic: mpr_iocfacts_allocate failed to get IOC Facts with error 6 cpuid =3D 1 time =3D 1663771142 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff8040a9ab =3D = db_trace_self_wrapper+0x2b/frame 0xfffffe02e759d6c0 vpanic() at 0xffffffff805f7ad1 =3D vpanic+0x151/frame 0xfffffe02e759d710 panic() at 0xffffffff805f7973 =3D panic+0x43/frame 0xfffffe02e759d770 mpr_iocfacts_allocate() at 0xffffffff8154bdad =3D = mpr_iocfacts_allocate+0x15dd/frame 0xfffffe02e759d8e0 mpr_reinit() at 0xffffffff81549e0c =3D mpr_reinit+0x14c/frame = 0xfffffe02e759d930 mpr_wait_command() at 0xffffffff8154fb69 =3D = mpr_wait_command+0x1d9/frame 0xfffffe02e759d9b0 mpr_ioctl() at 0xffffffff8155a113 =3D mpr_ioctl+0x1de3/frame = 0xfffffe02e759db40 devfs_ioctl() at 0xffffffff804bab8f =3D devfs_ioctl+0xaf/frame = 0xfffffe02e759db90 vn_ioctl() at 0xffffffff806e9664 =3D vn_ioctl+0x1a4/frame = 0xfffffe02e759dca0 devfs_ioctl_f() at 0xffffffff804bb21e =3D devfs_ioctl_f+0x1e/frame = 0xfffffe02e759dcc0 kern_ioctl() at 0xffffffff80667b2d =3D kern_ioctl+0x26d/frame = 0xfffffe02e759dd30 sys_ioctl() at 0xffffffff80667811 =3D sys_ioctl+0x101/frame = 0xfffffe02e759de00 amd64_syscall() at 0xffffffff8092ae4c =3D amd64_syscall+0x10c/frame = 0xfffffe02e759df30 fast_syscall_common() at 0xffffffff8090369b =3D = fast_syscall_common+0xf8/frame 0xfffffe02e759df30 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x8dfe3a, rsp =3D = 0x821837048, rbp =3D 0x4 --- Uptime: 18h2m58s Dumping 5338 out of 130903 = MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Ken =E2=80=94=20 Ken Merry ken@FreeBSD.ORG= --Apple-Mail=_2608340F-DAC8-41FE-A3D5-9D8DA3CFCA3C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><div = class=3D""><br class=3D""></div><div><blockquote type=3D"cite" = class=3D""><div class=3D"">On Oct 20, 2022, at 12:23, Alan Somers <<a = href=3D"mailto:asomers@freebsd.org" class=3D"">asomers@freebsd.org</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><meta= charset=3D"UTF-8" class=3D""><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Menlo-Regular; font-size: 11px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;" = class=3D"">On Mon, Oct 17, 2022 at 10:53 AM Kenneth D. Merry = <</span><a href=3D"mailto:ken@freebsd.org" style=3D"font-family: = Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px;" class=3D"">ken@freebsd.org</a><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 11px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;" class=3D"">> = wrote:</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;" class=3D""><blockquote type=3D"cite" style=3D"font-family: = Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><br = class=3D"">The branch main has been updated by ken:<br class=3D""><br = class=3D"">URL: <a = href=3D"https://cgit.FreeBSD.org/src/commit/?id=3D11778fca4a83f5e3b597c757= 85aa5c0ee0dc518e" = class=3D"">https://cgit.FreeBSD.org/src/commit/?id=3D11778fca4a83f5e3b597c= 75785aa5c0ee0dc518e</a><br class=3D""><br class=3D"">commit = 11778fca4a83f5e3b597c75785aa5c0ee0dc518e<br class=3D"">Author: = Kenneth D. Merry <<a = href=3D"mailto:ken@FreeBSD.org" class=3D"">ken@FreeBSD.org</a>><br = class=3D"">AuthorDate: 2022-10-17 16:48:34 +0000<br class=3D"">Commit: = Kenneth D. Merry <<a = href=3D"mailto:ken@FreeBSD.org" class=3D"">ken@FreeBSD.org</a>><br = class=3D"">CommitDate: 2022-10-17 16:48:34 +0000<br class=3D""><br = class=3D""> Fix mpr(4) panic during a firmware = update.<br class=3D""><br class=3D""> Issue = Description:<br class=3D""> The RequestCredits field of = IOCFacts got changed between the Phase23<br = class=3D""> firmware to Phase24 firmware. So as part of = firmware update operation,<br class=3D""> driver has to = free the resources & pools which are created with the Phase23<br = class=3D""> Firmware's IOCFacts data (i.e. during = driver load time) and has to<br class=3D""> reallocate = the resources and pools using Phase24's IOCFacts data. Here<br = class=3D""> driver has freed the interrupts but missed = to reallocate the interrupts and<br class=3D""> hence = config page read operation is getting timed out and controller is<br = class=3D""> going for recursive reinit (controller = reset) operations and leading to<br class=3D""> kernel = panic.<br class=3D""><br class=3D""> Fix:<br = class=3D""> Reallocate the interrupts if the interrupts = are disabled as part of<br class=3D""> firmware = update/downgrade operation.<br class=3D""><br = class=3D""> Submitted by: Sreekanth Ready = <<a href=3D"mailto:sreekanth.reddy@broadcom.com" = class=3D"">sreekanth.reddy@broadcom.com</a>><br = class=3D""> Tested by: = ken<br class=3D""> MFC = after: 3 days<br class=3D""></blockquote><br= style=3D"caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 11px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;" class=3D""><span style=3D"caret-color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;" class=3D"">Would this = commit fix the panic in bug 252575?</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><a = href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252575" = style=3D"font-family: Menlo-Regular; font-size: 11px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" = class=3D"">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252575</a><= /div></blockquote></div><br class=3D""><div class=3D"">Unfortunately, = no.</div><div class=3D""><br class=3D""></div><div class=3D"">This is = the panic I ran into on a firmware upgrade using storcli64 on a Broadcom = 9600-16i card going from Phase 23 to Phase 24:</div><div class=3D""><br = class=3D""></div><div class=3D"">mpr0: Reinitializing controller<br = class=3D"">mpr0: Firmware: 24.00.00.00, Driver: 23.00.00.00-fbsd<br = class=3D"">mpr0: IOCCapabilities: = 2fa84c<ScsiTaskFull,DiagTrace,EEDP,TransRetry,EventReplay<br = class=3D"">,MSIXIndex,HostDisc,FastPath,RDPQArray,AtomicReqDesc><br = class=3D"">mpr0: Calling Reinit from mpr_wait_command, timeout=3D30, = elapsed=3D30<br class=3D"">mpr0: Reinitializing controller<br = class=3D"">mpr_config_get_ioc_pg8: request for header completed with = error 0<br class=3D"">mpr_config_get_ioc_pg8: request for header = completed with error 16<br class=3D"">mpr_config_get_ioc_pg8: request = for header completed with error 16<br class=3D"">mpr_config_get_ioc_pg8: = request for header completed with error 16<br = class=3D"">mpr_config_get_ioc_pg8: request for header completed with = error 16<br class=3D"">mpr_config_get_ioc_pg8: request for header = completed with error 16<br class=3D"">mpr_config_get_iounit_pg8: request = for header completed with error 16<br = class=3D"">mpr_config_get_iounit_pg8: request for header completed with = error 16<br class=3D"">mpr_config_get_iounit_pg8: request for header = completed with error 16<br class=3D"">mpr_config_get_iounit_pg8: request = for header completed with error 16<br = class=3D"">mpr_config_get_iounit_pg8: request for header completed with = error 16<br class=3D"">mpr_config_get_iounit_pg8: request for header = completed with error 16<br class=3D"">mpr_config_get_man_pg11: request = for header completed with error 16<br class=3D"">mpr_config_get_man_pg11: = request for header completed with error 16<br = class=3D"">mpr_config_get_man_pg11: request for header completed with = error 16<br class=3D"">mpr_config_get_man_pg11: request for header = completed with error 16<br class=3D"">mpr_config_get_man_pg11: request = for header completed with error 16<br class=3D"">mpr_config_get_man_pg11: = request for header completed with error 16<br = class=3D"">mpr_config_get_dpm_pg0: request for header completed with = error 16<br class=3D"">mpr_config_get_dpm_pg0: request for header = completed with error 16<br class=3D"">mpr_config_get_dpm_pg0: request = for header completed with error 16<br class=3D"">mpr_config_get_dpm_pg0: = request for header completed with error 16<br class=3D"">mpr0: = Unfreezing SIM queue<br class=3D"">mpr0: fault_state(0x40001500)!<br = class=3D"">mpr0: Timeout while writing doorbell<br class=3D"">panic: = mpr_iocfacts_allocate failed to get IOC Facts with error 6<br = class=3D""><br class=3D"">cpuid =3D 1<br class=3D"">time =3D = 1663771142<br class=3D"">KDB: stack backtrace:<br = class=3D"">db_trace_self_wrapper() at 0xffffffff8040a9ab =3D = db_trace_self_wrapper+0x2b/frame 0xfffffe02e759d6c0<br class=3D"">vpanic()= at 0xffffffff805f7ad1 =3D vpanic+0x151/frame 0xfffffe02e759d710<br = class=3D"">panic() at 0xffffffff805f7973 =3D panic+0x43/frame = 0xfffffe02e759d770<br class=3D"">mpr_iocfacts_allocate() at = 0xffffffff8154bdad =3D mpr_iocfacts_allocate+0x15dd/frame = 0xfffffe02e759d8e0<br class=3D"">mpr_reinit() at 0xffffffff81549e0c =3D = mpr_reinit+0x14c/frame 0xfffffe02e759d930<br class=3D"">mpr_wait_command()= at 0xffffffff8154fb69 =3D mpr_wait_command+0x1d9/frame = 0xfffffe02e759d9b0<br class=3D"">mpr_ioctl() at 0xffffffff8155a113 =3D = mpr_ioctl+0x1de3/frame 0xfffffe02e759db40<br class=3D"">devfs_ioctl() at = 0xffffffff804bab8f =3D devfs_ioctl+0xaf/frame 0xfffffe02e759db90<br = class=3D"">vn_ioctl() at 0xffffffff806e9664 =3D vn_ioctl+0x1a4/frame = 0xfffffe02e759dca0<br class=3D"">devfs_ioctl_f() at 0xffffffff804bb21e =3D= devfs_ioctl_f+0x1e/frame 0xfffffe02e759dcc0<br class=3D"">kern_ioctl() = at 0xffffffff80667b2d =3D kern_ioctl+0x26d/frame 0xfffffe02e759dd30<br = class=3D"">sys_ioctl() at 0xffffffff80667811 =3D sys_ioctl+0x101/frame = 0xfffffe02e759de00<br class=3D"">amd64_syscall() at 0xffffffff8092ae4c =3D= amd64_syscall+0x10c/frame 0xfffffe02e759df30<br = class=3D"">fast_syscall_common() at 0xffffffff8090369b =3D = fast_syscall_common+0xf8/frame 0xfffffe02e759df30<br class=3D"">--- = syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x8dfe3a, rsp =3D = 0x821837048, rbp =3D 0x4 ---<br class=3D"">Uptime: 18h2m58s<br = class=3D"">Dumping 5338 out of 130903 = MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%<br = class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D"">Ken</div><div class=3D"">=E2=80=94 </div><div = class=3D"">Ken Merry</div><div class=3D""><a = href=3D"mailto:ken@FreeBSD.ORG" = class=3D"">ken@FreeBSD.ORG</a></div></body></html>= --Apple-Mail=_2608340F-DAC8-41FE-A3D5-9D8DA3CFCA3C--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F4FC4172-7C8A-4F9D-A194-AEE1C7BD734A>