Date: Thu, 20 Oct 2022 12:29:12 -0400 From: Ken Merry <ken@freebsd.org> To: Alan Somers <asomers@freebsd.org> Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 11778fca4a83 - main - Fix mpr(4) panic during a firmware update. Message-ID: <F4FC4172-7C8A-4F9D-A194-AEE1C7BD734A@freebsd.org> In-Reply-To: <CAOtMX2jLP1icxmw-Q-wry01StSLFt6n7tOVJOJfeUDivS9aEEg@mail.gmail.com> References: <202210171652.29HGqtSf007611@gitrepo.freebsd.org> <CAOtMX2jLP1icxmw-Q-wry01StSLFt6n7tOVJOJfeUDivS9aEEg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] > On Oct 20, 2022, at 12:23, Alan Somers <asomers@freebsd.org> wrote: > > On Mon, Oct 17, 2022 at 10:53 AM Kenneth D. Merry <ken@freebsd.org <mailto:ken@freebsd.org>> wrote: >> >> The branch main has been updated by ken: >> >> URL: https://cgit.FreeBSD.org/src/commit/?id=11778fca4a83f5e3b597c75785aa5c0ee0dc518e >> >> commit 11778fca4a83f5e3b597c75785aa5c0ee0dc518e >> Author: Kenneth D. Merry <ken@FreeBSD.org> >> AuthorDate: 2022-10-17 16:48:34 +0000 >> Commit: Kenneth D. Merry <ken@FreeBSD.org> >> CommitDate: 2022-10-17 16:48:34 +0000 >> >> Fix mpr(4) panic during a firmware update. >> >> Issue Description: >> The RequestCredits field of IOCFacts got changed between the Phase23 >> firmware to Phase24 firmware. So as part of firmware update operation, >> driver has to free the resources & pools which are created with the Phase23 >> Firmware's IOCFacts data (i.e. during driver load time) and has to >> reallocate the resources and pools using Phase24's IOCFacts data. Here >> driver has freed the interrupts but missed to reallocate the interrupts and >> hence config page read operation is getting timed out and controller is >> going for recursive reinit (controller reset) operations and leading to >> kernel panic. >> >> Fix: >> Reallocate the interrupts if the interrupts are disabled as part of >> firmware update/downgrade operation. >> >> Submitted by: Sreekanth Ready <sreekanth.reddy@broadcom.com> >> Tested by: ken >> MFC after: 3 days > > Would this commit fix the panic in bug 252575? > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252575 <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252575>; Unfortunately, no. This is the panic I ran into on a firmware upgrade using storcli64 on a Broadcom 9600-16i card going from Phase 23 to Phase 24: mpr0: Reinitializing controller mpr0: Firmware: 24.00.00.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: 2fa84c<ScsiTaskFull,DiagTrace,EEDP,TransRetry,EventReplay ,MSIXIndex,HostDisc,FastPath,RDPQArray,AtomicReqDesc> mpr0: Calling Reinit from mpr_wait_command, timeout=30, elapsed=30 mpr0: Reinitializing controller mpr_config_get_ioc_pg8: request for header completed with error 0 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr0: Unfreezing SIM queue mpr0: fault_state(0x40001500)! mpr0: Timeout while writing doorbell panic: mpr_iocfacts_allocate failed to get IOC Facts with error 6 cpuid = 1 time = 1663771142 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff8040a9ab = db_trace_self_wrapper+0x2b/frame 0xfffffe02e759d6c0 vpanic() at 0xffffffff805f7ad1 = vpanic+0x151/frame 0xfffffe02e759d710 panic() at 0xffffffff805f7973 = panic+0x43/frame 0xfffffe02e759d770 mpr_iocfacts_allocate() at 0xffffffff8154bdad = mpr_iocfacts_allocate+0x15dd/frame 0xfffffe02e759d8e0 mpr_reinit() at 0xffffffff81549e0c = mpr_reinit+0x14c/frame 0xfffffe02e759d930 mpr_wait_command() at 0xffffffff8154fb69 = mpr_wait_command+0x1d9/frame 0xfffffe02e759d9b0 mpr_ioctl() at 0xffffffff8155a113 = mpr_ioctl+0x1de3/frame 0xfffffe02e759db40 devfs_ioctl() at 0xffffffff804bab8f = devfs_ioctl+0xaf/frame 0xfffffe02e759db90 vn_ioctl() at 0xffffffff806e9664 = vn_ioctl+0x1a4/frame 0xfffffe02e759dca0 devfs_ioctl_f() at 0xffffffff804bb21e = devfs_ioctl_f+0x1e/frame 0xfffffe02e759dcc0 kern_ioctl() at 0xffffffff80667b2d = kern_ioctl+0x26d/frame 0xfffffe02e759dd30 sys_ioctl() at 0xffffffff80667811 = sys_ioctl+0x101/frame 0xfffffe02e759de00 amd64_syscall() at 0xffffffff8092ae4c = amd64_syscall+0x10c/frame 0xfffffe02e759df30 fast_syscall_common() at 0xffffffff8090369b = fast_syscall_common+0xf8/frame 0xfffffe02e759df30 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8dfe3a, rsp = 0x821837048, rbp = 0x4 --- Uptime: 18h2m58s Dumping 5338 out of 130903 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Ken — Ken Merry ken@FreeBSD.ORG [-- Attachment #2 --] <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div><div><blockquote type="cite" class=""><div class="">On Oct 20, 2022, at 12:23, Alan Somers <<a href="mailto:asomers@freebsd.org" class="">asomers@freebsd.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On Mon, Oct 17, 2022 at 10:53 AM Kenneth D. Merry <</span><a href="mailto:ken@freebsd.org" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">ken@freebsd.org</a><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">> wrote:</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="">The branch main has been updated by ken:<br class=""><br class="">URL: <a href="https://cgit.FreeBSD.org/src/commit/?id=11778fca4a83f5e3b597c75785aa5c0ee0dc518e" class="">https://cgit.FreeBSD.org/src/commit/?id=11778fca4a83f5e3b597c75785aa5c0ee0dc518e</a><br class=""><br class="">commit 11778fca4a83f5e3b597c75785aa5c0ee0dc518e<br class="">Author: Kenneth D. Merry <<a href="mailto:ken@FreeBSD.org" class="">ken@FreeBSD.org</a>><br class="">AuthorDate: 2022-10-17 16:48:34 +0000<br class="">Commit: Kenneth D. Merry <<a href="mailto:ken@FreeBSD.org" class="">ken@FreeBSD.org</a>><br class="">CommitDate: 2022-10-17 16:48:34 +0000<br class=""><br class=""> Fix mpr(4) panic during a firmware update.<br class=""><br class=""> Issue Description:<br class=""> The RequestCredits field of IOCFacts got changed between the Phase23<br class=""> firmware to Phase24 firmware. So as part of firmware update operation,<br class=""> driver has to free the resources & pools which are created with the Phase23<br class=""> Firmware's IOCFacts data (i.e. during driver load time) and has to<br class=""> reallocate the resources and pools using Phase24's IOCFacts data. Here<br class=""> driver has freed the interrupts but missed to reallocate the interrupts and<br class=""> hence config page read operation is getting timed out and controller is<br class=""> going for recursive reinit (controller reset) operations and leading to<br class=""> kernel panic.<br class=""><br class=""> Fix:<br class=""> Reallocate the interrupts if the interrupts are disabled as part of<br class=""> firmware update/downgrade operation.<br class=""><br class=""> Submitted by: Sreekanth Ready <<a href="mailto:sreekanth.reddy@broadcom.com" class="">sreekanth.reddy@broadcom.com</a>><br class=""> Tested by: ken<br class=""> MFC after: 3 days<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Would this commit fix the panic in bug 252575?</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252575" style="font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252575</a></div></blockquote></div><br class=""><div class="">Unfortunately, no.</div><div class=""><br class=""></div><div class="">This is the panic I ran into on a firmware upgrade using storcli64 on a Broadcom 9600-16i card going from Phase 23 to Phase 24:</div><div class=""><br class=""></div><div class="">mpr0: Reinitializing controller<br class="">mpr0: Firmware: 24.00.00.00, Driver: 23.00.00.00-fbsd<br class="">mpr0: IOCCapabilities: 2fa84c<ScsiTaskFull,DiagTrace,EEDP,TransRetry,EventReplay<br class="">,MSIXIndex,HostDisc,FastPath,RDPQArray,AtomicReqDesc><br class="">mpr0: Calling Reinit from mpr_wait_command, timeout=30, elapsed=30<br class="">mpr0: Reinitializing controller<br class="">mpr_config_get_ioc_pg8: request for header completed with error 0<br class="">mpr_config_get_ioc_pg8: request for header completed with error 16<br class="">mpr_config_get_ioc_pg8: request for header completed with error 16<br class="">mpr_config_get_ioc_pg8: request for header completed with error 16<br class="">mpr_config_get_ioc_pg8: request for header completed with error 16<br class="">mpr_config_get_ioc_pg8: request for header completed with error 16<br class="">mpr_config_get_iounit_pg8: request for header completed with error 16<br class="">mpr_config_get_iounit_pg8: request for header completed with error 16<br class="">mpr_config_get_iounit_pg8: request for header completed with error 16<br class="">mpr_config_get_iounit_pg8: request for header completed with error 16<br class="">mpr_config_get_iounit_pg8: request for header completed with error 16<br class="">mpr_config_get_iounit_pg8: request for header completed with error 16<br class="">mpr_config_get_man_pg11: request for header completed with error 16<br class="">mpr_config_get_man_pg11: request for header completed with error 16<br class="">mpr_config_get_man_pg11: request for header completed with error 16<br class="">mpr_config_get_man_pg11: request for header completed with error 16<br class="">mpr_config_get_man_pg11: request for header completed with error 16<br class="">mpr_config_get_man_pg11: request for header completed with error 16<br class="">mpr_config_get_dpm_pg0: request for header completed with error 16<br class="">mpr_config_get_dpm_pg0: request for header completed with error 16<br class="">mpr_config_get_dpm_pg0: request for header completed with error 16<br class="">mpr_config_get_dpm_pg0: request for header completed with error 16<br class="">mpr0: Unfreezing SIM queue<br class="">mpr0: fault_state(0x40001500)!<br class="">mpr0: Timeout while writing doorbell<br class="">panic: mpr_iocfacts_allocate failed to get IOC Facts with error 6<br class=""><br class="">cpuid = 1<br class="">time = 1663771142<br class="">KDB: stack backtrace:<br class="">db_trace_self_wrapper() at 0xffffffff8040a9ab = db_trace_self_wrapper+0x2b/frame 0xfffffe02e759d6c0<br class="">vpanic() at 0xffffffff805f7ad1 = vpanic+0x151/frame 0xfffffe02e759d710<br class="">panic() at 0xffffffff805f7973 = panic+0x43/frame 0xfffffe02e759d770<br class="">mpr_iocfacts_allocate() at 0xffffffff8154bdad = mpr_iocfacts_allocate+0x15dd/frame 0xfffffe02e759d8e0<br class="">mpr_reinit() at 0xffffffff81549e0c = mpr_reinit+0x14c/frame 0xfffffe02e759d930<br class="">mpr_wait_command() at 0xffffffff8154fb69 = mpr_wait_command+0x1d9/frame 0xfffffe02e759d9b0<br class="">mpr_ioctl() at 0xffffffff8155a113 = mpr_ioctl+0x1de3/frame 0xfffffe02e759db40<br class="">devfs_ioctl() at 0xffffffff804bab8f = devfs_ioctl+0xaf/frame 0xfffffe02e759db90<br class="">vn_ioctl() at 0xffffffff806e9664 = vn_ioctl+0x1a4/frame 0xfffffe02e759dca0<br class="">devfs_ioctl_f() at 0xffffffff804bb21e = devfs_ioctl_f+0x1e/frame 0xfffffe02e759dcc0<br class="">kern_ioctl() at 0xffffffff80667b2d = kern_ioctl+0x26d/frame 0xfffffe02e759dd30<br class="">sys_ioctl() at 0xffffffff80667811 = sys_ioctl+0x101/frame 0xfffffe02e759de00<br class="">amd64_syscall() at 0xffffffff8092ae4c = amd64_syscall+0x10c/frame 0xfffffe02e759df30<br class="">fast_syscall_common() at 0xffffffff8090369b = fast_syscall_common+0xf8/frame 0xfffffe02e759df30<br class="">--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8dfe3a, rsp = 0x821837048, rbp = 0x4 ---<br class="">Uptime: 18h2m58s<br class="">Dumping 5338 out of 130903 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%<br class=""></div><div class=""><br class=""></div><div class="">Ken</div><div class="">— </div><div class="">Ken Merry</div><div class=""><a href="mailto:ken@FreeBSD.ORG" class="">ken@FreeBSD.ORG</a></div></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F4FC4172-7C8A-4F9D-A194-AEE1C7BD734A>