From owner-freebsd-bugs@freebsd.org Sun Jul 1 15:56:48 2018 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D96C2FF9FCE for ; Sun, 1 Jul 2018 15:56:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 669E473FAC for ; Sun, 1 Jul 2018 15:56:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 29D87FF9FCD; Sun, 1 Jul 2018 15:56:47 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E020EFF9FCC for ; Sun, 1 Jul 2018 15:56:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 64D0C73FAB for ; Sun, 1 Jul 2018 15:56:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id BE34D1F038 for ; Sun, 1 Jul 2018 15:56:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w61Fujlf068227 for ; Sun, 1 Jul 2018 15:56:45 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w61Fujji068226 for bugs@FreeBSD.org; Sun, 1 Jul 2018 15:56:45 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: =?UTF-8?B?W0J1ZyAyMjk0NTRdIFN5c3RlbSBlbmFibGVkIGZvciBGQyB0YXJn?= =?UTF-8?B?ZXQgbW9kZSB1c2luZyBRTE9HSUMgSEJBIHBhbmljcyB3aXRoIOKAnGZhdWx0?= =?UTF-8?B?IGNvZGUgPSBzdXBlcnZpc29yIHdyaXRlIGRhdGEsIHBhZ2Ugbm90IHByZXNl?= =?UTF-8?B?bnQgLSBGYXRhbCB0cmFwIDEyOiBwYWdlIGZhdWx0IHdoaWxlIGluIGtlcm5l?= =?UTF-8?B?bCBtb2Rl4oCd?= Date: Sun, 01 Jul 2018 15:56:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: setsquare@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Jul 2018 15:56:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229454 Bug ID: 229454 Summary: System enabled for FC target mode using QLOGIC HBA panics with =E2=80=9Cfault code =3D supervisor write da= ta, page not present - Fatal trap 12: page fault while in kernel mode=E2=80=9D Product: Base System Version: 11.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: setsquare@gmail.com Hi, Pulling my hair out with this one. System overview: Intel(R) Xeon(R) CPU E3-1245 v6 @ 3.70GHz 32GB ECC RAM Quad port QLOGIC PCIe adaptor listed as =E2=80=9CQlogic ISP 2432 PCI FC-AL = Adapter=E2=80=9D (1 of the ports is not connected, but 3 are) No fibre channel switches are being used, connectivity is directly from the= FC HBA on the FreeBSD storage server to 3x ESXi hosts. FC Target devices for CTL, tried both physical block devices and ZVOLS on Z= FS, the problem occurs with both. FC Target devices for CTL are SAMSUNG SSD=E2=80=99s (EVO 840/850) & Crucial CT240M500SSD1 Problem does not seem to occur when using ISCSI using the same target devic= es for CTL, only seen the panic when using FC on the same server hardware. 3 LUNs were presented to the ESXi hosts (issue occurs with direct block dev= ice and ZVOL backing) Kernel compiled with following in config file: include GENERIC ident FCTARGETMODE device ispfw options ISP_TARGET_MODE options ISP_DEFAULT_ROLES=3D1 Problem description: The FreeBSD storage system as a FC target seems to function without issue u= sing a single ESXi host as an initiator (from what I can tell, no panics). After additional ESXi hosts are powered on and the system becomes more active, hi= gher load, storage vMotions a system panic occurs with the following. I have only ever noticed the panics when there are multiple systems accessing the target concurrently. Under FreeBSD 10.4-RELEASE-p8 (Seems to have more info in crash textdump): Fatal trap 12: page fault while in kernel mode cpuid =3D 2; apic id =3D 02 fault virtual address =3D 0x0 fault code =3D supervisor write data, page not present instruction pointer =3D 0x20:0xffffffff8057a028 stack pointer =3D 0x28:0xfffffe07c69fb230 frame pointer =3D 0x28:0xfffffe07c69fb2b0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 12 (irq270: isp3:0) trap number =3D 12 panic: page fault cpuid =3D 2 KDB: stack backtrace: #0 0xffffffff809d0340 at kdb_backtrace+0x60 #1 0xffffffff80990c16 at vpanic+0x126 #2 0xffffffff80990ae3 at panic+0x43 #3 0xffffffff80db03cd at trap_fatal+0x35d #4 0xffffffff80db06e8 at trap_pfault+0x308 #5 0xffffffff80dafd4a at trap+0x47a #6 0xffffffff80d9551c at calltrap+0x8 #7 0xffffffff80578702 at isp_async+0x13b2 #8 0xffffffff8058c9af at isp_target_notify+0xf9f #9 0xffffffff80570fd4 at isp_intr_atioq+0xa4 #10 0xffffffff805896bb at isp_pci_run_isr_2400+0xab #11 0xffffffff8057ab23 at isp_platform_intr+0x53 #12 0xffffffff8095a0a9 at intr_event_execute_handlers+0xb9 #13 0xffffffff8095a516 at ithread_loop+0x96 #14 0xffffffff8095796a at fork_exit+0x9a #15 0xffffffff80d95a5e at fork_trampoline+0xe Uptime: 20h32m44s Dumping 1447 out of 32633 MB:..2%..12%..21%..31%..41%..51%..61%..71%..81%..= 91% Reading symbols from /boot/kernel/ctl.ko.symbols...done. Loaded symbols for /boot/kernel/ctl.ko.symbols Reading symbols from /boot/kernel/iscsi.ko.symbols...done. Loaded symbols for /boot/kernel/iscsi.ko.symbols #0 doadump (textdump=3D) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=3D) at pcpu.h:219 #1 0xffffffff80990833 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #2 0xffffffff80990c55 in vpanic (fmt=3D,=20 ap=3D) at /usr/src/sys/kern/kern_shutdown.c:889 #3 0xffffffff80990ae3 in panic (fmt=3D0x0) at /usr/src/sys/kern/kern_shutdown.c:818 #4 0xffffffff80db03cd in trap_fatal (frame=3D,=20 eva=3D) at /usr/src/sys/amd64/amd64/trap.c:858 #5 0xffffffff80db06e8 in trap_pfault (frame=3D0xfffffe07c69fb180,=20 usermode=3D) at /usr/src/sys/amd64/amd64/trap.c:681 #6 0xffffffff80dafd4a in trap (frame=3D0xfffffe07c69fb180) at /usr/src/sys/amd64/amd64/trap.c:447 #7 0xffffffff80d9551c in calltrap () at /usr/src/sys/amd64/amd64/exception.S:238 #8 0xffffffff8057a028 in isp_handle_platform_atio7 ( isp=3D, aep=3D0xfffffe07c69fb5c0) at /usr/src/sys/dev/isp/isp_freebsd.c:1931 #9 0xffffffff80578702 in isp_async (isp=3D0xfffff80009723800,=20 cmd=3D) at /usr/src/sys/dev/isp/isp_freebsd.c:3941 #10 0xffffffff8058c9af in isp_target_notify (isp=3D,=20 vptr=3D, optrp=3D) at /usr/src/sys/dev/isp/isp_target.c:735 #11 0xffffffff80570fd4 in isp_intr_atioq (isp=3D0xfffff80009723800) at /usr/src/sys/dev/isp/isp.c:4943 #12 0xffffffff805896bb in isp_pci_run_isr_2400 (isp=3D0xfffff80009723800) at /usr/src/sys/dev/isp/isp_pci.c:1167 #13 0xffffffff8057ab23 in isp_platform_intr (arg=3D0xfffff80009723800) at /usr/src/sys/dev/isp/isp_freebsd.c:4155 #14 0xffffffff8095a0a9 in intr_event_execute_handlers ( p=3D, ie=3D0xfffff80009738700) at /usr/src/sys/kern/kern_intr.c:1264 #15 0xffffffff8095a516 in ithread_loop (arg=3D0xfffff80009750240) at /usr/src/sys/kern/kern_intr.c:1277 #16 0xffffffff8095796a in fork_exit ( callout=3D0xffffffff8095a480 , arg=3D0xfffff80009750240,= =20 frame=3D0xfffffe07c69fb9c0) at /usr/src/sys/kern/kern_fork.c:1032 #17 0xffffffff80d95a5e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:613 #18 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) Under FreeBSD 11.1-RELEASE-p10: Fatal trap 12: page fault while in kernel mode cpuid =3D 2; apic id =3D 02 fault virtual address =3D 0x0 fault code =3D supervisor write data, page not present instruction pointer =3D 0x20:0xffffffff805b3da0 stack pointer =3D 0x28:0xfffffe07c5d4e470 frame pointer =3D 0x28:0xfffffe07c5d4e4e0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 12 (irq270: isp3:0) trap number =3D 12 panic: page fault cpuid =3D 2 KDB: stack backtrace: #0 0xffffffff80ab97b7 at kdb_backtrace+0x67 #1 0xffffffff80a77896 at vpanic+0x186 #2 0xffffffff80a77703 at panic+0x43 #3 0xffffffff80eef192 at trap_fatal+0x322 #4 0xffffffff80eef1eb at trap_pfault+0x4b #5 0xffffffff80eee948 at trap+0x2a8 #6 0xffffffff80ecf950 at calltrap+0x8 #7 0xffffffff805b26bb at isp_async+0x156b #8 0xffffffff805c7ebe at isp_target_notify+0x186e #9 0xffffffff805aa733 at isp_intr_atioq+0xa3 #10 0xffffffff805c61f4 at isp_pci_run_isr_2400+0x84 #11 0xffffffff805b4a01 at isp_platform_intr+0x41 #12 0xffffffff80a3dedc at intr_event_execute_handlers+0xec #13 0xffffffff80a3e1c6 at ithread_loop+0xd6 #14 0xffffffff80a3b535 at fork_exit+0x85 #15 0xffffffff80ed081e at fork_trampoline+0xe Uptime: 3h29m17s What I have tried so far that hasn=E2=80=99t resolved the issue: Changing the FC ports on the HBA in the FreeBSD server. Set DataMover.HardwareAcceleratedMove, DataMover.HardwareAcceleratedInit and VMFS3.HardwareAcceleratedLocking on the ESXi hosts to 0 (disabled) Changed PCIe slot the Quad port HBA is using the FreeBSD server Disabled Hyper threading on the FreeBSD server Ran Memtest86+ on the FreeBSD server memory (no errors found) Use ZVOLS instead of direct block devices for ctld Going forward: Happy to try help as much as I can to help troubleshoot this going forward.= Can provide assisted remote access to the system that displays the issue if required. I can also get dumps if required, just let me know. Noticed this on the FreeNAS issue list, not sure if it=E2=80=99s related at= all https://redmine.ixsystems.com/issues/32370 --=20 You are receiving this mail because: You are the assignee for the bug.=