From owner-freebsd-scsi@freebsd.org Mon Mar 7 12:34:00 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBD739DB7EF for ; Mon, 7 Mar 2016 12:34:00 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 680913D2 for ; Mon, 7 Mar 2016 12:33:59 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id EA0579DD34A for ; Mon, 7 Mar 2016 13:33:50 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: FreeBSD 10.3 - nvme regression Message-Id: <5A6B5C6F-26D1-40C5-8CCF-26EB8F17C59A@sarenet.es> Date: Mon, 7 Mar 2016 13:33:50 +0100 To: FreeBSD-scsi Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2016 12:34:01 -0000 Hello, I am trying a SuperMicro server with NVME disks. The system boots = FreeBSD 10.2, panics when booting FreeBSD 10.3. It was compiled on March 7th and Revision 296191 is included. On 10.3 it=E2=80=99s crashing right after this line: nvme9: mem 0xfba10000-0xfba13fff irq 59 at = device 0.0 on pci134 with a panic.=20 panic: couldn=E2=80=99t find an APIC vector for IRQ 59. cpuid =3D 0 The backtrace is (sorry, copying from a screen video) #0 kdb_backtrace=3D0x60 #1 vpanic+0x126 #2 panic+0x43 #3 ioapic_disable_intr+0 #4 intr_add_handler+0xfb #5 nexus_setup_inter+0x8a #6 pci_setup_intr+0x33 #7 pci_setup_intr+0x33 #8 bus_setup_intr+0xac #9 nvme_ctrlr_configure_intx+0x88 #10 nvme_ctrlr_construct+0x407 #11 nvme_attach+0x20 #12 device_attach+0x43d #13 bus_generic_attach+0x2d #14 acpi_pci_attach+0x15c #15 device_attach+0x43d #16 bus_generic_attach+0x2d #17 acpi_pcib_attach+0x22c It said =E2=80=9CUptime 1s=E2=80=9D and did a cold reboot. dmesg.boot from 10.2 (the system in installed on a memory stick). root@ssd9:/usr/src/sys/dev/nvme # cat /var/run/dmesg.boot=20 Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz (2400.04-MHz K8-class = CPU) Origin=3D"GenuineIntel" Id=3D0x306f2 Family=3D0x6 Model=3D0x3f = Stepping=3D2 = Features=3D0xbfebfbff = Features2=3D0x7ffefbff,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TS= CDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=3D0x2c100800 AMD Features2=3D0x21 Structured Extended = Features=3D0x37ab XSAVE Features=3D0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory =3D 137438953472 (131072 MB) avail memory =3D 133409718272 (127229 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 8 cpu9 (AP): APIC ID: 9 cpu10 (AP): APIC ID: 10 cpu11 (AP): APIC ID: 11 cpu12 (AP): APIC ID: 12 cpu13 (AP): APIC ID: 13 cpu14 (AP): APIC ID: 14 cpu15 (AP): APIC ID: 15 cpu16 (AP): APIC ID: 16 cpu17 (AP): APIC ID: 17 cpu18 (AP): APIC ID: 18 cpu19 (AP): APIC ID: 19 cpu20 (AP): APIC ID: 20 cpu21 (AP): APIC ID: 21 cpu22 (AP): APIC ID: 22 cpu23 (AP): APIC ID: 23 cpu24 (AP): APIC ID: 24 cpu25 (AP): APIC ID: 25 cpu26 (AP): APIC ID: 26 cpu27 (AP): APIC ID: 27 cpu28 (AP): APIC ID: 28 cpu29 (AP): APIC ID: 29 cpu30 (AP): APIC ID: 30 cpu31 (AP): APIC ID: 31 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard random: initialized module_register_init: MOD_LOAD (vesa, 0xffffffff80db8eb0, 0) error 19 kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 cpu16: on acpi0 cpu17: on acpi0 cpu18: on acpi0 cpu19: on acpi0 cpu20: on acpi0 cpu21: on acpi0 cpu22: on acpi0 cpu23: on acpi0 cpu24: on acpi0 cpu25: on acpi0 cpu26: on acpi0 cpu27: on acpi0 cpu28: on acpi0 cpu29: on acpi0 cpu30: on acpi0 cpu31: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci255: on pcib0 pcib1: on acpi0 pci127: on pcib1 pcib2: port 0xcf8-0xcff on acpi0 pci0: on pcib2 pcib3: irq 26 at device 1.0 on pci0 pci1: on pcib3 ix0: = port 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq = 26 at device 0.0 on pci1 ix0: Using MSIX interrupts with 9 vectors ix0: Ethernet address: 0c:c4:7a:a3:24:1e ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: = port 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq = 28 at device 0.1 on pci1 ix1: Using MSIX interrupts with 9 vectors ix1: Ethernet address: 0c:c4:7a:a3:24:1f ix1: PCI Express Bus: Speed 5.0GT/s Width x8 pcib4: irq 32 at device 2.0 on pci0 pci2: on pcib4 nvme0: mem 0xc7610000-0xc7613fff irq 32 at = device 0.0 on pci2 pcib5: irq 32 at device 2.1 on pci0 pci3: on pcib5 nvme1: mem 0xc7510000-0xc7513fff irq 33 at = device 0.0 on pci3 pcib6: irq 32 at device 2.2 on pci0 pci4: on pcib6 pcib7: irq 40 at device 3.0 on pci0 pci5: on pcib7 nvme2: mem 0xc7410000-0xc7413fff irq 40 at = device 0.0 on pci5 pcib8: irq 40 at device 3.1 on pci0 pci6: on pcib8 nvme3: mem 0xc7310000-0xc7313fff irq 41 at = device 0.0 on pci6 pcib9: irq 40 at device 3.2 on pci0 pci7: on pcib9 nvme4: mem 0xc7210000-0xc7213fff irq 42 at = device 0.0 on pci7 pcib10: irq 40 at device 3.3 on pci0 pci8: on pcib10 nvme5: mem 0xc7110000-0xc7113fff irq 43 at = device 0.0 on pci8 pci0: at device 17.0 (no driver attached) ahci0: port = 0x7110-0x7117,0x7100-0x7103,0x70f0-0x70f7,0x70e0-0x70e3,0x7020-0x703f = mem 0xc7738000-0xc77387ff irq 16 at device 17.4 on pci0 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xc7700000-0xc770ffff irq = 19 at device 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) ehci0: mem 0xc7734000-0xc77343ff irq = 18 at device 26.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 pcib11: irq 16 at device 28.0 on pci0 pci9: on pcib11 pcib12: irq 18 at device 28.2 on pci0 pci10: on pcib12 pcib13: at device 0.0 on pci10 pci11: on pcib13 vgapci0: port 0x5000-0x507f mem = 0xc6000000-0xc6ffffff,0xc7000000-0xc701ffff irq 18 at device 0.0 on = pci11 vgapci0: Boot video device ehci1: mem 0xc7733000-0xc77333ff irq = 18 at device 29.0 on pci0 usbus2: EHCI version 1.0 usbus2 on ehci1 isab0: at device 31.0 on pci0 isa0: on isab0 ahci1: port = 0x7070-0x7077,0x7060-0x7063,0x7050-0x7057,0x7040-0x7043,0x7000-0x701f = mem 0xc7732000-0xc77327ff irq 16 at device 31.2 on pci0 ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich4: at channel 0 on ahci1 ahcich5: at channel 1 on ahci1 ahcich6: at channel 2 on ahci1 ahcich7: at channel 3 on ahci1 ahcich8: at channel 4 on ahci1 ahcich9: at channel 5 on ahci1 ahciem1: on ahci1 pcib14: on acpi0 pci128: on pcib14 pcib15: at device 0.0 on pci128 pci129: on pcib15 pcib16: irq 50 at device 1.0 on pci128 pci130: on pcib16 ix2: = port 0xf020-0xf03f mem 0xfb880000-0xfb8fffff,0xfb904000-0xfb907fff irq = 50 at device 0.0 on pci130 ix2: Using MSIX interrupts with 9 vectors ix2: Ethernet address: 0c:c4:7a:bd:70:26 ix2: PCI Express Bus: Speed 5.0GT/s Width x8 ix3: = port 0xf000-0xf01f mem 0xfb800000-0xfb87ffff,0xfb900000-0xfb903fff irq = 52 at device 0.1 on pci130 ix3: Using MSIX interrupts with 9 vectors ix3: Ethernet address: 0c:c4:7a:bd:70:27 ix3: PCI Express Bus: Speed 5.0GT/s Width x8 pcib17: irq 56 at device 2.0 on pci128 pci131: on pcib17 nvme6: mem 0xfbd10000-0xfbd13fff irq 56 at = device 0.0 on pci131 pcib18: irq 56 at device 2.1 on pci128 pci132: on pcib18 nvme7: mem 0xfbc10000-0xfbc13fff irq 57 at = device 0.0 on pci132 pcib19: irq 56 at device 2.2 on pci128 pci133: on pcib19 nvme8: mem 0xfbb10000-0xfbb13fff irq 58 at = device 0.0 on pci133 pcib20: irq 56 at device 2.3 on pci128 pci134: on pcib20 nvme9: mem 0xfba10000-0xfba13fff irq 59 at = device 0.0 on pci134 pcib21: irq 64 at device 3.0 on pci128 pci135: on pcib21 pcib22: irq 64 at device 3.2 on pci128 pci136: on pcib22 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 orm0: at iomem = 0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcafff on isa0 sc0: at flags 0x100 on isa0 sc0: CGA <16 virtual consoles, flags=3D0x300> vga0: at port 0x3d0-0x3db iomem 0xb8000-0xbffff on = isa0 ppc0: cannot reserve I/O port range est0: on cpu0 est1: on cpu1 est2: on cpu2 est3: on cpu3 est4: on cpu4 est5: on cpu5 est6: on cpu6 est7: on cpu7 est8: on cpu8 est9: on cpu9 est10: on cpu10 est11: on cpu11 est12: on cpu12 est13: on cpu13 est14: on cpu14 est15: on cpu15 est16: on cpu16 est17: on cpu17 est18: on cpu18 est19: on cpu19 est20: on cpu20 est21: on cpu21 est22: on cpu22 est23: on cpu23 est24: on cpu24 est25: on cpu25 est26: on cpu26 est27: on cpu27 est28: on cpu28 est29: on cpu29 est30: on cpu30 est31: on cpu31 random: unblocking device. usbus0: 5.0Gbps Super Speed USB v3.0 Timecounters tick every 1.000 msec usbus1: 480Mbps High Speed USB v2.0 usbus2: 480Mbps High Speed USB v2.0 ugen0.1: <0x8086> at usbus0 uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on = usbus0 ugen2.1: at usbus2 uhub1: on usbus2 ugen1.1: at usbus1 uhub2: on usbus1 ses0 at ahciem0 bus 0 scbus4 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus11 target 0 lun 0 ses1: SEMB S-E-S 2.00 device ses1: SEMB SES Device nvd0: NVMe namespace nvd0: 1907729MB (3907029168 512 byte sectors) nvd1: NVMe namespace nvd1: 1907729MB (3907029168 512 byte sectors) nvd2: NVMe namespace nvd2: 1907729MB (3907029168 512 byte sectors) nvd3: NVMe namespace nvd3: 1907729MB (3907029168 512 byte sectors) nvd4: NVMe namespace nvd4: 1907729MB (3907029168 512 byte sectors) nvd5: NVMe namespace nvd5: 1907729MB (3907029168 512 byte sectors) nvd6: NVMe namespace nvd6: 1907729MB (3907029168 512 byte sectors) nvd7: NVMe namespace nvd7: 1907729MB (3907029168 512 byte sectors) nvd8: NVMe namespace nvd8: 1907729MB (3907029168 512 byte sectors) nvd9: NVMe namespace nvd9: 1907729MB (3907029168 512 byte sectors) SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #28 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #16 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #27 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #31 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #17 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #29 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #30 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #18 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #21 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #24 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #23 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #22 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #25 Launched! SMP: AP CPU #19 Launched! SMP: AP CPU #26 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #20 Launched! Timecounter "TSC-low" frequency 1200022010 Hz quality 1000 Root mount waiting for: usbus2 usbus1 usbus0 uhub2: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub0: 21 ports with 21 removable, self powered ugen1.2: at usbus1 uhub3: = on usbus1 ugen2.2: at usbus2 uhub4: = on usbus2 uhub3: 6 ports with 6 removable, self powered uhub4: 8 ports with 8 removable, self powered Root mount waiting for: usbus2 usbus1 ugen2.3: at usbus2 uhub5: = on usbus2 uhub5: 4 ports with 3 removable, self powered Root mount waiting for: usbus2 usbus1 ugen2.4: at usbus2 ukbd0: = on usbus2 ugen1.3: at usbus1 umass0: on usbus1 umass0: SCSI over Bulk-Only; quirks =3D 0x0100 umass0:12:0:-1: Attached to scbus12 kbd0 at ukbd0 cd0 at umass-sim0 bus 0 scbus12 target 0 lun 0 cd0: Removable CD-ROM SCSI device cd0: 40.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present = - tray open cd0: quirks=3D0x10<10_BYTE_ONLY> ugen1.4: at usbus1 ukbd1: on usbus1 kbd2 at ukbd1 Root mount waiting for: usbus1 ugen1.5: at usbus1 umass1: on = usbus1 umass1: SCSI over Bulk-Only; quirks =3D 0x4000 umass1:13:1:-1: Attached to scbus13 Trying to mount root from ufs:/dev/da0s1a [rw]... da0 at umass-sim1 bus 1 scbus13 target 0 lun 0 mountroot: waiting for device /dev/da0s1a ... da0: Removable Direct Access SPC-2 SCSI device da0: 40.000MB/s transfers da0: 7681MB (15730688 512 byte sectors: 255H 63S/T 979C) da0: quirks=3D0x3 On 10.3=20