Date: Fri, 16 Jun 2000 11:49:49 -0700 (PDT) From: Jeff.Potter@Compaq.com To: freebsd-gnats-submit@FreeBSD.org Subject: i386/19338: ProLiant DL360 dual proc. locks when booting SMP kernel Message-ID: <20000616184949.BAC9437B96D@hub.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 19338 >Category: i386 >Synopsis: ProLiant DL360 dual proc. locks when booting SMP kernel >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Jun 16 11:50:00 PDT 2000 >Closed-Date: >Last-Modified: >Originator: Jeff Potter >Release: FreeBSD 4.0 - RELEASE >Organization: Compaq Computer Corp. >Environment: FreeBSD mm3.compaq.com 4.0-RELEASE 4.0 RELEASE #1: Mon Apr 10 00:51:54 CDT 2000 root@mm3.compaq.com:/usr/src/sys/compile/CPQ2 i386 >Description: Customer reported an SMP "lock up" issue with "...FreeBSD-4.0 Stable Snapshot (from around 31-5-2000)..." on a Dual 800MHZ processor Proliant DL360. Subsequently, we were able to reproduce "the lock" up issue in our lab with FreeBSD 4.0-RELEASE #1 version on a bare bone 667MHZ DL360 with 512MB RAM. We moved the drive from the DL360 to a DL380, and it booted the SMP kernel without issue. To verify the DL 360 hardware, we loaded and ran the RedHat 6.2 SMP kernel without issue. So, our focus moved to the differences between the DL360 & DL380 platforms with respect to SMP & APIC tables. The only obvious difference between the DL 380 & DL 360 is APIC ID for the BSPs is 1 & 3 respectively with the APs both set to ID 0. A summary of the main differnce is listed below: "The package on the [DL360] FC PGA (Pentium III Flat Pack) is different from the [DL380] SECC2 (Pentium III Slot 1). Intel ... required that the APIC IDs be 3 and 0 for a FCPGA dual processor design. The hardware controls this, not the ROM." Original E-Mail from customer listed below: > > Matthew, > > > > I have FreeBSD 4.0 running in Uniprocessor mode running on the DL360. It > > should not be a big stretch to get it to run SMP. <big grin> I will get > > a second processor, and try it as soon as time permits. My biggest issue > > was 1510 SCSI driver support. We found that in order to get the Symbios SCSI working it was necessary to remove the RAID card. > > Just curious: > > > > What APIC mode is being used on the DL380 (Full Table or Mapped)? > > What Version of FreeBSD is the customer running on the DL380? > > What SCSI controller is the customer using? We tried the DL360 in all three possible APIC modes, Full Table, Mapped, and disabled. with no luck. We were using a FreeBSD-4.0 Stable Snapshot (from around 31-5-2000) as we required a recent version for ida (Compaq RAID array adapter) support, which was not working for the 380 or 360 until the exisiting ida driver was "told" about the addresses etc for these machines by the one of the driver authors. Snapshots are available from ftp://releng4.freebsd.org/pub/FreeBSD/snapshots/i386/ We were (obviously :o) using the ida RAID adapter for disk support. There were no problems getting SMP to work on the 380, as we used the 'mptable' command to find out the correct number of APICs, BUSes, and INTRupts for compilation into the kernel. With the 360, it would pause at the point where the second CPU was launched,it would still respond to a <CTRL>-<ALT>-<DEL> key sequence though. Please contact me if you need any further information. Thanks, James Output from boot -hv DL360 -------------------------- connected Copyright (c) 1992-2000 The FreeBSD Project. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 4.0-20000531-STABLE #0: Thu Jun 1 16:03:10 BST 2000 root@fubar.noc.demon.net:/usr/src/sys/compile/CPQ Calibrating clock(s) ... TSC clock: 797430926 Hz, i8254 clock: 1193118 Hz CLK_USE_I8254_CALIBRATION not specified - using default frequency Timecounter "i8254" frequency 1193182 Hz CLK_USE_TSC_CALIBRATION not specified - using old calibration method CPU: Pentium III/Pentium III Xeon (797.48-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x683 Stepping = 3 Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,XMM> real memory = 1207943168 (1179632K bytes) Physical memory chunk(s): 0x00001000 - 0x0009efff, 647168 bytes (158 pages) 0x002c3000 - 0x47ff3fff, 1205014528 bytes (294193 pages) avail memory = 1170939904 (1143496K bytes) Programming 35 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 IOAPIC #0 intpin 24 -> irq 2 SMP: CPU0 apic_initialize(): lint0: 0x00000700 lint1: 0x00010400 TPR: 0x00000010 SVR: 0x000001ff FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 3, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 8, version: 0x00220011, at 0xfec00000 bios32: Found BIOS32 Service Directory header at 0xc00ffee0 bios32: Entry = 0xf0000 (c00f0000) Rev = 0 Len = 1 pcibios: PCI BIOS entry at 0x94 Other BIOS signatures found: ACPI: 000f4f90 Preloaded elf kernel "kernel" at 0xc02a7000. Pentium Pro MTRR support enabled SMP: CPU0 bsp_apic_configure(): lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000010 SVR: 0x000001ff pci_open(1): mode 1 addr port (0x0cf8) is 0x00000000 pci_open(1a): mode1res=0x80000000 (0x80000000) pci_cfgcheck: device 0 [class=060000] [hdr=80] is there (id=00091166) npx0: <math processor> on motherboard npx0: INT 16 interface pci_open(1): mode 1 addr port (0x0cf8) is 0x00000000 pci_open(1a): mode1res=0x80000000 (0x80000000) pci_cfgcheck: device 0 [class=060000] [hdr=80] is there (id=00091166) pcib0: <RCC LE host to PCI bridge> on motherboard found-> vendor=0x1166, dev=0x0009, revid=0x05 class=06-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 found-> vendor=0x1166, dev=0x0009, revid=0x05 class=06-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 Freeing (NOT implemented) redirected PCI irq 5. found-> vendor=0x1000, dev=0x0010, revid=0x02 class=01-04-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 intpin=a, irq=19 map[10]: type 1, range 32, base 00002000, size 8 map[14]: type 1, range 32, base c5000000, size 24 map[18]: type 1, range 32, base c4000000, size 24 found-> vendor=0x1002, dev=0x4756, revid=0x7a class=03-00-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 found-> vendor=0x0e11, dev=0xa0f0, revid=0x00 class=08-80-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 intpin=a, irq=255 map[10]: type 1, range 32, base 00001800, size 8 map[14]: type 1, range 32, base c3ffef00, size 8 found-> vendor=0x8086, dev=0x0960, revid=0x05 class=06-04-00, hdrtype=0x01, mfdev=1 subordinatebus=1 secondarybus=1 Freeing (NOT implemented) redirected PCI irq 3. found-> vendor=0x8086, dev=0x1960, revid=0x05 class=05-80-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 intpin=a, irq=21 map[10]: type 1, range 32, base c0fc0000, size 18 found-> vendor=0x1166, dev=0x0200, revid=0x4f class=06-01-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 found-> vendor=0x1166, dev=0x0211, revid=0x00 class=01-01-8a, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 map[20]: type 1, range 32, base 00002800, size 4 pci0: <PCI bus> on pcib0 ida0: <NCR/Compaq Integrated Array controller> port 0x2000-0x20ff mem 0xc4000000-0xc4ffffff,0xc5000000-0xc5ffffff irq 19 at device 1.0 on pci0 ida0: drives=1 firm_rev=1.40 idad0: <Compaq Logical Drive> on ida0 idad0: 8670MB (17756160 sectors), blocksize=512 Creating DISK idad0 pci0: <ATI Mach64-GV graphics accelerator> (vendor=0x1002, dev=0x4756) at 3.0 pci0: <unknown card> (vendor=0x0e11, dev=0xa0f0) at 4.0 pcib1: <PCI to PCI bridge (vendor=8086 device=0960)> at device 5.0 on pci0 found-> vendor=0x1002, dev=0x4756, revid=0x7a class=03-00-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 map[10]: type 1, range 32, base c2000000, size 24 map[14]: type 1, range 32, base 00003000, size 8 map[18]: type 1, range 32, base c6aff000, size 12 pci1: <PCI bus> on pcib1 pci1: <ATI Mach64-GV graphics accelerator> (vendor=0x1002, dev=0x4756) at 0.0 pci0: <unknown card> (vendor=0x8086, dev=0x1960) at 5.1 irq 21 isab0: <PCI to ISA bridge (vendor=1166 device=0200)> at device 15.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Unknown PCI ATA controller (generic mode)> port 0x2800-0x280f at device 15.1 on pci0 ata0: iobase=0x01f0 altiobase=0x03f6 bmaddr=0x2800 ata0: mask=03 status0=20 status1=30 ata0: mask=03 status0=20 status1=30 ata0: devices = 0x0 ata0: probe allocation failed ata1: iobase=0x0170 altiobase=0x0376 bmaddr=0x2808 ata1: mask=03 status0=50 status1=01 ata1: mask=03 status0=00 status1=01 ata1: devices = 0xc ata1: at 0x170 irq 15 on atapci0 pci_open(1): mode 1 addr port (0x0cf8) is 0x00000000 pci_open(1a): mode1res=0x80000000 (0x80000000) pci_cfgcheck: device 0 [class=060000] [hdr=80] is there (id=00091166) pcib3: <RCC LE host to PCI bridge> on motherboard Freeing (NOT implemented) redirected PCI irq 7. found-> vendor=0x8086, dev=0x1229, revid=0x08 class=02-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 intpin=a, irq=17 map[10]: type 1, range 32, base c6fff000, size 12 map[14]: type 1, range 32, base 00004000, size 6 map[18]: type 1, range 32, base c6e00000, size 20 Freeing (NOT implemented) redirected PCI irq 7. found-> vendor=0x0000, dev=0x0000, revid=0x00 class=07-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 intpin=a, irq=17 map[10]: type 1, range 32, base 00004040, size 3 map[14]: type 1, range 32, base c6dff000, size 12 Freeing (NOT implemented) redirected PCI irq 10. found-> vendor=0x8086, dev=0x1229, revid=0x08 class=02-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 intpin=a, irq=2 map[10]: type 1, range 32, base c6dfe000, size 12 map[14]: type 1, range 32, base 00004080, size 6 map[18]: type 1, range 32, base c6c00000, size 20 Freeing (NOT implemented) redirected PCI irq 10. found-> vendor=0x0000, dev=0x0000, revid=0x00 class=07-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 intpin=a, irq=2 map[10]: type 1, range 32, base 000040c0, size 3 map[14]: type 1, range 32, base c6bff000, size 12 pci3: <PCI bus> on pcib3 fxp0: <Intel EtherExpress Pro 10/100B Ethernet> port 0x4000-0x403f mem 0xc6e00000-0xc6efffff,0xc6fff000-0xc6ffffff irq 17 at device 4.0 on pci3 fxp0: Ethernet address 00:01:fa:ff:f4:f3 bpf: fxp0 attached pci3: <unknown card> (vendor=0x0000, dev=0x0000) at 4.1 irq 17 fxp1: <Intel EtherExpress Pro 10/100B Ethernet> port 0x4080-0x40bf mem 0xc6c00000-0xc6cfffff,0xc6dfe000-0xc6dfefff irq 2 at device 5.0 on pci3 fxp1: Ethernet address 00:01:fa:ff:f5:05 bpf: fxp1 attached pci3: <unknown card> (vendor=0x0000, dev=0x0000) at 5.1 irq 2 eisa0: <EISA bus> on motherboard mainboard0: <CPQ0685 (System Board)> on eisa0 slot 0 Trying Read_Port at 203 Trying Read_Port at 243 Trying Read_Port at 283 Trying Read_Port at 2c3 Trying Read_Port at 303 Trying Read_Port at 343 Trying Read_Port at 383 Trying Read_Port at 3c3 ata-: ata0 exists, using next available unit number ata-: ata1 exists, using next available unit number isa_probe_children: disabling PnP devices isa_probe_children: probing non-PnP devices fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> on fdc0 drive 0 ata2: iobase=0x01f0 altiobase=0x03f6 bmaddr=0x0000 ata2: mask=03 status0=20 status1=30 ata2: mask=03 status0=20 status1=30 ata2: devices = 0x0 ata2: probe allocation failed atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 atkbd: the current kbd controller command byte 0065 atkbd: keyboard ID 0x41ab (2) kbdc: RESET_KBD return code:00fa kbdc: RESET_KBD status:00aa kbd0: atkbd0, AT 101/102 (2), config:0x0, flags:0x1d0000 psm0: current command byte:0065 kbdc: TEST_AUX_PORT status:0000 kbdc: RESET_AUX return code:00fa kbdc: RESET_AUX status:00aa kbdc: RESET_AUX ID:0000 psm: status 00 02 64 psm: status 00 00 64 psm: status 00 03 64 psm: status 00 03 64 psm: data 08 00 00 psm: data 08 00 00 psm: status 00 02 64 psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: model IntelliMouse, device ID 3-00, 3 buttons psm0: config:00000000, flags:00000000, packet size:4 psm0: syncmask:08, syncbits:08 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 fb0: vga0, vga, type:VGA (5), flags:0x7007f fb0: port:0x3c0-0x3df, crtc:0x3d4, mem:0xa0000 0x20000 fb0: init mode:24, bios mode:3, current mode:24 fb0: window:0xc00b8000 size:32k gran:32k, buf:0 size:32k VGA parameters upon power-up 50 18 10 00 00 00 03 00 02 67 5f 4f 50 82 55 81 bf 1f 00 4f 0d 0e 00 00 05 50 9c 8e 8f 28 1f 96 b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff VGA parameters in BIOS for mode 24 50 18 10 00 10 00 03 00 02 67 5f 4f 50 82 55 81 bf 1f 00 4f 0d 0e 00 00 00 00 9c 8e 8f 28 1f 96 b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff EGA/VGA parameters to be used for mode 24 50 18 10 00 10 00 03 00 02 67 5f 4f 50 82 55 81 bf 1f 00 4f 0d 0e 00 00 00 00 9c 8e 8f 28 1f 96 b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff sc0: <System console> on isa0 sc0: VGA <16 virtual consoles, flags=0x0> sc0: fb0, kbd0, terminal emulator: sc (syscons terminal) sio0: irq maps: 0x8041 0x8051 0x8041 0x8041 sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A, console sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: irq maps: 0x8041 0x8041 0x8041 0x8041 sio1: probe failed test(s): 0 1 2 4 6 7 9 ppc0: parallel port not found. isa_probe_children: probing PnP devices SMP: enabled INTs: 1, 2, 4, 6, 12, 15, 17, 19, apic_imen: 0x00f56fa9 BIOS Geometries: 0:03fffe20 0..1023=1024 cylinders, 0..254=255 heads, 1..32=32 sectors 0 accounted for Device configuration finished. APIC_IO: routing 8254 via IOAPIC #0 intpin 2 bpf: lo0 attached SMP: AP CPU #1 Launched! SMP: CPU1 apic_initialize(): lint0: 0x00010700 lint1: 0x00010400 TPR: 0x00000010 SVR: 0x000001ff A three fingered salute then adds the following boot() called on cpu#1 Uptime: 0s Rebooting... cpu_reset called on cpu#1 cpu_reset: Stopping other CPUs cpu_reset: Restarting BSP pu_reset_proxy: Grabbed mp locck uf_orr sBeSP: BSP did not grab mp lock --------------------------------- Output from mptable when booting from the kernel.GENERIC DL360 =============================================================================== MPTable, version 2.0.15 ------------------------------------------------------------------------------- MP Floating Pointer Structure: location: BIOS physical address: 0x000f4ff0 signature: '_MP_' length: 16 bytes version: 1.4 checksum: 0x9f mode: Virtual Wire ------------------------------------------------------------------------------- MP Config Table Header: physical address: 0x000f29c9 signature: 'PCMP' base table length: 324 version: 1.4 checksum: 0x88 OEM ID: 'COMPAQ ' Product ID: 'PROLIANT ' OEM table pointer: 0x00000000 OEM table size: 0 entry count: 28 local APIC address: 0xfee00000 extended table length: 184 extended table checksum: 82 ------------------------------------------------------------------------------- MP Config Base Table Entries: -- Processors: APIC ID Version State Family Model Step Flags 3 0x10 BSP, usable 6 8 3 0x383fbff 0 0x10 AP, usable 6 8 3 0x383fbff -- Bus: Bus ID Type 0 PCI 3 PCI 9 ISA -- I/O APICs: APIC ID Version State Address 8 0x11 usable 0xfec00000 -- I/O Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# INT active-lo level 0 5:A 8 21 INT active-lo level 0 5:B 8 20 INT active-lo level 0 5:C 8 21 INT active-lo level 0 5:D 8 20 INT active-lo level 3 6:A 8 23 INT active-lo level 3 6:B 8 22 INT active-lo level 3 6:C 8 23 INT active-lo level 3 6:D 8 22 INT active-lo level 0 1:A 8 19 INT active-lo level 0 1:B 8 18 INT active-lo level 3 4:A 8 17 INT active-lo level 3 5:A 8 24 INT active-hi edge 9 1 8 1 INT active-hi edge 9 0 8 2 INT active-hi edge 9 4 8 4 INT active-hi edge 9 6 8 6 INT active-hi edge 9 8 8 8 INT active-hi edge 9 9 8 9 INT active-hi edge 9 11 8 11 INT active-hi edge 9 12 8 12 INT active-lo level 9 13 8 13 INT active-hi edge 9 14 8 14 INT active-hi edge 9 15 8 15 -- Local Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# ExtINT conforms conforms 9 0 255 0 NMI conforms conforms 9 0 255 1 ------------------------------------------------------------------------------- MP Config Extended Table Entries: Extended Table HOSED! -- James Guard System Administrator Interactive Services Demon Internet, Demon@THUS http://www.games.demon.net/ http://www.demon.net/ >How-To-Repeat: Configure an FreeBSD 4.0 RELEASE #1 SMP Kernel on a ProLiant DL360 with two FC PGA Pentium III processors configured for the Linux operating system, and attempt to boot it. >Fix: Possibly: FreeBSD 4.0 might "assume" that APIC ID's be sequential in ascending order. This is not a requirement per the Intel MP spec. In fact, in a current ProLiant MP boxes (with 3 or more cpu's), if a processor were to fail, the ROM would map it out, and the system would still try to boot with non-sequential APIC ID's (Compaq Hot Spare Boot CPU Patent). If this is the problem, then it may not be limited to FC PGA designs. Please contact me if you need more information. >Release-Note: >Audit-Trail: >Unformatted: To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20000616184949.BAC9437B96D>