Date: Wed, 13 Aug 2003 10:59:19 +0200 (CEST) From: "Hartmann, O." <ohartman@klima.physik.uni-mainz.de> To: freebsd-smp@freebsd.org Cc: freebsd-stable@freebsd.org Subject: 5.1-R-p2 crashes on SMP with AMI RAID and Intel 1000/Pro Message-ID: <20030813103509.Q49991@mail.physik.uni-mainz.de>
next in thread | raw e-mail | index | archive | help
Dear Sirs.
It seems to me a never ending story. We run a box with a TYAN Thunder
2500 Dual SMP mainboard, 2GB ECC Tyan certified memory, AMI Enterprise
1600 RAID adapter and additional Intel 1000/Pro server type (64 bit)
GBit LAN NIC. With FreeBSD 4.8 this was stable, but to achive this
state was really hard! It is a story similar to that what happend when
we changed towards FreeBSD 5.1-RELEASE-p2 on this machine.
It seems to be highly dependend in which PCI slot several cards are
attached, so I will report this here also.
Phenomenon:
After a while the machine was running, the SMP kernel reboots
spontanously. This is when heavy IO is done, compiling or, when in the
morning time our department gets up and our staff connects to the samba
server.
Dependend on which devices are switched on or off by BIOS, the kernel
freezes at the stage when the amr0 RAID got recognized. I can avoid this
by enabling the built in NIC (fxp0). I can force this by putting the em0
NIC into another slot, for instance in the one remaining 64BIT/66MHz
slot (which should be a separate bus).
This 'game' was identical to that I had with FreeBSD 4.X - 4.8 and I
found out, that putting an additional NIC into PCI slot No. 2 (counted
from AGP slot on) made things clear, but using both NICs together
(either additional fxp0 or the new em0) remains the systems completely
unstable.
In FreeBSD 5.1-RELEASE-p2 and especially in FreeBSD 5.1-CURRENT this
'gambling' seems to reach its climax. My kernel is built up with
SCHED_4BSD because SCHED_ULE and ADAPTIVE_MUTEXES crashes immediately
the same way as described (running a while, then coredumping or freeze
at the stage after the amr0-RAID showed up in the kernel boot messages,
see the dmesg output below).
I'm not an hardware expert, but all this wierd stuff looks like to me to be
a IRQ routing problem. I fiddled around with many hand-assigned IRQ configurations,
but nothing helped. Either the Intel 1000/Pro or the AMI RAID causing
problems in the TYAN Thunder 2500 SMP environment.
We have also a SMP machine with a similar hardware, based on an ASUS CV4X-D,
AMI Elite 1600 RAID controller and the same Intel em0 1GBit NIC. OS is
FreeBSD 4.8 and this system never had any problem!
I feel a little bit helpless this moment, because I think I tried every trick
and something seems to be wrong with the combination TYAN Thunder 2500 and FreeBSD
5.X SMP. It is also very courios, that a kernel without SMP/IO_APIC freezes after
booting at the same place (amr0 RAID recognition).
Is there any help outside?
I attach the kernel config file and the dmesg output. Please note: I disabled both
serial ports, the parallel port, sound and usb to get additional IRQs. But I have to
enable the built in NIC to get a bootable, but instable FreeBSD 5.1-R box.
====================================
DMESG output
====================================
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 5.1-RELEASE-p2 #14: Wed Aug 13 09:47:00 CEST 2003
root@atmos.physik.uni-mainz.de:/usr/obj/usr/src/sys/ATMOS
Preloaded elf kernel "/boot/kernel/kernel" at 0xc0458000.
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 868644793 Hz
CPU: Intel Pentium III (868.64-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x683 Stepping = 3
Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory = 2147483648 (2048 MB)
avail memory = 2085625856 (1989 MB)
Programming 16 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
Programming 16 pins in IOAPIC #1
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): apic id: 1, version: 0x00040011, at 0xfee00000
cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000
io0 (APIC): apic id: 2, version: 0x000f0011, at 0xfec00000
io1 (APIC): apic id: 3, version: 0x000f0011, at 0xfec01000
netsmb_dev: loaded
Pentium Pro MTRR support enabled
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcibios: BIOS version 2.10
Using $PIR table, 12 entries at 0xc00fdf00
pcib0: <Host to PCI bridge> at pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
IOAPIC #1 intpin 13 -> irq 2
IOAPIC #1 intpin 12 -> irq 16
IOAPIC #1 intpin 2 -> irq 17
IOAPIC #1 intpin 7 -> irq 18
pcib1: <PCI-PCI bridge> at device 0.1 on pci0
pci1: <PCI bus> on pcib1
IOAPIC #1 intpin 1 -> irq 19
pci1: <display, VGA> at device 0.0 (no driver attached)
sym0: <896> port 0xf800-0xf8ff mem 0xfeafe000-0xfeafffff,0xfeafac00-0xfeafafff irq 2 at device 1.0 on pci0
sym0: Symbios NVRAM, ID 7, Fast-40, SE, parity checking
sym0: open drain IRQ line driver, using on-chip SRAM
sym0: using LOAD/STORE-based firmware.
sym0: handling phase mismatch from SCRIPTS.
sym1: <896> port 0xf400-0xf4ff mem 0xfeafc000-0xfeafdfff,0xfeafa800-0xfeafabff irq 16 at device 1.1 on pci0
sym1: Symbios NVRAM, ID 7, Fast-40, LVD, parity checking
sym1: open drain IRQ line driver, using on-chip SRAM
sym1: using LOAD/STORE-based firmware.
sym1: handling phase mismatch from SCRIPTS.
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.5.31> port 0xfcc0-0xfcff mem 0xfeac0000-0xfeadffff irq 17 at device 4.0 on pci0
em0: Speed:1000 Mbps Duplex:Full
fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port 0xfc40-0xfc7f mem 0xfe900000-0xfe9fffff,0xfeaf9000-0xfeaf9fff irq 18 at device 7.0 on pci0
fxp0: Ethernet address 00:e0:81:00:f0:d7
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <PCI-ISA bridge> port 0x500-0x50f at device 15.0 on pci0
isa0: <ISA bus> on isab0
pci0: <mass storage, ATA> at device 15.1 (no driver attached)
pcib2: <ServerWorks host to PCI bridge> at pcibus 2 on motherboard
pci2: <PCI bus> on pcib2
pcib3: <PCI-PCI bridge> at device 2.0 on pci2
pci3: <PCI bus> on pcib3
IOAPIC #1 intpin 11 -> irq 20
IOAPIC #1 intpin 8 -> irq 21
pcib4: <PCI-PCI bridge> at device 0.0 on pci3
pci4: <PCI bus> on pcib4
IOAPIC #1 intpin 10 -> irq 22
amr0: <LSILogic MegaRAID> mem 0xf0000000-0xf3ffffff irq 22 at device 0.0 on pci4
amr0: <LSILogic MegaRAID Enterprise 1600> Firmware G170, BIOS F316, 64MB RAM
pci3: <mass storage, SCSI> at device 1.0 (no driver attached)
pci3: <mass storage, SCSI> at device 2.0 (no driver attached)
orm0: <Option ROMs> at iomem 0xca000-0xcdfff,0xc0000-0xc9fff on isa0
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> at port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x64,0x60 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model IntelliMouse, device ID 3
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <8 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
ppc0: parallel port not found.
unknown: <PNP0303> can't assign resources (port)
psmcpnp0: irq resource info is missing; assuming irq 12
unknown: <PNP0700> can't assign resources (port)
ppc1: parallel port not found.
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin 2
APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0
Timecounters tick every 1.000 msec
ipfw2 initialized, divert enabled, rule-based forwarding enabled, default to deny, logging unlimited
DUMMYNET initialized (011031)
Waiting 5 seconds for SCSI devices to settle
(noperiph:sym0:0:-1:-1): SCSI BUS reset delivered.
(noperiph:sym1:0:-1:-1): SCSI BUS reset delivered.
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 245014MB (501788672 sectors) RAID 5 (optimal)
===> freezing here!
sa0 at sym1 bus 0 target 5 lun 0
sa0: <HP C5713A H910> Removable Sequential Access SCSI-2 device
sa0: 40.000MB/s transfers (20.000MHz, offset 31, 16bit)
ch0 at sym1 bus 0 target 5 lun 1
ch0: <HP C5713A H910> Removable Changer SCSI-2 device
ch0: 40.000MB/s transfers (20.000MHz, offset 31, 16bit)
ch0: 6 slots, 1 drive, 0 pickers, 0 portals
SMP: AP CPU #1 Launched!
Mounting root from ufs:/dev/amrd0s1a
cd0 at sym0 bus 0 target 3 lun 0
cd0: <TEAC CD-ROM CD-532S 1.0A> Removable CD-ROM SCSI-2 device
cd0: 20.000MB/s transfers (20.000MHz, offset 16)
cd0: Attempt to query device size failed: NOT READY, Medium not present
========================
KERNEL config file
========================
machine i386
cpu I686_CPU
ident ATMOS
options SMP # Symmetric MultiProcessor Kernel
options APIC_IO # Symmetric (APIC) I/O
maxusers 0
hints "ATMOS.hints" #Default places to look for devices.
#options COMPAT_FREEBSD4
options SCHED_4BSD #4BSD scheduler
#options SCHED_ULE
#options ADAPTIVE_MUTEXES
#options PQ_CACHESIZE=256
options CPU_ENABLE_SSE
options CLK_USE_TSC_CALIBRATION
#options HZ=1000
#makeoptions CONF_CFLAGS=-fno-builtin
#options MAXDSIZ=(1024UL*1024*1024)
#options MAXSSIZ=(128UL*1024*1024)
#options DFLDSIZ=(1024UL*1024*1024)
options GEOM_AES
options GEOM_APPLE
options GEOM_BDE
options GEOM_BSD
options GEOM_GPT
options GEOM_MBR
options GEOM_PC98
options GEOM_SUNLABEL
options GEOM_VOL
options ROOTDEVNAME=\"ufs:amrd0s1a\"
options INET #InterNETworking
#options INET6 #IPv6 communications protocols
options FFS #Berkeley Fast Filesystem
options SOFTUPDATES #Enable FFS soft updates support
options UFS_ACL #Support for access control lists
options UFS_DIRHASH #Improve performance on big directories
options NFSCLIENT #Network Filesystem Client
options NFSSERVER #Network Filesystem Server
options MSDOSFS #MSDOS Filesystem
options CD9660 #ISO 9660 Filesystem
options PROCFS #Process filesystem (requires PSEUDOFS)
options PSEUDOFS #Pseudo-filesystem framework
options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!]
options SCSI_DELAY=5000 #Delay (in ms) before probing SCSI
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options NETSMB
options NETSMBCRYPTO
options LIBMCHAIN
options LIBICONV
#options WATCHDOG
options NETGRAPH
#options NETGRAPH_ASYNC
#options NETGRAPH_BPF
#options NETGRAPH_BRIDGE
#options NETGRAPH_CISCO
#options NETGRAPH_ECHO
#options NETGRAPH_ETHER
#options NETGRAPH_FRAME_RELAY
#options NETGRAPH_GIF
#options NETGRAPH_GIF_DEMUX
#options NETGRAPH_HOLE
#options NETGRAPH_IFACE
#options NETGRAPH_IP_INPUT
#options NETGRAPH_KSOCKET
#options NETGRAPH_L2TP
#options NETGRAPH_LMI
#options NETGRAPH_MPPC_ENCRYPTION
#options NETGRAPH_ONE2MANY
#options NETGRAPH_PPP
#options NETGRAPH_PPPOE
#options NETGRAPH_PPTPGRE
#options NETGRAPH_RFC1490
#options NETGRAPH_SOCKET
#options NETGRAPH_SPLIT
#options NETGRAPH_TEE
#options NETGRAPH_TTY
#options NETGRAPH_UI
#options NETGRAPH_VJC
options MROUTING
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_FORWARD
#options IPFIREWALL_VERBOSE_LIMIT=100
#options IPFIREWALL_DEFAULT_TO_ACCEPT
#options IPV6FIREWALL
#options IPV6FIREWALL_VERBOSE
#options IPV6FIREWALL_VERBOSE_LIMIT=100
#options IPV6FIREWALL_DEFAULT_TO_ACCEPT
options IPDIVERT
#options IPFILTER
#options IPFILTER_LOG
#options IPFILTER_DEFAULT_BLOCK
options IPSTEALTH
options RANDOM_IP_ID
options ACCEPT_FILTER_DATA
#options ACCEPT_FILTER_HTTP
options TCP_DROP_SYNFIN
options DUMMYNET
#options BRIDGE
options QUOTA
options _KPOSIX_PRIORITY_SCHEDULING
options P1003_1B_SEMAPHORES
#options MAC
#options MAC_BIBA
#options MAC_BSDEXTENDED
#options MAC_DEBUG
#options MAC_IFOFF
#options MAC_LOMAC
#options MAC_MLS
#options MAC_NONE
#options MAC_PARTITION
#options MAC_SEEOTHERUIDS
#options MAC_TEST
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
device isa
#options AUTO_EOI_1
device pci
device agp
# Floppy drives
device fdc
# SCSI Controllers
device sym # NCR/Symbios Logic (newer chipsets + those of `ncr')
#device ahc
# SCSI peripherals
device scbus # SCSI bus (required)
device ch # SCSI media changers
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
device ses # SCSI Environmental Services (and SAF-TE)
# RAID controllers
device amr # AMI MegaRAID
#options CHANGER_MIN_BUSY_SECONDS=2
#options CHANGER_MAX_BUSY_SECONDS=10
#options SA_IO_TIMEOUT=4
#options SA_SPACE_TIMEOUT=60
#options SA_REWIND_TIMEOUT=(2*60)
#options SA_ERASE_TIMEOUT=(4*60)
#options SA_1FM_AT_EOD
#options SCSI_PT_DEFAULT_TIMEOUT=60
options SES_ENABLE_PASSTHROUGH
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
options ATKBD_DFLT_KEYMAP
makeoptions ATKBD_DFLT_KEYMAP=us.iso
device psm # PS/2 mouse
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
options MAXCONS=8
#options SC_ALT_MOUSE_IMAGE
options SC_DFLT_FONT
makeoptions SC_DFLT_FONT=cp850
options SC_DISABLE_DDBKEY
options SC_DISABLE_REBOOT
options SC_HISTORY_SIZE=512
#options SC_MOUSE_CHAR=0x3
options SC_PIXEL_MODE
options SC_NORM_ATTR=(FG_GREEN|BG_BLACK)
options SC_NORM_REV_ATTR=(FG_YELLOW|BG_GREEN)
options SC_KERNEL_CONS_ATTR=(FG_RED|BG_BLACK)
options SC_KERNEL_CONS_REV_ATTR=(FG_BLACK|BG_RED)
#options SC_CUT_SPACES2TABS
#options SC_CUT_SEPCHARS=\"x09\"
#options SC_TWOBUTTON_MOUSE
#options SC_NO_CUTPASTE
#options SC_NO_FONT_LOADING
#options SC_NO_HISTORY
#options SC_NO_SYSMOUSE
#options SC_NO_SUSPEND_VTYSWITCH
device npx
#device pmtimer
#device sio # 8250, 16[45]50 based serial ports
# Parallel port
#device ppc
#device ppbus # Parallel port bus (required)
#device lpt # Printer
#device plip # TCP/IP over parallel
#device ppi # Parallel port interface device
#device vpo # Requires scbus and da
device miibus # MII bus support
device em
#device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device random # Entropy device
device loop # Network loopback
device ether # Ethernet support
#device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
#device gif # IPv6 and IPv4 tunneling
#device faith # IPv6-to-IPv4 relaying (translation)
device bpf # Berkeley packet filter
------------------
Thanks a lot for your help,
Oliver
--
MfG
O. Hartmann
ohartman@mail.physik.uni-mainz.de
------------------------------------------------------------------
Systemadministration des Institutes fuer Physik der Atmosphaere (IPA)
------------------------------------------------------------------
Johannes Gutenberg Universitaet Mainz
Becherweg 21
55099 Mainz
Tel: +496131/3924662 (Maschinenraum)
Tel: +496131/3924144 (Buero)
FAX: +496131/3923532
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030813103509.Q49991>