Date: Wed, 21 Feb 2001 14:38:44 -0800 (PST) From: Yevgeniy Aleynikov <eugenea@go2net.com> To: FreeBSD-gnats-submit@freebsd.org Subject: kern/25264: Kernel trap 12 in camisr Message-ID: <Pine.LNX.3.96.1010221143653.17437A-100000@hunches.go2net.com>
next in thread | raw e-mail | index | archive | help
>Number: 25264
>Category: kern
>Synopsis: Kernel trap 12 in camisr
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Feb 21 14:40:00 PST 2001
>Closed-Date:
>Last-Modified:
>Originator: Eugene Aleynikov
>Release: FreeBSD 4.2-RELEASE i386
>Organization:
Go2net
>Environment:
Standard BSDi X-treme server - Intel L440GX+ m/b
dmesg.boot:
Copyright (c) 1992-2000 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.2-RELEASE #5: Tue Feb 20 12:37:46 PST 2001
user@domain.com:/usr/src/sys/compile/HMV
Timecounter "i8254" frequency 1193182 Hz
CPU: Pentium III/Pentium III Xeon/Celeron (796.54-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x683 Stepping = 3
Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CM
OV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory = 536805376 (524224K bytes)
avail memory = 519602176 (507424K bytes)
Programming 24 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
FreeBSD/SMP: Multiprocessor motherboard
cpu0 (BSP): apic id: 1, version: 0x00040011, at 0xfee00000
cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000
io0 (APIC): apic id: 2, version: 0x00170011, at 0xfec00000
Preloaded elf kernel "kernel" at 0xc02ab000.
Pentium Pro MTRR support enabled
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Intel 82443GX host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pcib2: <Intel 82443GX (440 GX) PCI-PCI (AGP) bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib2
pcib3: <PCI to PCI bridge (vendor=1011 device=0023)> at device 15.0 on
pci1
pci2: <PCI bus> on pcib3
fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0x3000-0x303f mem
0xf4200000-0xf42f
ffff,0xf4300000-0xf4300fff irq 20 at device 4.0 on pci2
fxp0: Ethernet address 00:02:b3:08:76:45
ahc0: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0x2000-0x20ff mem
0xf4100000
-0xf4100fff irq 19 at device 12.0 on pci0
aic7896/97: Wide Channel A, SCSI Id=7, 32/255 SCBs
ahc1: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0x2400-0x24ff mem
0xf4101000
-0xf4101fff irq 19 at device 12.1 on pci0
aic7896/97: Wide Channel B, SCSI Id=7, 32/255 SCBs
fxp1: <Intel Pro 10/100B/100+ Ethernet> port 0x2800-0x283f mem
0xf4000000-0xf40f
ffff,0xf4102000-0xf4102fff irq 21 at device 14.0 on pci0
fxp1: Ethernet address 00:d0:b7:a9:c4:c6
isab0: <Intel 82371AB PCI to ISA bridge> at device 18.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 ATA33 controller> port 0x2860-0x286f at device 18.1
on pci
0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <Intel 82371AB/EB (PIIX4) USB controller> at 18.2 irq 21
Timecounter "PIIX" frequency 3579545 Hz
chip1: <Intel 82371AB Power management controller> port 0x1040-0x104f at
device
18.3 on pci0
pci0: <Cirrus Logic GD5480 SVGA controller> at 20.0
pcib1: <Intel 82443GX host to AGP bridge> on motherboard
pci3: <PCI bus> on pcib1
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> on isa0
sc0: VGA <16 virtual consoles, flags=0x0>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
IP packet filtering initialized, divert disabled, rule-based forwarding
enabled,
default to accept, logging limited to 100 packets/entry by default
DUMMYNET initialized (000608)
Waiting 15 seconds for SCSI devices to settle
SMP: AP CPU #1 Launched!
sa0 at ahc1 bus 0 target 3 lun 0
sa0: <HP C1537A L005> Removable Sequential Access SCSI-2 device
sa0: 10.000MB/s transfers (10.000MHz, offset 32)
da1 at ahc0 bus 0 target 2 lun 0
da1: <WSI FLASHDISK 0223> Fixed Direct Access SCSI-2 device
da1: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing
Enabled
da1: 35003MB (71686272 512 byte sectors: 255H 63S/T 4462C)
da0 at ahc0 bus 0 target 0 lun 0
da0: <SEAGATE ST39204LC 0006> Fixed Direct Access SCSI-3 device
da0: 80.000MB/s transfers (40.000MHz, offset 63, 16bit), Tagged Queueing
Enabled
da0: 8750MB (17921835 512 byte sectors: 255H 63S/T 1115C)
Mounting root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
------------
Kernel config:
makeoptions DEBUG=-g #Build kernel with gdb(1) debug
symbols
machine i386
#cpu I586_CPU
cpu I686_CPU
ident "HMV"
# This may be set way to high. It might we worth trying it lower if we
# are allocating too much of our resources for the kernel.
maxusers 256
options INET #InterNETworking
options FFS #Berkeley Fast Filesystem
options FFS_ROOT #FFS usable as root device [keep
this!]
# following available as modules
#options MFS #Memory Filesystem
#options UNION #Union filesystem
#options MFS_ROOT #MFS usable as root device, "MFS"
req'ed
#options NFS #Network Filesystem
#options NFS_ROOT #NFS usable as root device, "NFS"
req'ed
#options MSDOSFS #MSDOS Filesystem
#options "CD9660" #ISO 9660 Filesystem
options PROCFS #Process filesystem
options "COMPAT_43" #Compatible with BSD 4.3 [KEEP
THIS!]
options SCSI_DELAY=15000 #Be pessimistic about Joe SCSI
device
options UCONSOLE #Allow users to grab the console
options USERCONFIG #boot -c editor
options VISUAL_USERCONFIG #visual boot -c editor
options KTRACE #ktrace(1) syscall trace support
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options P1003_1B #Posix P1003_1B real-time
extensions
options _KPOSIX_PRIORITY_SCHEDULING
#options ICMP_BANDLIM #Rate limit bad replies
options PERFMON
# To make an SMP kernel, the next two are needed
options SMP # Symmetric MultiProcessor Kernel
options APIC_IO # Symmetric (APIC) I/O
device isa
device pci
# Floppy drives
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
device fd1 at fdc0 drive 1
# ATA and ATAPI devices
device ata0 at isa? port IO_WD1 irq 14
device ata1 at isa? port IO_WD2 irq 15
device ata
#device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID #Static device numbering
#options ATA_ENABLE_ATAPI_DMA #Enable DMA on ATAPI devices
# SCSI Controllers
# A single entry for any of these devices (ncr, ahb, ahc) is
# sufficient for any number of installed devices.
device ahc # AHA2940 and onboard AIC7xxx devices
device bt0 at isa?
# SCSI peripherals
# Only one of each of these is needed, they are dynamically allocated.
device scbus # SCSI bus (required)
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI)
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1
# flags 0x1
device psm0 at atkbdc? irq 12
device vga0 at isa?
# splash screen/screen saver
#pseudo-device splash
# syscons is the default console driver, resembling an SCO console
#device sc0 at isa? flags 0x100
device sc0 at isa?
options SC_DISABLE_DDBKEY # disable `debug' key
options SC_DISABLE_REBOOT # disable reboot key sequence
#options SC_MOUSE_CHAR=0x3 # char code for text mode mouse
cursor
options SC_NO_CUTPASTE
options SC_NO_FONT_LOADING
options SC_NO_SYSMOUSE
# Floating point support - do not disable.
device npx0 at nexus? port IO_NPX irq 13
# Serial (COM) ports
device sio0 at isa? port IO_COM1 flags 0x10 irq 4
device sio1 at isa? port IO_COM2 irq 3
#device sio2 at isa? disable port IO_COM3 tty irq 5
#device sio3 at isa? disable port IO_COM4 tty irq 9
# Parallel port
device ppc0 at isa? irq 7
device ppbus # Parallel port bus (required)
#device lpt # Printer
#device plip # TCP/IP over parallel
#device ppi # Parallel port interface device
#device vpo # Requires scbus and da
# PCI Ethernet NICs.
device fxp # Intel EtherExpress PRO/100B (82557,
82558)
# Pseudo devices - the number indicates how many units to allocated.
pseudo-device loop # Network loopback
pseudo-device ether # Ethernet support
# Jack this up just in case we have lots of automated stuff doing work.
pseudo-device pty # Pseudo-ttys (telnet etc)
pseudo-device gzip # Exec gzipped a.out's
# The `bpf' pseudo-device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# The number of devices determines the maximum number of
# simultaneous BPF clients programs runnable.
pseudo-device bpf #Berkeley packet filter
# We need to service many network requests. This should probably be tuned
# better to more closely fit our needs.
options NMBCLUSTERS=12288
# Set the number of PV entries per process. Increasing this can
# stop panics related to heavy use of shared memory. However, that can
# (combined with large amounts of physical memory) cause panics at
# boot time due the kernel running out of VM space.
#
# If you're tweaking this, you might also want to increase the sysctls
# "vm.v_free_min", "vm.v_free_reserved", and "vm.v_free_target".
#
# The value below is the one more than the default.
#
#options PMAP_SHPGPERPROC=201
options PMAP_SHPGPERPROC=401
options QUOTA
options IPFIREWALL #firewall
options IPFIREWALL_VERBOSE #print information about
options IPFIREWALL_FORWARD #enable transparent proxy support
options IPFIREWALL_VERBOSE_LIMIT=100 #limit verbosity
options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by
default
options DUMMYNET
#kldstat
Id Refs Address Size Name
1 2 0xc0100000 1a9ac0 kernel
2 1 0xc35b5000 4000 null.ko
(there's one nullfs mount)
>Description:
Several times a day kernel crashes with following diag.
Cant find coredump because SCSI system is not functioning during crash.
login: panic: lockmgr: pid 64908, not exclusive lock holder 380 unlocking
mp_lock = 00000001; cpuid = 0; lapic.id = 01000000
boot() called on cpu#0
syncing disks... 203 204 204 204 204 204 204 204 204 204 204 204 204 204
204 204
204 204 204 204
giving up on 202 buffers
Uptime: 15h47m27s
(da1:ahc0:0:2:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da1:ahc0:0:2:0): ILLEGAL REQUEST asc:20,0
(da1:ahc0:0:2:0): Invalid command operation code
dumping to dev #da/0x20001, offset 1048704
dump
Fatal trap 12: page fault while in kernel mode
mp_lock = 00000002; cpuid = 0; lapic.id = 01000000
fault virtual address = 0x0
fault code = supervisor write, page not present
instruction pointer = 0x8:0xc0121968
stack pointer = 0x10:0xdbc0095c
frame pointer = 0x10:0xdbc0096c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 64908 (httpd)
interrupt mask = net tty bio cam <- SMP: XXX
trap number = 12
panic: page fault
mp_lock = 00000002; cpuid = 0; lapic.id = 01000000
boot() called on cpu#0
Uptime: 15h47m28s
Fatal trap 12: page fault while in kernel mode
mp_lock = 00000003; cpuid = 0; lapic.id = 01000000
fault virtual address = 0x10
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc0121a84
stack pointer = 0x10:0xdbc00578
frame pointer = 0x10:0xdbc0058c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 64908 (httpd)
interrupt mask = net tty bio cam <- SMP: XXX
trap number = 12
panic: page fault
mp_lock = 00000003; cpuid = 0; lapic.id = 01000000
boot() called on cpu#0
Uptime: 15h47m28s
Fatal trap 12: page fault while in kernel mode
mp_lock = 00000004; cpuid = 0; lapic.id = 01000000
fault virtual address = 0x10
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc0121a84
stack pointer = 0x10:0xdbc00194
frame pointer = 0x10:0xdbc001a8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 64908 (httpd)
interrupt mask = net tty bio cam <- SMP: XXX
trap number = 12
panic: page fault
mp_lock = 00000004; cpuid = 0; lapic.id = 01000000
boot() called on cpu#0
Uptime: 15h47m28s
--------skipped several traps with the same instruction pointer-----
(kgdb) x/i 0xc0121968
0xc0121968 <camisr+200>: mov %eax,(%edx)
(kgdb) list *0xc0121968
0xc0121968 is in camisr (../../cam/cam_queue.h:224).
219
220 static __inline void
221 cam_ccbq_ccb_done(struct cam_ccbq *ccbq, union ccb *done_ccb)
222 {
223 TAILQ_REMOVE(&ccbq->active_ccbs, &done_ccb->ccb_h,
224 xpt_links.tqe);
225 ccbq->dev_active--;
226 ccbq->dev_openings++;
227 ccbq->held++;
228 }
(kgdb) x/i 0xc0121a84
0xc0121a84 <camisr+484>: pushl (%eax)
(kgdb) list *0xc0121a84
0xc0121a84 is in camisr (../../cam/cam_xpt.c:6332).
6327 } else if (runq) {
6328 xpt_run_dev_sendq(ccb_h->path->bus);
6329 }
6330
6331 /* Call the peripheral driver's callback */
6332 (*ccb_h->cbfcnp)(ccb_h->path->periph, (union ccb
*)ccb_h);
6333
6334 /* Raise IPL for while test */
6335 s = splcam();
6336 }
----------
There's another server also that crashes similar way.
SCSI is terminated propertly. SCSI tape (SE device) is on the second
SCSI bus alone.
>How-To-Repeat:
Just keep it running under heavy user load (web+cgi).
>Fix:
None
>Release-Note:
>Audit-Trail:
>Unformatted:
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.LNX.3.96.1010221143653.17437A-100000>