Date: Thu, 20 Jan 2011 17:50:35 GMT From: Bart van Leeuwen <bart@bartsplace.net> To: freebsd-gnats-submit@FreeBSD.org Subject: i386/154170: Panic in sched_switch (/usr/src/sys/kern/sched_ule.c:1814) Message-ID: <201101201750.p0KHoZUu097042@red.freebsd.org> Resent-Message-ID: <201101201800.p0KI0OX0022423@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 154170
>Category: i386
>Synopsis: Panic in sched_switch (/usr/src/sys/kern/sched_ule.c:1814)
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-i386
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu Jan 20 18:00:24 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator: Bart van Leeuwen
>Release: 8.1p2
>Organization:
Private
>Environment:
FreeBSD airport-new.bartsplace.net 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Wed Jan 19 00:41:12 CET 2011 root@airport-new.bartsplace.net:/usr/obj/usr/src/sys/AIRPORT i386
>Description:
System crashes, seemingly at random, tho it seems usually during or just after relatively heavy disk load.
In all cases the crash looks identical (happens at the same bit of code, with always the same process (11, [idle])
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x24
fault code = supervisor write, page not present
instruction pointer = 0x20:0xc08c07f7
stack pointer = 0x28:0xc51ecb98
frame pointer = 0x28:0xc51ecbc8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 11 (idle: cpu1)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 14h41m50s
Physical memory: 2035 MB
Dumping 253 MB: 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14
Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
#0 doadump () at pcpu.h:246
246 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump () at pcpu.h:246
#1 0xc089f5e7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
#2 0xc089f849 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:590
#3 0xc0be6f2c in trap_fatal (frame=0xc51ecb58, eva=36)
at /usr/src/sys/i386/i386/trap.c:938
#4 0xc0be7840 in trap (frame=0xc51ecb58) at /usr/src/sys/i386/i386/trap.c:328
#5 0xc0bc9c0b in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#6 0xc08c07f7 in sched_switch (td=0xc554e500, newtd=0x0, flags=1548)
at /usr/src/sys/kern/sched_ule.c:1814
#7 0xc08a805a in mi_switch (flags=1548, newtd=0x0)
at /usr/src/sys/kern/kern_synch.c:449
#8 0xc08c024c in sched_preempt (td=0xc554e500)
at /usr/src/sys/kern/sched_ule.c:2063
#9 0xc0bda5e4 in ipi_bitmap_handler (frame=
{tf_fs = -1061421048, tf_es = -987889624, tf_ds = -1061355480, tf_edi = -982889960, tf_esi = -982889984, tf_ebp = -987837
328, tf_isp = -987837348, tf_ebx = 500, tf_edx = 2000, tf_ecx = 500, tf_eax = 500, tf_trapno = 0, tf_err = 0, tf_eip = -1061420
683, tf_cs = 32, tf_eflags = 514, tf_esp = -987837284, tf_ss = -1068612521}) at /usr/src/sys/i386/i386/mp_machdep.c:1267
#10 0xc0bca32e in Xipi_intr_bitmap_handler () at apic_vector.s:298
#11 0xc0bc0175 in acpi_cpu_c1 ()
at /usr/src/sys/i386/acpica/acpi_machdep.c:556
#12 0xc04e4457 in acpi_cpu_idle () at /usr/src/sys/dev/acpica/acpi_cpu.c:938
#13 0xc0bd372b in cpu_idle_acpi (busy=1)
at /usr/src/sys/i386/i386/machdep.c:1225
#14 0xc0bd4fab in cpu_idle (busy=1) at /usr/src/sys/i386/i386/machdep.c:1313
#15 0xc08c0ea3 in sched_idletd (dummy=0x0)
at /usr/src/sys/kern/sched_ule.c:2550
#16 0xc0874a21 in fork_exit (callout=0xc08c0c60 <sched_idletd>, arg=0x0,
frame=0xc51ecd38) at /usr/src/sys/kern/kern_fork.c:844
#17 0xc0bc9c80 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:270
------------------------------------------------------------------------
ps -axl
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
0 0 0 0 -68 0 0 0 - DLs ?? -6173364:-59.55 [kernel]
0 1 0 0 57 0 2912 0 wait DLs ?? 23667260:40.00 [init]
0 2 0 0 -8 0 0 0 - RL ?? 12686974:36.00 [g_event]
0 3 0 0 -8 0 0 0 - DL ?? -15178934:-19.55 [g_up]
0 4 0 0 -8 0 0 0 - DL ?? -7287166:-35.55 [g_down]
0 5 0 0 -16 0 0 0 idle DL ?? 0:00.00 [ciss_notif
0 6 0 0 -16 0 0 0 - DL ?? 12943975:12.00 [fdc0]
0 7 0 0 -16 0 0 0 waitin DL ?? 2678:44.00 [sctp_itera
0 8 0 0 -60 0 0 0 ccb_sc DL ?? 0:00.00 [xpt_thrd]
0 9 0 0 -32 0 0 0 psleep DL ?? 7092141:12.00 [pagedaemon
0 10 0 0 -16 0 0 0 audit_ DL ?? 0:00.00 [audit]
0 11 0 0 171 0 0 0 - RL ?? 32968188:20.00 [idle]
0 12 0 0 -48 0 0 0 - WL ?? -11341039:-55.55 [intr]
0 13 0 0 -16 0 0 0 - DL ?? 30424824:28.00 [yarrow]
0 14 0 0 -64 0 0 0 - DL ?? 13686869:12.00 [usb]
0 15 0 0 -16 0 0 0 tzpoll DL ?? 27702534:56.00 [acpi_therm
0 16 0 0 -16 0 0 0 coolin DL ?? 2168942:40.00 [acpi_cooli
0 17 0 0 -32 0 0 0 psleep DL ?? 571:52.00 [vmdaemon]
0 18 0 0 76 0 0 0 pgzero DL ?? 50512:56.00 [pagezero]
0 19 0 0 44 0 0 0 psleep DL ?? -31897972:-55.55 [bufdaemon]
0 20 0 0 44 0 0 0 vlruwt DL ?? 21485856:24.00 [vnlru]
0 21 0 0 44 0 0 0 syncer DL ?? 8000403:40.00 [syncer]
0 22 0 0 44 0 0 0 sdflus DL ?? 2343381:00.00 [softdepflu
0 23 0 0 -32 0 0 0 flowcl DL ?? 8480586:32.00 [flowcleane
0 597 1 0 44 0 1888 0 select Ds ?? 4614079:48.00 [devd]
0 781 1 0 44 0 3348 0 select Ds ?? 8765502:36.00 [syslogd]
53 868 1 0 44 0 27036 0 kqread Ds ?? 34742034:00.00 [named]
0 1148 1 0 44 0 6704 0 select Ds ?? 406759:40.00 [sshd]
0 1156 1 0 44 0 6080 0 select Ds ?? -25313389:-59.55 [sendmail]
25 1160 1 0 44 0 6080 0 pause Ds ?? 858881:08.00 [sendmail]
0 1167 1 0 44 0 3376 0 nanslp Ds ?? 6660384:16.00 [cron]
0 1262 1 0 44 0 3348 0 select Ds ?? 5298226:40.00 [syslogd]
0 1370 1 0 44 0 14144 0 select Ds ?? 22528955:48.00 [perl]
88 1382 1 0 76 0 3628 0 wait Ds ?? 801893:40.00 [sh]
88 1464 1382 0 76 0 196516 0 sigwai D ?? -32496359:-31.55 [mysqld]
0 1490 1 0 44 0 3376 0 nanslp Ds ?? 6814799:56.00 [cron]
0 1608 1 0 44 0 3348 0 select Ds ?? 5192087:24.00 [syslogd]
0 1737 1 0 44 0 10144 0 select Ds ?? 3687423:08.00 [httpd]
0 1762 1 0 44 0 3376 0 nanslp Ds ?? 6611521:36.00 [cron]
0 1876 1 0 44 0 3348 0 select Ds ?? 5388062:16.00 [syslogd]
80 1877 1737 0 76 0 10144 0 accept D ?? 139905:28.00 [httpd]
80 1878 1737 0 76 0 10144 0 accept D ?? 117199:52.00 [httpd]
80 1879 1737 0 76 0 10144 0 accept D ?? 121229:12.00 [httpd]
80 1880 1737 0 76 0 10144 0 accept D ?? 103000:28.00 [httpd]
80 1881 1737 0 76 0 10144 0 accept D ?? 84894:08.00 [httpd]
0 2015 1 0 44 0 3376 0 nanslp Ds ?? 7286703:16.00 [cron]
0 2384 1 0 44 0 3348 0 select Ds ?? 5222692:56.00 [syslogd]
0 2520 1 0 44 0 3376 0 nanslp Ds ?? 6634484:00.00 [cron]
0 2615 1 0 76 0 3348 0 ttyin Ds+ ?? 301545:00.00 [getty]
0 2616 1 0 76 0 3348 0 ttyin Ds+ ?? 265864:52.00 [getty]
0 2617 1 0 76 0 3348 0 ttyin Ds+ ?? 295432:52.00 [getty]
0 2618 1 0 76 0 3348 0 ttyin Ds+ ?? 296500:32.00 [getty]
0 2619 1 0 76 0 3348 0 ttyin Ds+ ?? 291527:44.00 [getty]
0 2620 1 0 76 0 3348 0 ttyin Ds+ ?? 299679:52.00 [getty]
0 2621 1 0 76 0 3348 0 ttyin Ds+ ?? 250806:20.00 [getty]
0 2622 1 0 76 0 3348 0 ttyin Ds+ ?? 243478:24.00 [getty]
0 14090 1 0 44 0 3348 0 select Ds ?? 1184109:28.00 [syslogd]
0 14234 1 0 44 0 3376 0 nanslp Ds ?? 1905958:24.00 [cron]
0 16417 1148 0 44 0 9428 0 select Ds ?? 17866613:08.00 [sshd]
0 16419 16417 0 47 0 4628 0 pause Ds ?? 790395:56.00 [csh]
0 16423 16419 0 44 0 4564 0 wait D ?? 802982:24.00 [bash]
0 16503 16423 0 76 0 4564 0 ttyin D+ ?? 4508128:48.00 [bash]
0 17048 1 0 44 0 20588 0 select Ds ?? 14844112:40.00 [httpd]
0 17071 1148 0 44 0 9428 0 select Ds ?? 11617857:44.00 [sshd]
0 17073 17071 0 48 0 4628 0 pause Ds ?? 779509:48.00 [csh]
0 17077 17073 0 44 0 4564 0 ttyin D+ ?? 842272:56.00 [bash]
0 35113 1370 0 -84 0 0 0 - ZW ?? 0:00.00 <defunct>
0 54672 1148 0 49 0 9428 0 sbwait Ds ?? 3537124:56.00 [sshd]
1001 55132 54672 0 45 0 9428 0 select D ?? 331324:20.00 [sshd]
1001 55162 55132 0 46 0 4564 0 ttyin Ds+ ?? 501825:00.00 [bash]
80 85936 17048 0 46 0 22636 0 accept D ?? 0:00.00 [httpd]
80 85937 17048 0 52 0 20588 0 accept D ?? 0:00.00 [httpd]
80 85938 17048 0 53 0 20588 0 accept D ?? 0:00.00 [httpd]
80 85939 17048 0 52 0 20588 0 accept D ?? 0:00.00 [httpd]
80 85940 17048 0 52 0 20588 0 accept D ?? 0:00.00 [httpd]
80 86526 17048 0 50 0 20588 0 accept D ?? 0:00.00 [httpd]
------------------------------------------------------------------------
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE-p2 #0: Wed Jan 19 00:41:12 CET 2011
root@airport-new.bartsplace.net:/usr/obj/usr/src/sys/AIRPORT i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3180.87-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf25 Family = f Model = 2 Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE
2,SS,HTT,TM,PBE>
Features2=0x4400<CNXT-ID,xTPR>
real memory = 2147483648 (2048 MB)
avail memory = 2091089920 (1994 MB)
ACPI APIC Table: <COMPAQ 00000083>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s) x 2 HTT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP/HT): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP/HT): APIC ID: 7
ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20100331/tbfadt-707)
ACPI Warning: Invalid length for Pm1bControlBlock: 32, using default 16 (20100331/tbfadt-707)
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic2 <Version 1.1> irqs 32-47 on motherboard
ioapic3 <Version 1.1> irqs 48-63 on motherboard
kbd1 at kbdmux0
acpi0: <COMPAQ P29> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x2400-0x24ff mem 0xf6000000-0xf6ffffff,0xf5ff0000-0xf5ff0fff at device 3.0 on pci0
pci0: <base peripheral> at device 4.0 (no driver attached)
pci0: <base peripheral> at device 4.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks CSB5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2000-0x200f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xf5ef0000-0xf5ef0fff irq 7 at device 15.2 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
pcib1: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib1
ciss0: <Compaq Smart Array 5i> port 0x3000-0x30ff mem 0xf7dc0000-0xf7dfffff,0xf7cf0000-0xf7cf3fff irq 30 at device 3.0 on pci1
ciss0: PERFORMANT Transport
ciss0: [ITHREAD]
pcib2: <ACPI Host-PCI bridge> on acpi0
pci2: <ACPI PCI bus> on pcib2
bge0: <Compaq NC7781 Gigabit Server Adapter, ASIC rev. 0x001002> mem 0xf7ef0000-0xf7efffff irq 29 at device 1.0 on pci2
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:0e:7f:20:c2:ed
bge0: [ITHREAD]
bge1: <Compaq NC7781 Gigabit Server Adapter, ASIC rev. 0x001002> mem 0xf7ee0000-0xf7eeffff irq 31 at device 2.0 on pci2
miibus1: <MII bus> on bge1
brgphy1: <BCM5703 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:0e:7f:20:c2:ec
bge1: [ITHREAD]
pcib3: <ACPI Host-PCI bridge> on acpi0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pci6: <ACPI PCI bus> on pcib4
pci6: <base peripheral, PCI hot-plug controller> at device 30.0 (no driver attached)
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xcd7ff,0xee000-0xeffff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atrtc0: <AT Real Time Clock> at port 0x70 irq 8 on isa0
ppc0: parallel port not found.
p4tcc0: <CPU Frequency Thermal Control> on cpu0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
p4tcc2: <CPU Frequency Thermal Control> on cpu2
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat loadable, rule-based forwarding enabled, default to deny, logging disabled
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
usbus0: 12Mbps Full Speed USB v1.0
ugen0.1: <(0x1166)> at usbus0
uhub0: <(0x1166) OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub0: 4 ports with 4 removable, self powered
acd0: CDROM <COMPAQ CD-ROM SN-124/N104> at ata0-master PIO4
da0 at ciss0 bus 0 scbus0 target 0 lun 0
da0: <COMPAQ RAID 1 VOLUME OK> Fixed Direct Access SCSI-0 device
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 34727MB (71122560 512 byte sectors: 255H 32S/T 8716C)
da1 at ciss0 bus 0 scbus0 target 1 lun 0
da1: <COMPAQ RAID 1 VOLUME OK> Fixed Direct Access SCSI-0 device
da1: 135.168MB/s transfers
da1: Command Queueing enabled
da1: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/da0s1a
..... (lots of 'spam' from ipfw denying packets)
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x24
fault code = supervisor write, page not present
instruction pointer = 0x20:0xc08c07f7
stack pointer = 0x28:0xc51ecb98
frame pointer = 0x28:0xc51ecbc8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 11 (idle: cpu1)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 14h41m50s
Physical memory: 2035 MB
Dumping 253 MB: 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14
--------------------------------------------------------------------------------
kernel config is a copy of GENERIC with the following options added:
options DUMMYNET
options IPFIREWALL_FORWARD
options IPDIVERT
options IPFIREWALL_VERBOSE_LIMIT=1000
options IPFIREWALL_VERBOSE
options IPFIREWALL
This is to enable IPFIREWALL_FORWARD, which is an absolutely mandatory feature for this configuration.
If any other information is needed, I have multiple crashdumps available.
>How-To-Repeat:
No known scenario, other then leaving the machine powered on, and compiling or running software.
>Fix:
Not a fix but a workaround, using 4BSD scheduler does not result in crashes so far.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201101201750.p0KHoZUu097042>