From owner-freebsd-i386@FreeBSD.ORG Thu Jan 20 18:00:25 2011 Return-Path: Delivered-To: freebsd-i386@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20A1E1065675 for ; Thu, 20 Jan 2011 18:00:25 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D59388FC1C for ; Thu, 20 Jan 2011 18:00:24 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p0KI0O8r022445 for ; Thu, 20 Jan 2011 18:00:24 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p0KI0OX0022423; Thu, 20 Jan 2011 18:00:24 GMT (envelope-from gnats) Resent-Date: Thu, 20 Jan 2011 18:00:24 GMT Resent-Message-Id: <201101201800.p0KI0OX0022423@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-i386@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Bart van Leeuwen Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A1AC1065670 for ; Thu, 20 Jan 2011 17:50:36 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (unknown [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 778308FC08 for ; Thu, 20 Jan 2011 17:50:36 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p0KHoZ16097045 for ; Thu, 20 Jan 2011 17:50:35 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id p0KHoZUu097042; Thu, 20 Jan 2011 17:50:35 GMT (envelope-from nobody) Message-Id: <201101201750.p0KHoZUu097042@red.freebsd.org> Date: Thu, 20 Jan 2011 17:50:35 GMT From: Bart van Leeuwen To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: i386/154170: Panic in sched_switch (/usr/src/sys/kern/sched_ule.c:1814) X-BeenThere: freebsd-i386@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: I386-specific issues for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jan 2011 18:00:25 -0000 >Number: 154170 >Category: i386 >Synopsis: Panic in sched_switch (/usr/src/sys/kern/sched_ule.c:1814) >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-i386 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Jan 20 18:00:24 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Bart van Leeuwen >Release: 8.1p2 >Organization: Private >Environment: FreeBSD airport-new.bartsplace.net 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Wed Jan 19 00:41:12 CET 2011 root@airport-new.bartsplace.net:/usr/obj/usr/src/sys/AIRPORT i386 >Description: System crashes, seemingly at random, tho it seems usually during or just after relatively heavy disk load. In all cases the crash looks identical (happens at the same bit of code, with always the same process (11, [idle]) GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x24 fault code = supervisor write, page not present instruction pointer = 0x20:0xc08c07f7 stack pointer = 0x28:0xc51ecb98 frame pointer = 0x28:0xc51ecbc8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 11 (idle: cpu1) trap number = 12 panic: page fault cpuid = 1 Uptime: 14h41m50s Physical memory: 2035 MB Dumping 253 MB: 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/nullfs.ko #0 doadump () at pcpu.h:246 246 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:246 #1 0xc089f5e7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0xc089f849 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:590 #3 0xc0be6f2c in trap_fatal (frame=0xc51ecb58, eva=36) at /usr/src/sys/i386/i386/trap.c:938 #4 0xc0be7840 in trap (frame=0xc51ecb58) at /usr/src/sys/i386/i386/trap.c:328 #5 0xc0bc9c0b in calltrap () at /usr/src/sys/i386/i386/exception.s:165 #6 0xc08c07f7 in sched_switch (td=0xc554e500, newtd=0x0, flags=1548) at /usr/src/sys/kern/sched_ule.c:1814 #7 0xc08a805a in mi_switch (flags=1548, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:449 #8 0xc08c024c in sched_preempt (td=0xc554e500) at /usr/src/sys/kern/sched_ule.c:2063 #9 0xc0bda5e4 in ipi_bitmap_handler (frame= {tf_fs = -1061421048, tf_es = -987889624, tf_ds = -1061355480, tf_edi = -982889960, tf_esi = -982889984, tf_ebp = -987837 328, tf_isp = -987837348, tf_ebx = 500, tf_edx = 2000, tf_ecx = 500, tf_eax = 500, tf_trapno = 0, tf_err = 0, tf_eip = -1061420 683, tf_cs = 32, tf_eflags = 514, tf_esp = -987837284, tf_ss = -1068612521}) at /usr/src/sys/i386/i386/mp_machdep.c:1267 #10 0xc0bca32e in Xipi_intr_bitmap_handler () at apic_vector.s:298 #11 0xc0bc0175 in acpi_cpu_c1 () at /usr/src/sys/i386/acpica/acpi_machdep.c:556 #12 0xc04e4457 in acpi_cpu_idle () at /usr/src/sys/dev/acpica/acpi_cpu.c:938 #13 0xc0bd372b in cpu_idle_acpi (busy=1) at /usr/src/sys/i386/i386/machdep.c:1225 #14 0xc0bd4fab in cpu_idle (busy=1) at /usr/src/sys/i386/i386/machdep.c:1313 #15 0xc08c0ea3 in sched_idletd (dummy=0x0) at /usr/src/sys/kern/sched_ule.c:2550 #16 0xc0874a21 in fork_exit (callout=0xc08c0c60 , arg=0x0, frame=0xc51ecd38) at /usr/src/sys/kern/kern_fork.c:844 #17 0xc0bc9c80 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:270 ------------------------------------------------------------------------ ps -axl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 -68 0 0 0 - DLs ?? -6173364:-59.55 [kernel] 0 1 0 0 57 0 2912 0 wait DLs ?? 23667260:40.00 [init] 0 2 0 0 -8 0 0 0 - RL ?? 12686974:36.00 [g_event] 0 3 0 0 -8 0 0 0 - DL ?? -15178934:-19.55 [g_up] 0 4 0 0 -8 0 0 0 - DL ?? -7287166:-35.55 [g_down] 0 5 0 0 -16 0 0 0 idle DL ?? 0:00.00 [ciss_notif 0 6 0 0 -16 0 0 0 - DL ?? 12943975:12.00 [fdc0] 0 7 0 0 -16 0 0 0 waitin DL ?? 2678:44.00 [sctp_itera 0 8 0 0 -60 0 0 0 ccb_sc DL ?? 0:00.00 [xpt_thrd] 0 9 0 0 -32 0 0 0 psleep DL ?? 7092141:12.00 [pagedaemon 0 10 0 0 -16 0 0 0 audit_ DL ?? 0:00.00 [audit] 0 11 0 0 171 0 0 0 - RL ?? 32968188:20.00 [idle] 0 12 0 0 -48 0 0 0 - WL ?? -11341039:-55.55 [intr] 0 13 0 0 -16 0 0 0 - DL ?? 30424824:28.00 [yarrow] 0 14 0 0 -64 0 0 0 - DL ?? 13686869:12.00 [usb] 0 15 0 0 -16 0 0 0 tzpoll DL ?? 27702534:56.00 [acpi_therm 0 16 0 0 -16 0 0 0 coolin DL ?? 2168942:40.00 [acpi_cooli 0 17 0 0 -32 0 0 0 psleep DL ?? 571:52.00 [vmdaemon] 0 18 0 0 76 0 0 0 pgzero DL ?? 50512:56.00 [pagezero] 0 19 0 0 44 0 0 0 psleep DL ?? -31897972:-55.55 [bufdaemon] 0 20 0 0 44 0 0 0 vlruwt DL ?? 21485856:24.00 [vnlru] 0 21 0 0 44 0 0 0 syncer DL ?? 8000403:40.00 [syncer] 0 22 0 0 44 0 0 0 sdflus DL ?? 2343381:00.00 [softdepflu 0 23 0 0 -32 0 0 0 flowcl DL ?? 8480586:32.00 [flowcleane 0 597 1 0 44 0 1888 0 select Ds ?? 4614079:48.00 [devd] 0 781 1 0 44 0 3348 0 select Ds ?? 8765502:36.00 [syslogd] 53 868 1 0 44 0 27036 0 kqread Ds ?? 34742034:00.00 [named] 0 1148 1 0 44 0 6704 0 select Ds ?? 406759:40.00 [sshd] 0 1156 1 0 44 0 6080 0 select Ds ?? -25313389:-59.55 [sendmail] 25 1160 1 0 44 0 6080 0 pause Ds ?? 858881:08.00 [sendmail] 0 1167 1 0 44 0 3376 0 nanslp Ds ?? 6660384:16.00 [cron] 0 1262 1 0 44 0 3348 0 select Ds ?? 5298226:40.00 [syslogd] 0 1370 1 0 44 0 14144 0 select Ds ?? 22528955:48.00 [perl] 88 1382 1 0 76 0 3628 0 wait Ds ?? 801893:40.00 [sh] 88 1464 1382 0 76 0 196516 0 sigwai D ?? -32496359:-31.55 [mysqld] 0 1490 1 0 44 0 3376 0 nanslp Ds ?? 6814799:56.00 [cron] 0 1608 1 0 44 0 3348 0 select Ds ?? 5192087:24.00 [syslogd] 0 1737 1 0 44 0 10144 0 select Ds ?? 3687423:08.00 [httpd] 0 1762 1 0 44 0 3376 0 nanslp Ds ?? 6611521:36.00 [cron] 0 1876 1 0 44 0 3348 0 select Ds ?? 5388062:16.00 [syslogd] 80 1877 1737 0 76 0 10144 0 accept D ?? 139905:28.00 [httpd] 80 1878 1737 0 76 0 10144 0 accept D ?? 117199:52.00 [httpd] 80 1879 1737 0 76 0 10144 0 accept D ?? 121229:12.00 [httpd] 80 1880 1737 0 76 0 10144 0 accept D ?? 103000:28.00 [httpd] 80 1881 1737 0 76 0 10144 0 accept D ?? 84894:08.00 [httpd] 0 2015 1 0 44 0 3376 0 nanslp Ds ?? 7286703:16.00 [cron] 0 2384 1 0 44 0 3348 0 select Ds ?? 5222692:56.00 [syslogd] 0 2520 1 0 44 0 3376 0 nanslp Ds ?? 6634484:00.00 [cron] 0 2615 1 0 76 0 3348 0 ttyin Ds+ ?? 301545:00.00 [getty] 0 2616 1 0 76 0 3348 0 ttyin Ds+ ?? 265864:52.00 [getty] 0 2617 1 0 76 0 3348 0 ttyin Ds+ ?? 295432:52.00 [getty] 0 2618 1 0 76 0 3348 0 ttyin Ds+ ?? 296500:32.00 [getty] 0 2619 1 0 76 0 3348 0 ttyin Ds+ ?? 291527:44.00 [getty] 0 2620 1 0 76 0 3348 0 ttyin Ds+ ?? 299679:52.00 [getty] 0 2621 1 0 76 0 3348 0 ttyin Ds+ ?? 250806:20.00 [getty] 0 2622 1 0 76 0 3348 0 ttyin Ds+ ?? 243478:24.00 [getty] 0 14090 1 0 44 0 3348 0 select Ds ?? 1184109:28.00 [syslogd] 0 14234 1 0 44 0 3376 0 nanslp Ds ?? 1905958:24.00 [cron] 0 16417 1148 0 44 0 9428 0 select Ds ?? 17866613:08.00 [sshd] 0 16419 16417 0 47 0 4628 0 pause Ds ?? 790395:56.00 [csh] 0 16423 16419 0 44 0 4564 0 wait D ?? 802982:24.00 [bash] 0 16503 16423 0 76 0 4564 0 ttyin D+ ?? 4508128:48.00 [bash] 0 17048 1 0 44 0 20588 0 select Ds ?? 14844112:40.00 [httpd] 0 17071 1148 0 44 0 9428 0 select Ds ?? 11617857:44.00 [sshd] 0 17073 17071 0 48 0 4628 0 pause Ds ?? 779509:48.00 [csh] 0 17077 17073 0 44 0 4564 0 ttyin D+ ?? 842272:56.00 [bash] 0 35113 1370 0 -84 0 0 0 - ZW ?? 0:00.00 0 54672 1148 0 49 0 9428 0 sbwait Ds ?? 3537124:56.00 [sshd] 1001 55132 54672 0 45 0 9428 0 select D ?? 331324:20.00 [sshd] 1001 55162 55132 0 46 0 4564 0 ttyin Ds+ ?? 501825:00.00 [bash] 80 85936 17048 0 46 0 22636 0 accept D ?? 0:00.00 [httpd] 80 85937 17048 0 52 0 20588 0 accept D ?? 0:00.00 [httpd] 80 85938 17048 0 53 0 20588 0 accept D ?? 0:00.00 [httpd] 80 85939 17048 0 52 0 20588 0 accept D ?? 0:00.00 [httpd] 80 85940 17048 0 52 0 20588 0 accept D ?? 0:00.00 [httpd] 80 86526 17048 0 50 0 20588 0 accept D ?? 0:00.00 [httpd] ------------------------------------------------------------------------ Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RELEASE-p2 #0: Wed Jan 19 00:41:12 CET 2011 root@airport-new.bartsplace.net:/usr/obj/usr/src/sys/AIRPORT i386 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3180.87-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf25 Family = f Model = 2 Stepping = 5 Features=0xbfebfbff Features2=0x4400 real memory = 2147483648 (2048 MB) avail memory = 2091089920 (1994 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) x 2 HTT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP/HT): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP/HT): APIC ID: 7 ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20100331/tbfadt-707) ACPI Warning: Invalid length for Pm1bControlBlock: 32, using default 16 (20100331/tbfadt-707) MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-15 on motherboard ioapic1 irqs 16-31 on motherboard ioapic2 irqs 32-47 on motherboard ioapic3 irqs 48-63 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: on acpi0 pci0: on pcib0 vgapci0: port 0x2400-0x24ff mem 0xf6000000-0xf6ffffff,0xf5ff0000-0xf5ff0fff at device 3.0 on pci0 pci0: at device 4.0 (no driver attached) pci0: at device 4.2 (no driver attached) isab0: at device 15.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2000-0x200f at device 15.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] ohci0: mem 0xf5ef0000-0xf5ef0fff irq 7 at device 15.2 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 pcib1: on acpi0 pci1: on pcib1 ciss0: port 0x3000-0x30ff mem 0xf7dc0000-0xf7dfffff,0xf7cf0000-0xf7cf3fff irq 30 at device 3.0 on pci1 ciss0: PERFORMANT Transport ciss0: [ITHREAD] pcib2: on acpi0 pci2: on pcib2 bge0: mem 0xf7ef0000-0xf7efffff irq 29 at device 1.0 on pci2 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:0e:7f:20:c2:ed bge0: [ITHREAD] bge1: mem 0xf7ee0000-0xf7eeffff irq 31 at device 2.0 on pci2 miibus1: on bge1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge1: Ethernet address: 00:0e:7f:20:c2:ec bge1: [ITHREAD] pcib3: on acpi0 pci3: on pcib3 pcib4: on acpi0 pci6: on pcib4 pci6: at device 30.0 (no driver attached) acpi_tz0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] fdc0: port 0x3f2-0x3f5 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xcd7ff,0xee000-0xeffff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atrtc0: at port 0x70 irq 8 on isa0 ppc0: parallel port not found. p4tcc0: on cpu0 p4tcc1: on cpu1 p4tcc2: on cpu2 p4tcc3: on cpu3 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert enabled, nat loadable, rule-based forwarding enabled, default to deny, logging disabled load_dn_sched dn_sched FIFO loaded load_dn_sched dn_sched PRIO loaded load_dn_sched dn_sched QFQ loaded load_dn_sched dn_sched RR loaded load_dn_sched dn_sched WF2Q+ loaded usbus0: 12Mbps Full Speed USB v1.0 ugen0.1: <(0x1166)> at usbus0 uhub0: <(0x1166) OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 uhub0: 4 ports with 4 removable, self powered acd0: CDROM at ata0-master PIO4 da0 at ciss0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device da0: 135.168MB/s transfers da0: Command Queueing enabled da0: 34727MB (71122560 512 byte sectors: 255H 32S/T 8716C) da1 at ciss0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-0 device da1: 135.168MB/s transfers da1: Command Queueing enabled da1: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C) SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/da0s1a ..... (lots of 'spam' from ipfw denying packets) kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x24 fault code = supervisor write, page not present instruction pointer = 0x20:0xc08c07f7 stack pointer = 0x28:0xc51ecb98 frame pointer = 0x28:0xc51ecbc8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 11 (idle: cpu1) trap number = 12 panic: page fault cpuid = 1 Uptime: 14h41m50s Physical memory: 2035 MB Dumping 253 MB: 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 -------------------------------------------------------------------------------- kernel config is a copy of GENERIC with the following options added: options DUMMYNET options IPFIREWALL_FORWARD options IPDIVERT options IPFIREWALL_VERBOSE_LIMIT=1000 options IPFIREWALL_VERBOSE options IPFIREWALL This is to enable IPFIREWALL_FORWARD, which is an absolutely mandatory feature for this configuration. If any other information is needed, I have multiple crashdumps available. >How-To-Repeat: No known scenario, other then leaving the machine powered on, and compiling or running software. >Fix: Not a fix but a workaround, using 4BSD scheduler does not result in crashes so far. >Release-Note: >Audit-Trail: >Unformatted: