From owner-freebsd-bugs@FreeBSD.ORG Fri Dec 10 04:40:23 2004 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5C74616A4CF for ; Fri, 10 Dec 2004 04:40:23 +0000 (GMT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0FF8343D5E for ; Fri, 10 Dec 2004 04:40:23 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.1/8.13.1) with ESMTP id iBA4eNYE080353 for ; Fri, 10 Dec 2004 04:40:23 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.1/8.13.1/Submit) id iBA4eMQx080352; Fri, 10 Dec 2004 04:40:22 GMT (envelope-from gnats) Resent-Date: Fri, 10 Dec 2004 04:40:22 GMT Resent-Message-Id: <200412100440.iBA4eMQx080352@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Geoffrey Lassner Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BD28816A4CE for ; Fri, 10 Dec 2004 04:30:43 +0000 (GMT) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id 853C543D54 for ; Fri, 10 Dec 2004 04:30:43 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id iBA4UhCc086664 for ; Fri, 10 Dec 2004 04:30:43 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id iBA4Uhmc086663; Fri, 10 Dec 2004 04:30:43 GMT (envelope-from nobody) Message-Id: <200412100430.iBA4Uhmc086663@www.freebsd.org> Date: Fri, 10 Dec 2004 04:30:43 GMT From: Geoffrey Lassner To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Subject: misc/74909: SMP crash FreeBSD-5.3-p2 (swi1: net) / ipllog X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Dec 2004 04:40:23 -0000 >Number: 74909 >Category: misc >Synopsis: SMP crash FreeBSD-5.3-p2 (swi1: net) / ipllog >Confidential: no >Severity: critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Dec 10 04:40:22 GMT 2004 >Closed-Date: >Last-Modified: >Originator: Geoffrey Lassner >Release: 5.3-RELEASE-p2 >Organization: None >Environment: FreeBSD oak.zyni.com 5.3-RELEASE-p2 FreeBSD 5.3-RELEASE-p2 #2: Wed Dec 8 22:06:28 MST 2004 root@oak.zyni.com:/usr/src/sys/i386/compile/OAK-TEST i386 >Description: Crashes with SMP enabled on 5.3 and 5.3-p2 (two systems identical hardware) Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 00 fault virtual address = 0x4 fault code = supervisor write, page not present instruction pointer = 0x8:0xc0464757 stack pointer = 0x10:0xd4233b10 frame pointer = 0x10:0xd4233b1c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 31 (swi1: net) [thread 100027] Stopped at ipllog+0x43: incl 0x4(%eax) db> show thread 100115 (0xc1f2e7d0) fork_trampoline() at fork_trampoline 100100 (0xc1f2d000) sched_switch(c1f2d000,0,1) at sched_switch+0x143 100066 (0xc1a4e960) sched_switch(c1a4e960,0,2) at sched_switch+0x143 console 3%Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 00 fault virtual address = 0x4 fault code = supervisor write, page not present instruction pointer = 0x8:0xc0464757 stack pointer = 0x10:0xd4233b10 frame pointer = 0x10:0xd4233b1c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 31 (swi1: net) [thread 100027] Stopped at ipllog+0x43: incl 0x4(%eax) db> show thread 100115 (0xc1f2e7d0) fork_trampoline() at fork_trampoline 100100 (0xc1f2d000) sched_switch(c1f2d000,0,1) at sched_switch+0x143 100066 (0xc1a4e960) sched_switch(c1a4e960,0,2) at sched_switch+0x143 100051 (0xc1a4b190) sched_switch(c1a4b190,0,2) at sched_switch+0x143 100118 (0xc1f2ec80) sched_switch(c1f2ec80,0,1) at sched_switch+0x143 100102 (0xc1f2d320) sched_switch(c1f2d320,0,1) at sched_switch+0x143 100144 (0xc20d0640) sched_switch(c20d0640,0,1) at sched_switch+0x143 100104 (0xc1f2d640) sched_switch(c1f2d640,0,1) at sched_switch+0x143 100145 (0xc20d07d0) sched_switch(c20d07d0,0,1) at sched_switch+0x143 100121 (0xc1f53190) sched_switch(c1f53190,0,2) at sched_switch+0x143 100186 (0xc237e960) sched_switch(c237e960,0,1) at sched_switch+0x143 100138 (0xc20cfc80) sched_switch(c20cfc80,0,1) at sched_switch+0x143 100119 (0xc1f2ee10) sched_switch(c1f2ee10,0,2) at sched_switch+0x143 100135 (0xc20cf7d0) sched_switch(c20cf7d0,0,1) at sched_switch+0x143 100134 (0xc20cf640) sched_switch(c20cf640,0,1) at sched_switch+0x143 100058 (0xc1a4bc80) sched_switch(c1a4bc80,0,1) at sched_switch+0x143 100122 (0xc1f53320) sched_switch(c1f53320,0,1) at sched_switch+0x143 geoff@cyclone % more console Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 00 fault virtual address = 0x4 fault code = supervisor write, page not present instruction pointer = 0x8:0xc0464757 stack pointer = 0x10:0xd4233b10 frame pointer = 0x10:0xd4233b1c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 31 (swi1: net) [thread 100027] Stopped at ipllog+0x43: incl 0x4(%eax) db> show thread 100115 (0xc1f2e7d0) fork_trampoline() at fork_trampoline 100100 (0xc1f2d000) sched_switch(c1f2d000,0,1) at sched_switch+0x143 100066 (0xc1a4e960) sched_switch(c1a4e960,0,2) at sched_switch+0x143 100051 (0xc1a4b190) sched_switch(c1a4b190,0,2) at sched_switch+0x143 100118 (0xc1f2ec80) sched_switch(c1f2ec80,0,1) at sched_switch+0x143 100102 (0xc1f2d320) sched_switch(c1f2d320,0,1) at sched_switch+0x143 100144 (0xc20d0640) sched_switch(c20d0640,0,1) at sched_switch+0x143 100104 (0xc1f2d640) sched_switch(c1f2d640,0,1) at sched_switch+0x143 100145 (0xc20d07d0) sched_switch(c20d07d0,0,1) at sched_switch+0x143 100121 (0xc1f53190) sched_switch(c1f53190,0,2) at sched_switch+0x143 100186 (0xc237e960) sched_switch(c237e960,0,1) at sched_switch+0x143 100138 (0xc20cfc80) sched_switch(c20cfc80,0,1) at sched_switch+0x143 100119 (0xc1f2ee10) sched_switch(c1f2ee10,0,2) at sched_switch+0x143 100135 (0xc20cf7d0) sched_switch(c20cf7d0,0,1) at sched_switch+0x143 100134 (0xc20cf640) sched_switch(c20cf640,0,1) at sched_switch+0x143 100058 (0xc1a4bc80) sched_switch(c1a4bc80,0,1) at sched_switch+0x143 100122 (0xc1f53320) sched_switch(c1f53320,0,1) at sched_switch+0x143 100081 (0xc1c6f190) sched_switch(c1c6f190,0,2) at sched_switch+0x143 100083 (0xc1c6f4b0) sched_switch(c1c6f4b0,0,1) at sched_switch+0x143 100069 (0xc1a4ee10) sched_switch(c1a4ee10,0,1) at sched_switch+0x143 100075 (0xc1c6e7d0) sched_switch(c1c6e7d0,0,1) at sched_switch+0x143 100113 (0xc1f2e4b0) sched_switch(c1f2e4b0,0,1) at sched_switch+0x143 100093 (0xc1c724b0) sched_switch(c1c724b0,0,1) at sched_switch+0x143 100084 (0xc1c6f640) sched_switch(c1c6f640,0,1) at sched_switch+0x143 100055 (0xc1a4b7d0) sched_switch(c1a4b7d0,0,1) at sched_switch+0x143 100062 (0xc1a4e320) sched_switch(c1a4e320,0,1) at sched_switch+0x143 100070 (0xc1c6e000) sched_switch(c1c6e000,0,1) at sched_switch+0x143 100111 (0xc1f2e190) sched_switch(c1f2e190,0,2) at sched_switch+0x143 100078 (0xc1c6ec80) sched_switch(c1c6ec80,0,1) at sched_switch+0x143 100110 (0xc1f2e000) sched_switch(c1f2e000,0,1) at sched_switch+0x143 100098 (0xc1c72c80) sched_switch(c1c72c80,0,1) at sched_switch+0x143 100099 (0xc1c72e10) sched_switch(c1c72e10,0,1) at sched_switch+0x143 100036 (0xc19a1960) sched_switch(c19a1960,0,1) at sched_switch+0x143 100085 (0xc1c6f7d0) sched_switch(c1c6f7d0,0,1) at sched_switch+0x143 100082 (0xc1c6f320) sched_switch(c1c6f320,0,1) at sched_switch+0x143 100112 (0xc1f2e320) sched_switch(c1f2e320,0,1) at sched_switch+0x143 100056 (0xc1a4b960) sched_switch(c1a4b960,0,1) at sched_switch+0x143 100109 (0xc1f2de10) sched_switch(c1f2de10,0,1) at sched_switch+0x143 100117 (0xc1f2eaf0) sched_switch(c1f2eaf0,0,1) at sched_switch+0x143 100092 (0xc1c72320) sched_switch(c1c72320,0,1) at sched_switch+0x143 100090 (0xc1c72000) sched_switch(c1c72000,0,1) at sched_switch+0x143 100080 (0xc1c6f000) sched_switch(c1c6f000,0,1) at sched_switch+0x143 100087 (0xc1c6faf0) sched_switch(c1c6faf0,0,1) at sched_switch+0x143 100065 (0xc1a4e7d0) sched_switch(c1a4e7d0,0,1) at sched_switch+0x143 100067 (0xc1a4eaf0) sched_switch(c1a4eaf0,0,1) at sched_switch+0x143 100101 (0xc1f2d190) sched_switch(c1f2d190,0,1) at sched_switch+0x143 100106 (0xc1f2d960) sched_switch(c1f2d960,0,1) at sched_switch+0x143 100126 (0xc1f53960) sched_switch(c1f53960,0,1) at sched_switch+0x143 100089 (0xc1c6fe10) sched_switch(c1c6fe10,0,1) at sched_switch+0x143 100096 (0xc1c72960) sched_switch(c1c72960,0,1) at sched_switch+0x143 100086 (0xc1c6f960) sched_switch(c1c6f960,0,1) at sched_switch+0x143 100039 (0xc19a1e10) sched_switch(c19a1e10,0,1) at sched_switch+0x143 100095 (0xc1c727d0) sched_switch(c1c727d0,0,1) at sched_switch+0x143 100057 (0xc1a4baf0) sched_switch(c1a4baf0,0,1) at sched_switch+0x143 100068 (0xc1a4ec80) sched_switch(c1a4ec80,0,1) at sched_switch+0x143 100091 (0xc1c72190) sched_switch(c1c72190,0,1) at sched_switch+0x143 100064 (0xc1a4e640) sched_switch(c1a4e640,0,1) at sched_switch+0x143 100063 (0xc1a4e4b0) sched_switch(c1a4e4b0,0,1) at sched_switch+0x143 100052 (0xc1a4b320) sched_switch(c1a4b320,0,1) at sched_switch+0x143 100094 (0xc1c72640) sched_switch(c1c72640,0,1) at sched_switch+0x143 100127 (0xc1f53af0) sched_switch(c1f53af0,0,2) at sched_switch+0x143 100128 (0xc1f53c80) sched_switch(c1f53c80,0,1) at sched_switch+0x143 100103 (0xc1f2d4b0) sched_switch(c1f2d4b0,0,1) at sched_switch+0x143 100097 (0xc1c72af0) sched_switch(c1c72af0,0,1) at sched_switch+0x143 100123 (0xc1f534b0) sched_switch(c1f534b0,0,1) at sched_switch+0x143 100108 (0xc1f2dc80) sched_switch(c1f2dc80,0,1) at sched_switch+0x143 100088 (0xc1c6fc80) sched_switch(c1c6fc80,0,1) at sched_switch+0x143 100059 (0xc1a4be10) sched_switch(c1a4be10,0,1) at sched_switch+0x143 100073 (0xc1c6e4b0) sched_switch(c1c6e4b0,0,1) at sched_switch+0x143 100072 (0xc1c6e320) sched_switch(c1c6e320,0,1) at sched_switch+0x143 100060 (0xc1a4e000) sched_switch(c1a4e000,0,1) at sched_switch+0x143 100038 (0xc19a1c80) sched_switch(c19a1c80,0,1) at sched_switch+0x143 100071 (0xc1c6e190) sched_switch(c1c6e190,0,1) at sched_switch+0x143 100053 (0xc1a4b4b0) sched_switch(c1a4b4b0,0,1) at sched_switch+0x143 100079 (0xc1c6ee10) sched_switch(c1c6ee10,0,1) at sched_switch+0x143 100074 (0xc1c6e640) sched_switch(c1c6e640,0,1) at sched_switch+0x143 100054 (0xc1a4b640) sched_switch(c1a4b640,0,1) at sched_switch+0x143 100076 (0xc1c6e960) sched_switch(c1c6e960,0,1) at sched_switch+0x143 100061 (0xc1a4e190) sched_switch(c1a4e190,0,1) at sched_switch+0x143 100077 (0xc1c6eaf0) sched_switch(c1c6eaf0,0,1) at sched_switch+0x143 100025 (0xc196e7d0) sched_switch(c196e7d0,0,1) at sched_switch+0x143 100024 (0xc196e640) sched_switch(c196e640,0,1) at sched_switch+0x143 100023 (0xc196e4b0) sched_switch(c196e4b0,0,1) at sched_switch+0x143 100049 (0xc1a48e10) sched_switch(c1a48e10,0,1) at sched_switch+0x143 100048 (0xc1a48c80) sched_switch(c1a48c80,0,1) at sched_switch+0x143 100047 (0xc1a48af0) sched_switch(c1a48af0,0,1) at sched_switch+0x143 100046 (0xc1a48960) sched_switch(c1a48960,0,1) at sched_switch+0x143 100045 (0xc1a487d0) sched_switch(c1a487d0,0,1) at sched_switch+0x143 100044 (0xc1a48640) sched_switch(c1a48640,0,1) at sched_switch+0x143 100043 (0xc1a484b0) sched_switch(c1a484b0,0,1) at sched_switch+0x143 100042 (0xc1a48320) sched_switch(c1a48320,0,1) at sched_switch+0x143 100041 (0xc1a48190) sched_switch(c1a48190,0,1) at sched_switch+0x143 100040 (0xc1a48000) fork_trampoline() at fork_trampoline 100012 (0xc196a320) fork_trampoline() at fork_trampoline 100011 (0xc196a190) sched_switch(c196a190,0,1) at sched_switch+0x143 100035 (0xc19a17d0) sched_switch(c19a17d0,0,1) at sched_switch+0x143 100034 (0xc19a1640) fork_trampoline() at fork_trampoline 100033 (0xc19a14b0) fork_trampoline() at fork_trampoline 100032 (0xc19a1320) sched_switch(c19a1320,0,1) at sched_switch+0x143 100031 (0xc19a1190) sched_switch(c19a1190,0,1) at sched_switch+0x143 100030 (0xc19a1000) sched_switch(18f1420,c1b0d800,c1c34800,c1bcab00,d5413c7c) at sched_switch+ 0x143 100029 (0xc196ee10) sched_switch(c196ee10,0,1) at sched_switch+0x143 100028 (0xc196ec80) sched_switch(c196ec80,0,1) at sched_switch+0x143 100027 (0xc196eaf0) ipllog(0,d4233bd8,d4233b50,d4233b48,d4233b40) at ipllog+0x43 100026 (0xc196e960) fork_trampoline() at fork_trampoline 100000 (0xc1965000) sched_switch(c1965000,0,1) at sched_switch+0x143 100022 (0xc196e320) fork_trampoline() at fork_trampoline 100021 (0xc196e190) fork_trampoline() at fork_trampoline 100020 (0xc196e000) fork_trampoline() at fork_trampoline 100019 (0xc196ae10) fork_trampoline() at fork_trampoline 100018 (0xc196ac80) fork_trampoline() at fork_trampoline 100017 (0xc196aaf0) sched_switch(c196aaf0,0,1) at sched_switch+0x143 100016 (0xc196a960) sched_switch(c196a960,0,1) at sched_switch+0x143 100015 (0xc196a7d0) sched_switch(c196a7d0,0,1) at sched_switch+0x143 100014 (0xc196a640) fork_trampoline() at fork_trampoline 100013 (0xc196a4b0) sched_switch(c196a4b0,0,1) at sched_switch+0x143 100010 (0xc196a000) fork_trampoline() at fork_trampoline 100009 (0xc1965e10) fork_trampoline() at fork_trampoline 100008 (0xc1965c80) fork_trampoline() at fork_trampoline 100007 (0xc1965af0) fork_trampoline() at fork_trampoline 100006 (0xc1965960) fork_trampoline() at fork_trampoline 100005 (0xc19657d0) fork_trampoline() at fork_trampoline 100004 (0xc1965640) sched_switch(c1965640,0,1) at sched_switch+0x143 100003 (0xc19654b0) sched_switch(c19654b0,0,1) at sched_switch+0x143 100002 (0xc1965320) sched_switch(c1965320,0,1) at sched_switch+0x143 100001 (0xc1965190) sched_switch(c1965190,0,1) at sched_switch+0x143 0 (0xc08bce00) sched_switch(c08bce00,0,1) at sched_switch+0x143 db> show all procs pid proc uarea uid ppid pgrp flag stat wmesg wchan cmd 28532 c20d5388 d6a8a000 0 753 753 0000002 [RUNQ] find 27240 c20d554c d6a8b000 0 735 27240 0004002 [SLPQ nanslp 0xc08c256c][SLP] sleep 27239 c20d4e20 d6a87000 0 735 27239 0004002 [RUNQ] tcpdump 23076 c20d4000 d6a7f000 1000 23075 23075 0004002 [RUNQ] ssh 23075 c20d4710 d6a83000 1000 718 23075 0004002 [SLPQ pipdwt 0xc2268480][SLP] scp 17334 c20d41c4 d6a80000 0 17331 17320 0000002 [SLPQ pause 0xc20d41fc][SLP] dump 17333 c20ce710 d6a6b000 0 17331 17320 0000002 [SLPQ pause 0xc20ce748][SLP] dump 17332 c20d454c d6a82000 0 17331 17320 0000002 [SLPQ pipdwt 0xc2268c00][SLP] dump 17331 c20ce54c d6a6a000 0 17320 17320 0000002 [SLPQ sbwait 0xc209ee84][SLP] dump 17321 c20d51c4 d6a89000 0 732 17320 0004002 [RUNQ] gzip 17320 c2378388 d6aff000 0 732 17320 0004002 [SLPQ wait 0xc2378388][SLP] dump 753 c20d11c4 d6a71000 0 724 753 0004003 [SLPQ wait 0xc20d11c4][SLP] find 752 c20d4c5c d6a86000 1000 700 752 0004002 [RUNQ] bzip2 743 c20d1710 d6a74000 0 740 738 0000002 [SLPQ pause 0xc20d1748][SLP] dump 742 c20d18d4 d6a75000 0 740 738 0000002 [SLPQ pause 0xc20d190c][SLP] dump 741 c1c5c388 d6896000 0 740 738 0000002 [SLPQ pipdwt 0xc1d9f180][SLP] dump 740 c2039000 d689e000 0 738 738 0000002 [SLPQ sbwait 0xc209eab8][SLP] dump 739 c1f29710 d693f000 0 729 738 0004002 [RUNQ] gzip 738 c1f29388 d693d000 0 729 738 0004002 [SLPQ wait 0xc1f29388][SLP] dump 735 c1c71388 d68e5000 0 712 735 0004003 [SLPQ pause 0xc1c713c0][SLP] tcsh 732 c1c6d8d4 d68df000 0 706 732 0004003 [SLPQ pause 0xc1c6d90c][SLP] tcsh 729 c20391c4 d6918000 0 694 729 0004003 [SLPQ pause 0xc20391fc][SLP] tcsh 724 c1f281c4 d6928000 1000 723 724 0004002 [SLPQ pause 0xc1f281fc][SLP] tcsh 723 c1f291c4 d693c000 1000 690 690 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 718 c1c5c8d4 d6899000 1000 717 718 0004002 [SLPQ pause 0xc1c5c90c][SLP] tcsh 717 c1a4ac5c d54a5000 1000 687 687 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 712 c1c711c4 d68e4000 1000 711 712 0004002 [SLPQ pause 0xc1c711fc][SLP] tcsh 711 c1f29a98 d697d000 1000 684 684 0000100 [RUNQ] sshd 706 c1c6d388 d68dc000 1000 705 706 0004002 [SLPQ pause 0xc1c6d3c0][SLP] tcsh 705 c1f29c5c d697e000 1000 681 681 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 700 c203954c d698d000 1000 699 700 0004002 [SLPQ pause 0xc2039584][SLP] tcsh 699 c2039710 d698f000 1000 678 678 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 694 c1a4a710 d5466000 1000 693 694 0004002 [SLPQ pause 0xc1a4a748][SLP] tcsh 693 c1f29000 d693b000 1000 675 675 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 690 c1f2954c d693e000 0 461 690 0000100 [SLPQ sbwait 0xc209b5a8][SLP] sshd 687 c1f298d4 d6940000 0 461 687 0000100 [SLPQ sbwait 0xc1f85e84][SLP] sshd 684 c1c5c710 d6898000 0 461 684 0000100 [SLPQ sbwait 0xc1f85bfc][SLP] sshd 681 c1f29e20 d697f000 0 461 681 0000100 [SLPQ sbwait 0xc1d95d40][SLP] sshd 678 c2039388 d692f000 0 461 678 0000100 [SLPQ sbwait 0xc1f86320][SLP] sshd 675 c1f28388 d6930000 0 461 675 0000100 [SLPQ sbwait 0xc1f86974][SLP] sshd 674 c1f28710 d6936000 0 617 617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5 673 c1c6d000 d689d000 0 617 617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5 geoff@cyclone % geoff@cyclone % geoff@cyclone % geoff@cyclone % geoff@cyclone % geoff@cyclone % geoff@cyclone % geoff@cyclone % geoff@cyclone % ls console instruction_pointer mptable geoff@cyclone % cat console Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 00 fault virtual address = 0x4 fault code = supervisor write, page not present instruction pointer = 0x8:0xc0464757 stack pointer = 0x10:0xd4233b10 frame pointer = 0x10:0xd4233b1c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 31 (swi1: net) [thread 100027] Stopped at ipllog+0x43: incl 0x4(%eax) db> show thread 100115 (0xc1f2e7d0) fork_trampoline() at fork_trampoline 100100 (0xc1f2d000) sched_switch(c1f2d000,0,1) at sched_switch+0x143 100066 (0xc1a4e960) sched_switch(c1a4e960,0,2) at sched_switch+0x143 100051 (0xc1a4b190) sched_switch(c1a4b190,0,2) at sched_switch+0x143 100118 (0xc1f2ec80) sched_switch(c1f2ec80,0,1) at sched_switch+0x143 100102 (0xc1f2d320) sched_switch(c1f2d320,0,1) at sched_switch+0x143 100144 (0xc20d0640) sched_switch(c20d0640,0,1) at sched_switch+0x143 100104 (0xc1f2d640) sched_switch(c1f2d640,0,1) at sched_switch+0x143 100145 (0xc20d07d0) sched_switch(c20d07d0,0,1) at sched_switch+0x143 100121 (0xc1f53190) sched_switch(c1f53190,0,2) at sched_switch+0x143 100186 (0xc237e960) sched_switch(c237e960,0,1) at sched_switch+0x143 100138 (0xc20cfc80) sched_switch(c20cfc80,0,1) at sched_switch+0x143 100119 (0xc1f2ee10) sched_switch(c1f2ee10,0,2) at sched_switch+0x143 100135 (0xc20cf7d0) sched_switch(c20cf7d0,0,1) at sched_switch+0x143 100134 (0xc20cf640) sched_switch(c20cf640,0,1) at sched_switch+0x143 100058 (0xc1a4bc80) sched_switch(c1a4bc80,0,1) at sched_switch+0x143 100122 (0xc1f53320) sched_switch(c1f53320,0,1) at sched_switch+0x143 100081 (0xc1c6f190) sched_switch(c1c6f190,0,2) at sched_switch+0x143 100083 (0xc1c6f4b0) sched_switch(c1c6f4b0,0,1) at sched_switch+0x143 100069 (0xc1a4ee10) sched_switch(c1a4ee10,0,1) at sched_switch+0x143 100075 (0xc1c6e7d0) sched_switch(c1c6e7d0,0,1) at sched_switch+0x143 100113 (0xc1f2e4b0) sched_switch(c1f2e4b0,0,1) at sched_switch+0x143 100093 (0xc1c724b0) sched_switch(c1c724b0,0,1) at sched_switch+0x143 100084 (0xc1c6f640) sched_switch(c1c6f640,0,1) at sched_switch+0x143 100055 (0xc1a4b7d0) sched_switch(c1a4b7d0,0,1) at sched_switch+0x143 100062 (0xc1a4e320) sched_switch(c1a4e320,0,1) at sched_switch+0x143 100070 (0xc1c6e000) sched_switch(c1c6e000,0,1) at sched_switch+0x143 100111 (0xc1f2e190) sched_switch(c1f2e190,0,2) at sched_switch+0x143 100078 (0xc1c6ec80) sched_switch(c1c6ec80,0,1) at sched_switch+0x143 100110 (0xc1f2e000) sched_switch(c1f2e000,0,1) at sched_switch+0x143 100098 (0xc1c72c80) sched_switch(c1c72c80,0,1) at sched_switch+0x143 100099 (0xc1c72e10) sched_switch(c1c72e10,0,1) at sched_switch+0x143 100036 (0xc19a1960) sched_switch(c19a1960,0,1) at sched_switch+0x143 100085 (0xc1c6f7d0) sched_switch(c1c6f7d0,0,1) at sched_switch+0x143 100082 (0xc1c6f320) sched_switch(c1c6f320,0,1) at sched_switch+0x143 100112 (0xc1f2e320) sched_switch(c1f2e320,0,1) at sched_switch+0x143 100056 (0xc1a4b960) sched_switch(c1a4b960,0,1) at sched_switch+0x143 100109 (0xc1f2de10) sched_switch(c1f2de10,0,1) at sched_switch+0x143 100117 (0xc1f2eaf0) sched_switch(c1f2eaf0,0,1) at sched_switch+0x143 100092 (0xc1c72320) sched_switch(c1c72320,0,1) at sched_switch+0x143 100090 (0xc1c72000) sched_switch(c1c72000,0,1) at sched_switch+0x143 100080 (0xc1c6f000) sched_switch(c1c6f000,0,1) at sched_switch+0x143 100087 (0xc1c6faf0) sched_switch(c1c6faf0,0,1) at sched_switch+0x143 100065 (0xc1a4e7d0) sched_switch(c1a4e7d0,0,1) at sched_switch+0x143 100067 (0xc1a4eaf0) sched_switch(c1a4eaf0,0,1) at sched_switch+0x143 100101 (0xc1f2d190) sched_switch(c1f2d190,0,1) at sched_switch+0x143 100106 (0xc1f2d960) sched_switch(c1f2d960,0,1) at sched_switch+0x143 100126 (0xc1f53960) sched_switch(c1f53960,0,1) at sched_switch+0x143 100089 (0xc1c6fe10) sched_switch(c1c6fe10,0,1) at sched_switch+0x143 100096 (0xc1c72960) sched_switch(c1c72960,0,1) at sched_switch+0x143 100086 (0xc1c6f960) sched_switch(c1c6f960,0,1) at sched_switch+0x143 100039 (0xc19a1e10) sched_switch(c19a1e10,0,1) at sched_switch+0x143 100095 (0xc1c727d0) sched_switch(c1c727d0,0,1) at sched_switch+0x143 100057 (0xc1a4baf0) sched_switch(c1a4baf0,0,1) at sched_switch+0x143 100068 (0xc1a4ec80) sched_switch(c1a4ec80,0,1) at sched_switch+0x143 100091 (0xc1c72190) sched_switch(c1c72190,0,1) at sched_switch+0x143 100064 (0xc1a4e640) sched_switch(c1a4e640,0,1) at sched_switch+0x143 100063 (0xc1a4e4b0) sched_switch(c1a4e4b0,0,1) at sched_switch+0x143 100052 (0xc1a4b320) sched_switch(c1a4b320,0,1) at sched_switch+0x143 100094 (0xc1c72640) sched_switch(c1c72640,0,1) at sched_switch+0x143 100127 (0xc1f53af0) sched_switch(c1f53af0,0,2) at sched_switch+0x143 100128 (0xc1f53c80) sched_switch(c1f53c80,0,1) at sched_switch+0x143 100103 (0xc1f2d4b0) sched_switch(c1f2d4b0,0,1) at sched_switch+0x143 100097 (0xc1c72af0) sched_switch(c1c72af0,0,1) at sched_switch+0x143 100123 (0xc1f534b0) sched_switch(c1f534b0,0,1) at sched_switch+0x143 100108 (0xc1f2dc80) sched_switch(c1f2dc80,0,1) at sched_switch+0x143 100088 (0xc1c6fc80) sched_switch(c1c6fc80,0,1) at sched_switch+0x143 100059 (0xc1a4be10) sched_switch(c1a4be10,0,1) at sched_switch+0x143 100073 (0xc1c6e4b0) sched_switch(c1c6e4b0,0,1) at sched_switch+0x143 100072 (0xc1c6e320) sched_switch(c1c6e320,0,1) at sched_switch+0x143 100060 (0xc1a4e000) sched_switch(c1a4e000,0,1) at sched_switch+0x143 100038 (0xc19a1c80) sched_switch(c19a1c80,0,1) at sched_switch+0x143 100071 (0xc1c6e190) sched_switch(c1c6e190,0,1) at sched_switch+0x143 100053 (0xc1a4b4b0) sched_switch(c1a4b4b0,0,1) at sched_switch+0x143 100079 (0xc1c6ee10) sched_switch(c1c6ee10,0,1) at sched_switch+0x143 100074 (0xc1c6e640) sched_switch(c1c6e640,0,1) at sched_switch+0x143 100054 (0xc1a4b640) sched_switch(c1a4b640,0,1) at sched_switch+0x143 100076 (0xc1c6e960) sched_switch(c1c6e960,0,1) at sched_switch+0x143 100061 (0xc1a4e190) sched_switch(c1a4e190,0,1) at sched_switch+0x143 100077 (0xc1c6eaf0) sched_switch(c1c6eaf0,0,1) at sched_switch+0x143 100025 (0xc196e7d0) sched_switch(c196e7d0,0,1) at sched_switch+0x143 100024 (0xc196e640) sched_switch(c196e640,0,1) at sched_switch+0x143 100023 (0xc196e4b0) sched_switch(c196e4b0,0,1) at sched_switch+0x143 100049 (0xc1a48e10) sched_switch(c1a48e10,0,1) at sched_switch+0x143 100048 (0xc1a48c80) sched_switch(c1a48c80,0,1) at sched_switch+0x143 100047 (0xc1a48af0) sched_switch(c1a48af0,0,1) at sched_switch+0x143 100046 (0xc1a48960) sched_switch(c1a48960,0,1) at sched_switch+0x143 100045 (0xc1a487d0) sched_switch(c1a487d0,0,1) at sched_switch+0x143 100044 (0xc1a48640) sched_switch(c1a48640,0,1) at sched_switch+0x143 100043 (0xc1a484b0) sched_switch(c1a484b0,0,1) at sched_switch+0x143 100042 (0xc1a48320) sched_switch(c1a48320,0,1) at sched_switch+0x143 100041 (0xc1a48190) sched_switch(c1a48190,0,1) at sched_switch+0x143 100040 (0xc1a48000) fork_trampoline() at fork_trampoline 100012 (0xc196a320) fork_trampoline() at fork_trampoline 100011 (0xc196a190) sched_switch(c196a190,0,1) at sched_switch+0x143 100035 (0xc19a17d0) sched_switch(c19a17d0,0,1) at sched_switch+0x143 100034 (0xc19a1640) fork_trampoline() at fork_trampoline 100033 (0xc19a14b0) fork_trampoline() at fork_trampoline 100032 (0xc19a1320) sched_switch(c19a1320,0,1) at sched_switch+0x143 100031 (0xc19a1190) sched_switch(c19a1190,0,1) at sched_switch+0x143 100030 (0xc19a1000) sched_switch(18f1420,c1b0d800,c1c34800,c1bcab00,d5413c7c) at sched_switch+0x143 100029 (0xc196ee10) sched_switch(c196ee10,0,1) at sched_switch+0x143 100028 (0xc196ec80) sched_switch(c196ec80,0,1) at sched_switch+0x143 100027 (0xc196eaf0) ipllog(0,d4233bd8,d4233b50,d4233b48,d4233b40) at ipllog+0x43 100026 (0xc196e960) fork_trampoline() at fork_trampoline 100000 (0xc1965000) sched_switch(c1965000,0,1) at sched_switch+0x143 100022 (0xc196e320) fork_trampoline() at fork_trampoline 100021 (0xc196e190) fork_trampoline() at fork_trampoline 100020 (0xc196e000) fork_trampoline() at fork_trampoline 100019 (0xc196ae10) fork_trampoline() at fork_trampoline 100018 (0xc196ac80) fork_trampoline() at fork_trampoline 100017 (0xc196aaf0) sched_switch(c196aaf0,0,1) at sched_switch+0x143 100016 (0xc196a960) sched_switch(c196a960,0,1) at sched_switch+0x143 100015 (0xc196a7d0) sched_switch(c196a7d0,0,1) at sched_switch+0x143 100014 (0xc196a640) fork_trampoline() at fork_trampoline 100013 (0xc196a4b0) sched_switch(c196a4b0,0,1) at sched_switch+0x143 100010 (0xc196a000) fork_trampoline() at fork_trampoline 100009 (0xc1965e10) fork_trampoline() at fork_trampoline 100008 (0xc1965c80) fork_trampoline() at fork_trampoline 100007 (0xc1965af0) fork_trampoline() at fork_trampoline 100006 (0xc1965960) fork_trampoline() at fork_trampoline 100005 (0xc19657d0) fork_trampoline() at fork_trampoline 100004 (0xc1965640) sched_switch(c1965640,0,1) at sched_switch+0x143 100003 (0xc19654b0) sched_switch(c19654b0,0,1) at sched_switch+0x143 100002 (0xc1965320) sched_switch(c1965320,0,1) at sched_switch+0x143 100001 (0xc1965190) sched_switch(c1965190,0,1) at sched_switch+0x143 0 (0xc08bce00) sched_switch(c08bce00,0,1) at sched_switch+0x143 db> show all procs pid proc uarea uid ppid pgrp flag stat wmesg wchan cmd 28532 c20d5388 d6a8a000 0 753 753 0000002 [RUNQ] find 27240 c20d554c d6a8b000 0 735 27240 0004002 [SLPQ nanslp 0xc08c256c][SLP] sleep 27239 c20d4e20 d6a87000 0 735 27239 0004002 [RUNQ] tcpdump 23076 c20d4000 d6a7f000 1000 23075 23075 0004002 [RUNQ] ssh 23075 c20d4710 d6a83000 1000 718 23075 0004002 [SLPQ pipdwt 0xc2268480][SLP] scp 17334 c20d41c4 d6a80000 0 17331 17320 0000002 [SLPQ pause 0xc20d41fc][SLP] dump 17333 c20ce710 d6a6b000 0 17331 17320 0000002 [SLPQ pause 0xc20ce748][SLP] dump 17332 c20d454c d6a82000 0 17331 17320 0000002 [SLPQ pipdwt 0xc2268c00][SLP] dump 17331 c20ce54c d6a6a000 0 17320 17320 0000002 [SLPQ sbwait 0xc209ee84][SLP] dump 17321 c20d51c4 d6a89000 0 732 17320 0004002 [RUNQ] gzip 17320 c2378388 d6aff000 0 732 17320 0004002 [SLPQ wait 0xc2378388][SLP] dump 753 c20d11c4 d6a71000 0 724 753 0004003 [SLPQ wait 0xc20d11c4][SLP] find 752 c20d4c5c d6a86000 1000 700 752 0004002 [RUNQ] bzip2 743 c20d1710 d6a74000 0 740 738 0000002 [SLPQ pause 0xc20d1748][SLP] dump 742 c20d18d4 d6a75000 0 740 738 0000002 [SLPQ pause 0xc20d190c][SLP] dump 741 c1c5c388 d6896000 0 740 738 0000002 [SLPQ pipdwt 0xc1d9f180][SLP] dump 740 c2039000 d689e000 0 738 738 0000002 [SLPQ sbwait 0xc209eab8][SLP] dump 739 c1f29710 d693f000 0 729 738 0004002 [RUNQ] gzip 738 c1f29388 d693d000 0 729 738 0004002 [SLPQ wait 0xc1f29388][SLP] dump 735 c1c71388 d68e5000 0 712 735 0004003 [SLPQ pause 0xc1c713c0][SLP] tcsh 732 c1c6d8d4 d68df000 0 706 732 0004003 [SLPQ pause 0xc1c6d90c][SLP] tcsh 729 c20391c4 d6918000 0 694 729 0004003 [SLPQ pause 0xc20391fc][SLP] tcsh 724 c1f281c4 d6928000 1000 723 724 0004002 [SLPQ pause 0xc1f281fc][SLP] tcsh 723 c1f291c4 d693c000 1000 690 690 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 718 c1c5c8d4 d6899000 1000 717 718 0004002 [SLPQ pause 0xc1c5c90c][SLP] tcsh 717 c1a4ac5c d54a5000 1000 687 687 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 712 c1c711c4 d68e4000 1000 711 712 0004002 [SLPQ pause 0xc1c711fc][SLP] tcsh 711 c1f29a98 d697d000 1000 684 684 0000100 [RUNQ] sshd 706 c1c6d388 d68dc000 1000 705 706 0004002 [SLPQ pause 0xc1c6d3c0][SLP] tcsh 705 c1f29c5c d697e000 1000 681 681 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 700 c203954c d698d000 1000 699 700 0004002 [SLPQ pause 0xc2039584][SLP] tcsh 699 c2039710 d698f000 1000 678 678 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 694 c1a4a710 d5466000 1000 693 694 0004002 [SLPQ pause 0xc1a4a748][SLP] tcsh 693 c1f29000 d693b000 1000 675 675 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 690 c1f2954c d693e000 0 461 690 0000100 [SLPQ sbwait 0xc209b5a8][SLP] sshd 687 c1f298d4 d6940000 0 461 687 0000100 [SLPQ sbwait 0xc1f85e84][SLP] sshd 684 c1c5c710 d6898000 0 461 684 0000100 [SLPQ sbwait 0xc1f85bfc][SLP] sshd 681 c1f29e20 d697f000 0 461 681 0000100 [SLPQ sbwait 0xc1d95d40][SLP] sshd 678 c2039388 d692f000 0 461 678 0000100 [SLPQ sbwait 0xc1f86320][SLP] sshd 675 c1f28388 d6930000 0 461 675 0000100 [SLPQ sbwait 0xc1f86974][SLP] sshd 674 c1f28710 d6936000 0 617 617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5 673 c1c6d000 d689d000 0 617 617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5 672 c1f28c5c d6939000 0 617 617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5 671 c1c71a98 d68e9000 0 617 617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5 670 c1c71710 d68e7000 0 617 617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5 669 c20398d4 d69c2000 0 1 669 0004002 [SLPQ ttyin 0xc1c1d410][SLP] getty 668 c2039a98 d69c3000 0 1 668 0004002 [SLPQ ttyin 0xc1afa210][SLP] getty 667 c2039c5c d69c4000 0 1 667 0004002 [SLPQ ttyin 0xc1afa410][SLP] getty 666 c1f288d4 d6937000 0 1 666 0004002 [SLPQ ttyin 0xc1afa610][SLP] getty 665 c1c71c5c d68ea000 0 1 665 0004002 [SLPQ ttyin 0xc1afa810][SLP] getty 664 c1f28e20 d693a000 0 1 664 0004002 [SLPQ ttyin 0xc1afaa10][SLP] getty 663 c1a4a1c4 d5463000 0 1 663 0004002 [SLPQ ttyin 0xc1a62210][SLP] getty 662 c1c71e20 d6909000 0 1 662 0004002 [SLPQ ttyin 0xc195b610][SLP] getty 661 c1c5c54c d6897000 0 1 661 0004002 [SLPQ ttyin 0xc195b410][SLP] getty 656 c1c7154c d68e6000 0 1 656 0000000 [SLPQ select 0xc08c7cc4][SLP] inetd 625 c1f2854c d6935000 0 1 624 0000000 [SLPQ select 0xc08c7cc4][SLP] snmpd 617 c1a4a8d4 d5467000 0 1 617 0000000 [SLPQ pause 0xc1a4a90c][SLP] perl5.8.5 615 c1a4aa98 d54a4000 20001 560 560 0000100 [SLPQ lockf 0xc1f252c0][SLP] perl 614 c1c5ce20 d689c000 20001 560 560 0000100 [SLPQ accept 0xc1ceb17e][SLP] perl 605 c1f28000 d6919000 0 1 605 0000100 [SLPQ select 0xc08c7cc4][SLP] perl 604 c1c718d4 d68e8000 88 579 47 000c182 (threaded) mysqld thread 0xc1f53af0 ksegrp 0xc1a49b60 [RUNQ] thread 0xc1f53c80 ksegrp 0xc196c460 [SLPQ kserel 0xc196c4a0][SLP] thread 0xc1f2d4b0 ksegrp 0xc1a49b60 [SLPQ kserel 0xc1a49ba0][SLP] thread 0xc1c72af0 ksegrp 0xc1a49b60 [SLPQ select 0xc08c7cc4][SLP] thread 0xc1f534b0 ksegrp 0xc1a49150 [SLPQ sigwait 0xd699ec2c][SLP] thread 0xc1f2dc80 ksegrp 0xc1a491c0 [SLPQ ksesigwait 0xc1c719d4][SLP] 579 c1f28a98 d6938000 0 1 47 0004002 [SLPQ wait 0xc1f28a98][SLP] sh 567 c1c5c1c4 d6895000 106 1 567 0000100 [SLPQ accept 0xc1cebce2][SLP] clamd 560 c1c6dc5c d68e1000 20001 1 560 0000100 [SLPQ select 0xc08c7cc4][SLP] perl 474 c1c6de20 d68e2000 0 1 474 0000000 [SLPQ nanslp 0xc08c256c][SLP] cron 461 c1c5c000 d6893000 0 1 461 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd 434 c1a4a388 d5464000 0 1 47 0004002 [SLPQ nanslp 0xc08c256c][SLP] perl 433 c1c71000 d68e3000 20000 431 431 0004100 [SLPQ select 0xc08c7cc4][SLP] qmgr 432 c1c5cc5c d689b000 20000 431 431 0004100 [SLPQ select 0xc08c7cc4][SLP] pickup 431 c1c6d1c4 d68db000 0 1 431 0004100 [SLPQ select 0xc08c7cc4][SLP] master 384 c1c6da98 d68e0000 65534 1 384 0000100 [SLPQ bpf 0xc1ce6400][SLP] snort 290 c1c5ca98 d689a000 0 1 290 0000000 [SLPQ ffsfsn 0xc1cfb134][SLP] syslogd 270 c1c6d710 d68de000 0 1 270 0000000 [SLPQ select 0xc08c7cc4][SLP] devd 187 c1a4ae20 d54a6000 0 1 187 0000000 [SLPQ pause 0xc1a4ae58][SLP] adjkerntz 165 c1c6d54c d68dd000 0 1 165 0000000 [SLPQ nanslp 0xc08c256c][SLP] ipmon 46 c19b0a98 d5438000 0 0 0 0000204 [SLPQ - 0xd422dd18][SLP] schedcpu 45 c19b0c5c d5439000 0 0 0 0000204 [SLPQ hptrdy 0xc0768be4][SLP] hpt_wt 44 c19b0e20 d543a000 0 0 0 0000204 [SLPQ vlruwt 0xc19b0e20][SLP] vnlru 43 c1a47000 d543b000 0 0 0 0000204 [SLPQ syncer 0xc08c22ec][SLP] syncer 42 c1a471c4 d545a000 0 0 0 0000204 [SLPQ psleep 0xc08c828c][SLP] bufdaemon 41 c1a47388 d545b000 0 0 0 000020c [RUNQ] pagezero 40 c1a4754c d545c000 0 0 0 0000204 [SLPQ psleep 0xc08d0024][SLP] vmdaemon 39 c1a47710 d545d000 0 0 0 0000204 [SLPQ psleep 0xc08cffe0][SLP] pagedaemon 38 c1a478d4 d545e000 0 0 0 0000204 [IWAIT] swi0: sio 9 c1a47a98 d545f000 0 0 0 0000204 [SLPQ - 0xc19a8d3c][SLP] fdc0 8 c1a47c5c d5460000 0 0 0 0000204 [SLPQ idle 0xc195a400][SLP] aic_recovery0 7 c1a47e20 d5461000 0 0 0 0000204 [SLPQ idle 0xc195a400][SLP] aic_recovery0 37 c1a4a000 d5462000 0 0 0 0000204 [IWAIT] swi6:+ 36 c19a054c d540e000 0 0 0 0000204 [IWAIT] swi6: task queue 6 c19a0710 d540f000 0 0 0 0000204 [SLPQ - 0xc19ec580][SLP] kqueue taskq 35 c19a08d4 d5410000 0 0 0 0000204 [IWAIT] swi3: cambio 34 c19a0a98 d542f000 0 0 0 0000204 [IWAIT] swi2: camnet 33 c19a0c5c d5430000 0 0 0 0000204 [IWAIT] swi6:+ 5 c19a0e20 d5431000 0 0 0 0000204 [SLPQ - 0xc19ec7c0][SLP] thread taskq 32 c19b0000 d5432000 0 0 0 0000204 [SLPQ - 0xc08b8100][SLP] yarrow 4 c19b01c4 d5433000 0 0 0 0000204 [CPU 0] g_down 3 c19b0388 d5434000 0 0 0 0000204 [SLPQ - 0xc08bcb24][SLP] g_up 2 c19b054c d5435000 0 0 0 0000204 [SLPQ - 0xc08bcb1c][SLP] g_event 31 c19b0710 d5436000 0 0 0 0000204 [CPU 1] swi1: net 30 c19b08d4 d5437000 0 0 0 0000204 [IWAIT] swi4: vm 29 c196d1c4 d421a000 0 0 0 000020c [RUNQ] swi5: clock sio 28 c196d388 d421b000 0 0 0 0000204 [IWAIT] irq0: clk 27 c196d54c d423a000 0 0 0 0000204 [IWAIT] irq15: 26 c196d710 d423b000 0 0 0 0000204 [IWAIT] irq14: atapci0 ata0 25 c196d8d4 d423c000 0 0 0 0000204 [IWAIT] irq13: 24 c196da98 d423d000 0 0 0 0000204 [IWAIT] irq12: 23 c196dc5c d423e000 0 0 0 0000204 [IWAIT] irq11: fxp1 22 c196de20 d423f000 0 0 0 0000204 [IWAIT] irq10: xl0 fxp0 21 c19a0000 d540b000 0 0 0 0000204 [IWAIT] irq9: ahc0 20 c19a01c4 d540c000 0 0 0 0000204 [IWAIT] irq8: rtc 19 c19a0388 d540d000 0 0 0 0000204 [IWAIT] irq7: ep0 18 c1964000 d41d4000 0 0 0 0000204 [IWAIT] irq6: fdc0 17 c19641c4 d4211000 0 0 0 0000204 [IWAIT] irq5: 16 c1964388 d4212000 0 0 0 0000204 [IWAIT] irq4: sio0 15 c196454c d4213000 0 0 0 0000204 [IWAIT] irq3: sio1 14 c1964710 d4214000 0 0 0 0000204 [IWAIT] irq2: 13 c19648d4 d4215000 0 0 0 0000204 [IWAIT] irq1: atkbd0 12 c1964a98 d4216000 0 0 0 000020c [Can run] idle: cpu0 11 c1964c5c d4217000 0 0 0 000020c [Can run] idle: cpu1 1 c1964e20 d4218000 0 0 1 0004200 [SLPQ wait 0xc1964e20][SLP] init 10 c196d000 d4219000 0 0 0 0000204 [SLPQ ktrace 0xc08c0518][SLP] ktrace 0 c08bcc20 c0c1f000 0 0 0 0000200 [SLPQ sched 0xc08bcc20][SLP] swapper db> show pcpu 0 cpuid = 0 curthread = 0xc19a1000: pid 4 "g_down" curpcb = 0xd5413da0 fpcurthread = none idlethread = 0xc1965640: pid 12 "idle: cpu0" APIC ID = 1 currentldt = 0x28 db> show pcpu 1 cpuid = 1 curthread = 0xc196eaf0: pid 31 "swi1: net" curpcb = 0xd4233da0 fpcurthread = none idlethread = 0xc19654b0: pid 11 "idle: cpu1" APIC ID = 0 currentldt = 0x28 db> trace ipllog(0,d4233bd8,d4233b50,d4233b48,d4233b40) at ipllog+0x43 ipflog(1a,c2692820,d4233bd8,c1a72500) at ipflog+0x18f fr_check(c2692820,14,c1a44000,0,d4233c80) at fr_check+0x820 fr_check_wrapper(0,d4233c80,c1a44000,1,0) at fr_check_wrapper+0x2a pfil_run_hooks(c08c99c0,d4233ccc,c1a44000,1,0) at pfil_run_hooks+0xe3 ip_input(c1a72500) at ip_input+0x211 netisr_processqueue(c08c8c58) at netisr_processqueue+0xa3 swi_net(0) at swi_net+0xf2 ithread_loop(c1962680,d4233d48) at ithread_loop+0x159 fork_exit(c060bb5c,c1962680,d4233d48) at fork_exit+0x75 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd4233d7c, ebp = 0 --- db> call doadump Dumping 512 MB 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 Dump complete 0xf db> reset cpu_reset called on cpu#1 cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 oak# nm -n /boot/kernel/kernel | grep c04647 c0464714 T ipllog oak# oak# mptable -dmesg -verbose -grope =============================================================================== MPTable, version 2.0.15 looking for EBDA pointer @ 0x040e, found, searching EBDA @ 0x0009f800 searching CMOS 'top of mem' @ 0x0009f400 (637K) searching default 'top of mem' @ 0x0009fc00 (639K) searching BIOS @ 0x000f0000 MP FPS found in BIOS @ physical addr: 0x000f99a0 ------------------------------------------------------------------------------- MP Floating Pointer Structure: location: BIOS physical address: 0x000f99a0 signature: '_MP_' length: 16 bytes version: 1.1 checksum: 0x4b mode: Virtual Wire ------------------------------------------------------------------------------- MP Config Table Header: physical address: 0x000f99b0 signature: 'PCMP' base table length: 236 version: 1.1 checksum: 0xe0 OEM ID: 'HP ' Product ID: 'XX ' OEM table pointer: 0x00000000 OEM table size: 0 entry count: 21 local APIC address: 0xfee00000 extended table length: 0 extended table checksum: 0 ------------------------------------------------------------------------------- MP Config Base Table Entries: -- Processors: APIC ID Version State Family Model Step Flags 1 0x11 BSP, usable 6 1 0 0xfbff 0 0x11 AP, usable 6 1 0 0xfbff -- Bus: Bus ID Type 1 ISA 0 PCI -- I/O APICs: APIC ID Version State Address 16 0x11 usable 0xfec00000 -- I/O Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# INT conforms conforms 1 1 255 1 INT conforms conforms 1 0 255 2 INT conforms conforms 1 3 255 3 INT conforms conforms 1 4 255 4 INT conforms conforms 1 5 255 5 INT conforms conforms 1 6 255 6 INT conforms conforms 1 7 255 7 INT conforms conforms 1 8 255 8 INT conforms conforms 1 9 255 9 INT conforms conforms 1 10 255 10 INT conforms conforms 1 11 255 11 INT conforms conforms 1 12 255 12 INT conforms conforms 1 13 255 13 INT conforms conforms 1 14 255 14 INT conforms conforms 1 15 255 15 -- Local Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# NMI conforms conforms 1 2 255 1 ------------------------------------------------------------------------------- dmesg output: Copyright (c) 1992-2004 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.3-RELEASE-p2 #2: Wed Dec 8 22:06:28 MST 2004 root@oak.zyni.com:/usr/src/sys/i386/compile/OAK-TEST Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Pentium Pro (199.74-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x619 Stepping = 9 Features=0xfbff real memory = 536870912 (512 MB) avail memory = 515813376 (491 MB) MPTable: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 1 cpu1 (AP): APIC ID: 0 ioapic0: Changing APIC ID to 16 ioapic0: Assuming intbase of 0 MPTable: Ignoring global interrupt entry for pin 1 MPTable: Ignoring global interrupt entry for pin 2 MPTable: Ignoring global interrupt entry for pin 3 MPTable: Ignoring global interrupt entry for pin 4 MPTable: Ignoring global interrupt entry for pin 5 MPTable: Ignoring global interrupt entry for pin 6 MPTable: Ignoring global interrupt entry for pin 7 MPTable: Ignoring global interrupt entry for pin 8 MPTable: Ignoring global interrupt entry for pin 9 MPTable: Ignoring global interrupt entry for pin 10 MPTable: Ignoring global interrupt entry for pin 11 MPTable: Ignoring global interrupt entry for pin 12 MPTable: Ignoring global interrupt entry for pin 13 MPTable: Ignoring global interrupt entry for pin 14 MPTable: Ignoring global interrupt entry for pin 15 ioapic0 irqs 0-15 on motherboard npx0: [FAST] npx0: on motherboard npx0: INT 16 interface pcib0: pcibus 0 on motherboard pci0: on pcib0 isab0: at device 0.0 on pci0 isa0: on isab0 atapci0: port 0xfc00-0xfc0f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 irq 14 at device 1.0 on pci0 atapci0: HW has secondary channel disabled ata0: channel #0 on atapci0 ahc0: port 0xf800-0xf8ff mem 0xffbfc000-0xffbfcfff irq 9 at device 2.0 on pci0 ahc0: Using left over BIOS settings ahc0: [GIANT-LOCKED] aic7880: Ultra Single Channel A, SCSI Id=7, 16/253 SCBs xl0: <3Com 3c905B-TX Fast Etherlink XL> port 0xf400-0xf47f mem 0xffbec000-0xffbec07f irq 10 at device 11.0 on pci0 miibus0: on xl0 xlphy0: <3Com internal media interface> on miibus0 xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl0: Ethernet address: 00:10:5a:0d:79:54 fxp0: port 0xf000-0xf01f mem 0xffa00000-0xffafffff,0xffbdc000-0xffbdcfff irq 10 at device 12.0 on pci0 miibus1: on fxp0 nsphy0: on miibus1 nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:a0:c9:57:e8:69 fxp1: port 0xec00-0xec1f mem 0xff900000-0xff9fffff,0xffbcc000-0xffbccfff irq 11 at device 13.0 on pci0 miibus2: on fxp1 nsphy1: on miibus2 nsphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: Ethernet address: 00:a0:c9:3e:19:1d pci0: at device 14.0 (no driver attached) pci0: at device 20.0 (no driver attached) cpu0 on motherboard cpu1 on motherboard orm0: at iomem 0xc8000-0xcc7ff,0xc0000-0xc7fff on isa0 pmtimer0 on isa0 ep0: <3Com 3C509-TP EtherLink III> at port 0x290-0x29f irq 7 on isa0 ep0: Ethernet address: 00:60:8c:b9:5e:18 atkbdc0: at port 0x64,0x60 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] fdc0: at port 0x3f0-0x3f5 irq 6 drq 2 on isa0 fdc0: [FAST] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 ppc0: parallel port not found. sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A, console sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 unknown: can't assign resources (port) unknown: can't assign resources (port) unknown: can't assign resources (port) ppc1: parallel port not found. unknown: can't assign resources (port) Timecounters tick every 10.000 msec IP Filter: v3.4.35 initialized. Default = pass all, Logging = enabled Waiting 15 seconds for SCSI devices to settle da0 at ahc0 bus 0 target 1 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 20.000MB/s transfers (20.000MHz, offset 15), Tagged Queueing Enabled da0: 17522MB (35885168 512 byte sectors: 64H 32S/T 17522C) SMP: AP CPU #1 Launched! Mounting root from ufs:/dev/da0s1a WARNING: / was not properly dismounted ep0: promiscuous mode enabled =============================================================================== oak# diff GENERIC OAK-TEST 25c25 < ident GENERIC --- > ident OAK-TEST 29a30,35 > makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols > options IPFILTER #ipfilter support > options IPFILTER_LOG #ipfilter logging > options SMP > options DDB > options KDB 38,40c44,46 < options NFSCLIENT # Network Filesystem Client < options NFSSERVER # Network Filesystem Server < options NFS_ROOT # NFS usable as /, requires NFSCLIENT --- > #options NFSCLIENT # Network Filesystem Client > #options NFSSERVER # Network Filesystem Server > #options NFS_ROOT # NFS usable as /, requires NFSCLIENT oak# I also have the core file from the crash if any additional information can be obtained from that please let me know. >How-To-Repeat: Running these commands simultaneously will get a crash out within 45 minutes w/o debugging kernel on my systems. With the kdb/ddb stuff turned on it seems to be more on the order of hours rather than minutes. Also when the tcpdump starts and stops there is around 10 seconds or so where network traffic stops moving. find / -type f -exec strings {} \; > /dev/null dump 0sf 300000 - /home | gzip -9 > dumphome.gz bzip2 .iso scp .iso :/var/tmp dump 0sf 300000 - / | gzip -9 > dumproot.gz while 1 tcpdump -i fxp0 & sleep 360 kill -9 `ps -auxww | grep tcpdump | grep -v grep | awk '{print $2}'` sleep 60 end >Fix: >Release-Note: >Audit-Trail: >Unformatted: