From owner-freebsd-stable@FreeBSD.ORG Tue Aug 24 14:17:19 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1BA161065694 for ; Tue, 24 Aug 2010 14:17:19 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2-6.sentex.ca [IPv6:2607:f3e0:80:80::2]) by mx1.freebsd.org (Postfix) with ESMTP id D1E378FC19 for ; Tue, 24 Aug 2010 14:17:17 +0000 (UTC) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost2.sentex.ca (8.14.4/8.14.4) with ESMTP id o7OEHCx5051927 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 24 Aug 2010 10:17:12 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.14.4/8.14.3) with ESMTP id o7OEHCwa042809 for ; Tue, 24 Aug 2010 10:17:12 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <201008241417.o7OEHCwa042809@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 24 Aug 2010 10:17:16 -0400 To: freebsd-stable@freebsd.org From: Mike Tancsa In-Reply-To: <201008202104.o7KL4xi8012899@lava.sentex.ca> References: <201008202104.o7KL4xi8012899@lava.sentex.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.67 on 205.211.164.50 Subject: Re: RELENG_8 panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Aug 2010 14:17:19 -0000 At 05:04 PM 8/20/2010, Mike Tancsa wrote: >The box is a moderately busy LNS running mpd5. I have another box >running the same load that has not crashed so I am wondering if its >hardware or this box is just "lucky" ? its crashed a couple of times >now, but the watchdog rebooted it prior to the dump being written out. I had disabled the watchdog and was able to get a full crash dump this time. Seems to be about 3 days between crashes Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x24 fault code = supervisor read, page not present instruction pointer = 0x20:0xc5d11e15 stack pointer = 0x28:0xc4e70704 frame pointer = 0x28:0xc4e70718 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (em1 taskq) trap number = 12 panic: page fault cpuid = 0 Uptime: 3d20h45m41s Physical memory: 2036 MB Dumping 228 MB: 212 196 180 164 148 132 116 100 84 68 52 36 20 4 Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /boot/kernel/coretemp.ko.symbols...done. done. Loaded symbols for /boot/kernel/coretemp.ko Reading symbols from /boot/kernel/if_disc.ko...Reading symbols from /boot/kernel/if_disc.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_disc.ko Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done. done. Loaded symbols for /boot/kernel/ipfw.ko Reading symbols from /boot/kernel/libalias.ko...Reading symbols from /boot/kernel/libalias.ko.symbols...done. done. Loaded symbols for /boot/kernel/libalias.ko Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_socket.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_mppc.ko Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done. done. Loaded symbols for /boot/kernel/rc4.ko Reading symbols from /boot/kernel/ichwd.ko...Reading symbols from /boot/kernel/ichwd.ko.symbols...done. done. Loaded symbols for /boot/kernel/ichwd.ko Reading symbols from /boot/kernel/ng_l2tp.ko...Reading symbols from /boot/kernel/ng_l2tp.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_l2tp.ko Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ksocket.ko Reading symbols from /boot/kernel/ng_tee.ko...Reading symbols from /boot/kernel/ng_tee.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_tee.ko Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from /boot/kernel/ng_iface.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_iface.ko Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ppp.ko Reading symbols from /boot/kernel/ng_tcpmss.ko...Reading symbols from /boot/kernel/ng_tcpmss.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_tcpmss.ko #0 doadump () at pcpu.h:231 231 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:231 #1 0xc06b0ac3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0xc06b0d29 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:590 #3 0xc092239c in trap_fatal (frame=0xc4e706c4, eva=36) at /usr/src/sys/i386/i386/trap.c:938 #4 0xc0922620 in trap_pfault (frame=0xc4e706c4, usermode=0, eva=36) at /usr/src/sys/i386/i386/trap.c:851 #5 0xc0922f0c in trap (frame=0xc4e706c4) at /usr/src/sys/i386/i386/trap.c:533 #6 0xc0904a9c in calltrap () at /usr/src/sys/i386/i386/exception.s:166 #7 0xc5d11e15 in ng_address_hook (here=0x0, item=0xc5da0540, hook=0xc65ce400, retaddr=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3504 #8 0xc5dddbcb in ng_tee_rcvdata (hook=0xca997580, item=0xc5da0540) at /usr/src/sys/modules/netgraph/tee/../../../netgraph/ng_tee.c:326 #9 0xc5d137c4 in ng_apply_item (node=0xc6499b80, item=0xc5da0540, rw=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336 #10 0xc5d1279f in ng_snd_item (item=0xc5da0540, flags=Variable "flags" is not available. ) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253 #11 0xc5eca84a in ng_ppp_link_xmit (node=Variable "node" is not available. ) at /usr/src/sys/modules/netgraph/ppp/../../../netgraph/ng_ppp.c:1350 #12 0xc5ecb605 in ng_ppp_mp_xmit (node=0xcaad1a00, item=0xc5da0540, proto=Variable "proto" is not available. ) at /usr/src/sys/modules/netgraph/ppp/../../../netgraph/ng_ppp.c:1998 #13 0xc5d137c4 in ng_apply_item (node=0xcaad1a00, item=0xc5da0540, rw=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336 #14 0xc5d1279f in ng_snd_item (item=0xc5da0540, flags=Variable "flags" is not available. ) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253 #15 0xc5d137c4 in ng_apply_item (node=0xc5eeae80, item=0xc5da0540, rw=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336 #16 0xc5d1279f in ng_snd_item (item=0xc5da0540, flags=Variable "flags" is not available. ) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253 #17 0xc5ec57e1 in ng_iface_send (ifp=0xc527d400, m=0xcb478200, sa=Variable "sa" is not available. ) at /usr/src/sys/modules/netgraph/iface/../../../netgraph/ng_iface.c:475 #18 0xc5ec5c08 in ng_iface_output (ifp=0xc527d400, m=0xcb478200, dst=0xc4e70bdc, ro=0xc4e70bd4) at /usr/src/sys/modules/netgraph/iface/../../../netgraph/ng_iface.c:410 #19 0xc077cde2 in ip_fastforward (m=0xcb478200) at /usr/src/sys/netinet/ip_fastfwd.c:552 #20 0xc0759015 in ether_demux (ifp=0xc523c800, m=0xcb478200) at /usr/src/sys/net/if_ethersubr.c:837 #21 0xc0759593 in ether_input (ifp=0xc523c800, m=0xcb478200) at /usr/src/sys/net/if_ethersubr.c:760 #22 0xc0536d2a in lem_handle_rxtx (context=0xc5277000, pending=1) at /usr/src/sys/dev/e1000/if_lem.c:3626 #23 0xc06ea2c2 in taskqueue_run (queue=0xc5266280) at /usr/src/sys/kern/subr_taskqueue.c:239 #24 0xc06ea4cd in taskqueue_thread_loop (arg=0xc527b5e8) at /usr/src/sys/kern/subr_taskqueue.c:360 #25 0xc0686561 in fork_exit (callout=0xc06ea410 , arg=0xc527b5e8, frame=0xc4e70d38) at /usr/src/sys/kern/kern_fork.c:844 #26 0xc0904b14 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:273 (kgdb) ------------------------------------------------------------------------ ps -axl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 -68 0 0 0 - DLs ?? 13090934:27.00 [kernel] 0 1 0 0 44 0 2912 0 wait DLs ?? 5028681:45.00 [init] 0 2 0 0 -8 0 0 0 - RL ?? -14489947:-26.55 [g_event] 0 3 0 0 -8 0 0 0 - DL ?? -30170063:-16.55 [g_up] 0 4 0 0 -8 0 0 0 - DL ?? 20877214:22.00 [g_down] 0 5 0 0 -16 0 0 0 - DL ?? 19420261:44.00 [fdc0] 0 6 0 0 -60 0 0 0 waitin DL ?? 1587:45.00 [sctp_itera 0 7 0 0 -60 0 0 0 ccb_sc DL ?? 0:00.00 [xpt_thrd] 0 8 0 0 -16 0 0 0 psleep DL ?? 7421017:48.00 [pagedaemon 0 9 0 0 -16 0 0 0 psleep DL ?? 51:45.00 [vmdaemon] 0 10 0 0 171 0 0 0 - RL ?? 25156623:14.00 [idle] 0 11 0 0 -48 0 0 0 - WL ?? -29218365:-21.55 [intr] 0 12 0 0 44 0 0 0 - RL ?? 24812477:16.00 [yarrow] 0 13 0 0 76 0 0 0 pgzero DL ?? 103176:00.00 [pagezero] 0 14 0 0 44 0 0 0 psleep DL ?? 31116207:18.00 [bufdaemon] 0 15 0 0 -16 0 0 0 vlruwt DL ?? 31653040:03.00 [vnlru] 0 16 0 0 44 0 0 0 - RL ?? 7034379:11.00 [syncer] 0 17 0 0 44 0 0 0 - RL ?? -9165157:-23.55 [softdepflu 0 113 1 0 76 0 1540 0 pause Ds ?? 8264:42.00 [adjkerntz] 101 574 1 0 44 0 2948 0 select Ds ?? -22918381:-22.55 [zebra] 101 580 1 0 44 0 3456 0 select Ds ?? 7761146:41.00 [ospfd] 0 955 1 0 1 0 3348 0 - Rs ?? 3529521:14.00 [syslogd] 0 1153 0 0 45 0 0 0 - RL ?? 27744613:25.00 [ng_queue] 0 1160 1 0 44 0 3352 0 select Ds ?? 269548:12.00 [ntpd] 123 1161 1160 0 44 0 3352 0 select D ?? 8588592:09.00 [ntpd] 123 1163 1161 0 76 0 3352 0 select D ?? 21072:54.00 [ntpd] 0 1307 1 0 44 0 6704 0 select Ds ?? 17624560:39.00 [sshd] 0 1315 1 0 44 0 6080 0 - Rs ?? 34559384:02.00 [sendmail] 25 1321 1 0 44 0 6080 0 pause Ds ?? 1682274:36.00 [sendmail] 0 1328 1 0 44 0 3376 0 nanslp Ds ?? 17758332:54.00 [cron] 0 1418 1 0 76 0 3348 0 ttyin Ds+ ?? 55567:03.00 [getty] 0 1419 1 0 76 0 3348 0 ttyin Ds+ ?? 43887:36.00 [getty] 0 1420 1 0 76 0 3348 0 ttyin Ds+ ?? 50276:15.00 [getty] 0 1421 1 0 76 0 3348 0 ttyin Ds+ ?? 45877:21.00 [getty] 0 1422 1 0 76 0 3348 0 ttyin Ds+ ?? 46096:30.00 [getty] 0 1423 1 0 76 0 3348 0 ttyin Ds+ ?? 45587:15.00 [getty] 0 1424 1 0 76 0 3348 0 ttyin Ds+ ?? 44476:30.00 [getty] 0 1425 1 0 76 0 3348 0 ttyin Ds+ ?? 50589:36.00 [getty] 0 1426 1 0 44 0 3812 0 wait Ds ?? 395870:24.00 [login] 0 4087 1426 0 44 0 5652 0 pause D ?? 723658:48.00 [csh] 0 4090 4087 0 44 0 3288 0 kqread D+ ?? 64711:30.00 [tail] 0 4333 1 0 44 0 28440 0 select Ds ?? -32284891:-49.55 [mpd5] 0 12120 1307 0 48 0 11372 0 sbwait Ds ?? 796169:42.00 [sshd] 1001 12122 12120 0 44 0 11372 0 select D ?? 1822270:12.00 [sshd] 1001 12123 12122 0 44 0 5652 0 pause Ds ?? 376237:48.00 [csh] 0 12134 12123 0 44 0 3808 0 wait D ?? 166535:15.00 [su] 0 12135 12134 0 44 0 5652 0 ttyin D+ ?? 1204515:36.00 [csh] 0 12139 1 0 45 0 34744 0 - Rs ?? -20815152:-14.55 [argus] 0 17075 1307 0 47 0 11372 0 sbwait Ds ?? 795308:24.00 [sshd] 1001 17077 17075 0 44 0 11372 0 select D ?? 369119:51.00 [sshd] 1001 17078 17077 0 44 0 5652 0 ttyin Ds+ ?? 769947:27.00 [csh] ------------------------------------------------------------------------ vmstat -s 3478376649 cpu context switches 2032445813 device interrupts 48096996 software interrupts 15353777 traps 1467200553 system calls 18 kernel threads created 14876 fork() calls 2517 vfork() calls 0 rfork() calls 0 swap pager pageins 0 swap pager pages paged in 0 swap pager pageouts 0 swap pager pages paged out 1040 vnode pager pageins 8157 vnode pager pages paged in 0 vnode pager pageouts 0 vnode pager pages paged out 0 page daemon wakeups 0 pages examined by the page daemon 237 pages reactivated 580900 copy-on-write faults 354 copy-on-write optimized faults 1024989 zero fill pages zeroed 34096 zero fill pages prezeroed 192 intransit blocking page faults 2153200 total VM faults taken 0 pages affected by kernel thread creation 2820899 pages affected by fork() 489397 pages affected by vfork() 0 pages affected by rfork() 335 pages cached 2003757 pages freed 0 pages freed by daemon 1367292 pages freed by exiting processes 10919 pages active 159341 pages inactive 65 pages in VM cache 45709 pages wired down 295745 pages free 4096 bytes per page 4892890 total name lookups cache hits (59% pos + 7% neg) system 2% per-directory deletions 0%, falsehits 0%, toolong 0% ------------------------------------------------------------------------ vmstat -m Type InUse MemUse HighUse Requests Size(s) filedesc 57 17K - 17428 16,32,256,512,1024,2048 kenv 114 8K - 117 16,32,64,128,4096 kqueue 2 2K - 10932 128,1024 CAM SIM 1 1K - 1 128 proc-args 33 2K - 87399 16,32,64,128,256 ithread 89 9K - 89 16,64,128 UART 3 2K - 3 16,256,1024 KTRACE 100 13K - 100 128 acpidev 59 2K - 59 32 linker 188 446K - 375 16,32,256,1024,2048,4096 lockf 23 2K - 857 32,64 ata_generic 1 1K - 9 16,512,1024 ip6ndp 795 75K - 47295 64,128 ip6opt 0 0K - 2 128 temp 821 247K - 20925999 16,32,64,128,256,512,1024,2048,4096 devbuf 364 1483K - 401 16,32,64,128,256,512,1024,2048,4096 module 320 21K - 320 64,128 mtx_pool 2 8K - 2 4096 ad_driver 1 1K - 1 32 ar_driver 0 0K - 6 512,2048 subproc 154 230K - 17612 256,4096 proc 2 8K - 2 4096 session 25 2K - 3169 64 pgrp 29 2K - 3386 64 cred 70 7K - 1187398 64,128 uidinfo 6 2K - 2880 64,1024 plimit 18 5K - 25406 256 sysctltmp 0 0K - 49868 16,32,64,128,4096 sysctloid 3021 94K - 3093 16,32,64,128 sysctl 0 0K - 20643 16,32,64 callout 1 256K - 1 umtx 204 13K - 204 64 p1003.1b 1 1K - 1 16 SWAP 2 413K - 2 64 bus-sc 56 90K - 1829 16,32,64,128,256,512,1024,2048,4096 bus 886 43K - 35234 16,32,64,128,1024 devstat 6 13K - 6 16,4096 eventhandler 81 4K - 81 32,64,128 kobj 214 428K - 250 2048 Per-cpu 1 1K - 1 16 rman 157 10K - 563 32,64 sbuf 0 0K - 940 16,32,64,128,256,512,1024,2048,4096 CAM periph 2 1K - 12 16,32,64,128 stack 0 0K - 2 128 taskqueue 19 1K - 19 16,64 Unitno 11 1K - 125295 16,64 iov 0 0K - 4457191 16,64,128,256 select 146 10K - 146 64 ioctlops 0 0K - 484247 16,32,64,128,256,512,1024,2048 msg 4 25K - 4 1024,4096 sem 4 6K - 4 256,512,1024,4096 shm 1 12K - 1 tty 22 11K - 34 512,2048 pts 2 1K - 12 128 mbuf_tag 1 1K - 998739350 32,64 ksem 1 4K - 1 4096 shmfd 1 4K - 1 4096 pci_link 16 2K - 16 32,128 pcb 186 84K - 161996 16,32,64,512,1024,2048,4096 soname 8 1K - 34723809 16,32,128 acl 0 0K - 32 4096 biobuf 1 2K - 45 2048 vfscache 1 512K - 1 cl_savebuf 0 0K - 3937 32,64 vfs_hash 1 256K - 1 vnodes 2 1K - 2 128 vnodemarker 0 0K - 86176 512 mount 69 3K - 132 16,32,64,128,256 BPF 403 90K - 17250 16,64,256 ether_multi 4019 121K - 157478 16,32,64 ifaddr 3233 748K - 126468 16,32,64,128,256,512,2048 ifnet 400 401K - 15820 64,128,256,512,1024,2048 clone 6 24K - 8 16,128,4096 arpcom 5 1K - 5 16 lltable 1202 300K - 47535 128,256 acpi_perf 2 1K - 2 64 acpica 1804 91K - 60858 16,32,64,128,256,512 DEVFS1 73 19K - 91 256 DEVFS3 90 12K - 109 128 routetbl 2761 247K - 117560 16,32,64,128,256,512 igmp 399 50K - 15814 128 DEVFS 10 1K - 11 16,64 DEVFSP 1 1K - 359 32 ip_moptions 2 1K - 2 32,128 in_multi 397 50K - 15741 128 in_mfilter 1 1K - 1 512 sctp_iter 0 0K - 9 256 sctp_ifn 6 1K - 6 128 sctp_ifa 11 2K - 11 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 9 16 hostcache 1 16K - 1 syncache 1 72K - 1 ip6_moptions 2 1K - 2 32,128 in6_multi 3180 422K - 125932 16,256 in6_mfilter 1 1K - 1 512 pfs_nodes 21 3K - 21 128 mld 399 50K - 15814 128 NFS FHA 1 1K - 1 1024 rpc 2 5K - 2 128,4096 savedino 0 0K - 729 256 newdirblk 0 0K - 1 32 dirrem 0 0K - 5260 32 mkdir 0 0K - 90 32 diradd 0 0K - 5442 64 freefile 0 0K - 1094 32 freeblks 0 0K - 1122 256 freefrag 0 0K - 17250 32 allocindir 1 1K - 49037 64 indirdep 2 1K - 16134 32 allocdirect 0 0K - 4060 128 bmsafemap 1 1K - 9184 64 newblk 1 1K - 53098 64,256 inodedep 1 256K - 6340 128 pagedep 1 64K - 789 64 ufs_dirhash 1146 211K - 1146 16,32,64,128,256,512 ufs_mount 12 49K - 12 256,2048,4096 UMAHash 1 2K - 4 256,512,1024,2048 GEOM 63 15K - 447 16,32,64,128,512,1024 vm_pgdata 2 65K - 2 64 acpitask 1 1K - 1 1024 atkbddev 2 1K - 2 32 kbdmux 6 10K - 6 16,256,1024,2048,4096 LED 4 1K - 4 64 apmdev 1 1K - 1 64 CAM XPT 12 2K - 26 16,32,64,1024 acpisem 15 2K - 15 64,128 isadev 21 2K - 21 64 io_apic 2 2K - 2 1024 memdesc 1 4K - 1 4096 msi 1 1K - 1 64 nexusdev 4 1K - 4 16 CAM dev queue 1 1K - 1 128 entropy 1024 64K - 1024 64 cdev 9 2K - 9 128 CAM queue 3 1K - 7 16 sigio 1 1K - 1 32 disc 1 1K - 1 16 IpFw/IpAcct 13 2K - 24 16,32,64,128 netgraph_msg 0 0K - 19647245 64,128,256,512,1024 netgraph_node 1800 225K - 188552 128 netgraph_hook 4248 531K - 401764 128 netgraph 1176 147K - 63692 64,256 netgraph_sock 80 5K - 70376 64 netgraph_path 0 0K - 10195923 16,32 netgraph_mppc 0 0K - 2 1024 netgraph_l2tp 468 101K - 35378 64,1024 netgraph_ksock 76 5K - 19333 64 netgraph_iface 392 25K - 15807 64 netgraph_ppp 392 3136K - 15807 ------------------------------------------------------------------------ vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 128, 0, 87, 3, 87, 0 UMA Zones: 888, 0, 87, 1, 87, 0 UMA Slabs: 284, 0, 1496, 16, 24018, 0 UMA RCntSlabs: 544, 0, 1267, 0, 1267, 0 UMA Hash: 128, 0, 2, 28, 3, 0 16 Bucket: 76, 0, 60, 40, 82, 0 32 Bucket: 140, 0, 92, 20, 114, 0 64 Bucket: 268, 0, 114, 12, 160, 14 128 Bucket: 524, 0, 244, 1, 1077, 112 VM OBJECT: 136, 0, 56609, 318, 418318, 0 MAP: 140, 0, 7, 21, 7, 0 KMAP ENTRY: 72, 57505, 25, 399, 85251, 0 MAP ENTRY: 72, 0, 930, 395, 783736, 0 DP fakepg: 72, 0, 0, 0, 0, 0 SG fakepg: 72, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 249, 238, 249, 0 16: 16, 0, 6475, 630, 45472233, 0 32: 32, 0, 6482, 524, 999601063, 0 64: 64, 0, 7957, 3548, 10862360, 0 128: 128, 0, 9460, 560, 15223676, 0 256: 256, 0, 4810, 200, 376877, 0 512: 512, 0, 1238, 266, 787258, 0 1024: 1024, 0, 497, 87, 8592861, 0 2048: 2048, 0, 247, 85, 11756451, 0 4096: 4096, 0, 105, 68, 73770, 0 Files: 56, 0, 273, 330, 1297543, 0 TURNSTILE: 72, 0, 205, 35, 205, 0 umtx pi: 52, 0, 0, 0, 0, 0 PROC: 680, 0, 50, 52, 17411, 0 THREAD: 636, 0, 128, 76, 47327, 0 SLEEPQUEUE: 44, 0, 205, 90, 205, 0 VMSPACE: 232, 0, 33, 103, 17395, 0 cpuset: 40, 0, 2, 182, 2, 0 mbuf_packet: 256, 0, 2252, 189, 1770562500, 0 mbuf: 256, 0, 30660, 328, 2623840453, 0 mbuf_cluster: 2048, 25600, 2432, 14, 2432, 0 mbuf_jumbo_page: 4096, 12800, 0, 44, 266, 0 mbuf_jumbo_9k: 9216, 19200, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 12800, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 ttyinq: 152, 0, 165, 69, 480, 0 ttyoutq: 256, 0, 88, 32, 256, 0 g_bio: 140, 0, 0, 3808, 1251337, 0 ata_request: 204, 0, 1, 1044, 301616, 0 ata_composite: 180, 0, 0, 0, 0, 0 VNODE: 268, 0, 80955, 105, 1591300, 0 VNODEPOLL: 60, 0, 1, 125, 1, 0 S VFS Cache: 72, 0, 87335, 804, 1613030, 0 L VFS Cache: 292, 0, 226, 632, 12424, 0 NAMEI: 1024, 0, 0, 48, 3452761, 0 NFSMOUNT: 524, 0, 0, 0, 0, 0 NFSNODE: 468, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 1373, 19, 1373, 0 pipe: 392, 0, 6, 44, 7635, 0 ksiginfo: 80, 0, 91, 965, 11791, 0 itimer: 220, 0, 0, 72, 97, 0 KNOTE: 72, 0, 2, 210, 10960, 0 socket: 412, 25605, 280, 89, 540850, 0 unpcb: 172, 25622, 23, 92, 12892, 0 ipq: 32, 904, 0, 0, 0, 0 udp_inpcb: 220, 25614, 84, 96, 417278, 0 udpcb: 8, 25781, 84, 322, 417278, 0 tcp_inpcb: 220, 25614, 11, 61, 1510, 0 tcpcb: 632, 25602, 11, 25, 1510, 0 tcptw: 52, 5184, 0, 216, 59, 0 syncache: 112, 15365, 0, 105, 1241, 0 hostcache: 76, 15400, 1, 149, 21, 0 tcpreass: 20, 1690, 0, 0, 0, 0 sackhole: 20, 0, 0, 169, 9, 0 sctp_ep: 860, 25600, 0, 0, 0, 0 sctp_asoc: 1484, 40000, 0, 0, 0, 0 sctp_laddr: 24, 80040, 0, 290, 10, 0 sctp_raddr: 432, 80001, 0, 0, 0, 0 sctp_chunk: 92, 400008, 0, 0, 0, 0 sctp_readq: 76, 400000, 0, 0, 0, 0 sctp_stream_msg_out: 64, 400020, 0, 0, 0, 0 sctp_asconf: 24, 400055, 0, 0, 0, 0 sctp_asconf_ack: 24, 400055, 0, 0, 0, 0 ripcb: 220, 25614, 2, 52, 32, 0 rtentry: 108, 0, 2194, 254, 79838, 0 selfd: 28, 0, 414, 475, 2819104093, 0 SWAPMETA: 276, 121576, 0, 0, 0, 0 Mountpoints: 644, 0, 5, 13, 5, 0 FFS inode: 116, 0, 80922, 159, 1591194, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 80922, 138, 1591194, 0 IPFW dynamic rule: 108, 0, 0, 0, 0, 0 NetGraph items: 36, 4130, 31, 205, 766085117, 0 NetGraph data items: 36, 531, 1, 412, 1762181728, 0 ------------------------------------------------------------------------ vmstat -i interrupt total rate irq1: atkbd0 6 0 irq4: uart0 2642 14 irq19: atapci1 298104 1665 irq30: em0 1055249819 5895250 irq31: em1 976895242 5457515 cpu0: timer 667882096 3731184 cpu1: timer 667868364 3731108 Total 3368196273 18816738 ------------------------------------------------------------------------ pstat -T 273/12328 files 0M/3071M swap space ------------------------------------------------------------------------ pstat -s Device 512-blocks Used Avail Capacity /dev/ad4s1b 6291200 0 6291200 0% ------------------------------------------------------------------------ iostat iostat: kvm_read(_tk_nin): invalid address (0x0) iostat: disabling TTY statistics iostat: kvm_getcptime: invalid address (0x0) iostat: disabling CPU time statistics ad4 KB/t tps MB/s 13.27 1662 21.53 ------------------------------------------------------------------------ ipcs -a Message Queues: T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME Shared Memory: T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Semaphores: T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME ------------------------------------------------------------------------ ipcs -T msginfo: msgmax: 16384 (max characters in a message) msgmni: 40 (# of message queues) msgmnb: 2048 (max characters in a message queue) msgtql: 40 (max # of messages in system) msgssz: 8 (size of a message segment) msgseg: 2048 (# of message segments in system) shminfo: shmmax: 33554432 (max shared memory segment size) shmmin: 1 (min shared memory segment size) shmmni: 192 (max number of shared memory identifiers) shmseg: 128 (max shared memory segments per process) shmall: 8192 (max amount of shared memory in pages) seminfo: semmap: 30 (# of entries in semaphore map) semmni: 10 (# of semaphore identifiers) semmns: 60 (# of semaphores in system) semmnu: 30 (# of undo structures in system) semmsl: 60 (max # of semaphores per id) semopm: 100 (max # of operations per semop call) semume: 10 (max # of undo entries per process) semusz: 136 (size in bytes of undo structure) semvmx: 32767 (semaphore maximum value) semaem: 16384 (adjust on exit max value) ------------------------------------------------------------------------ netstat -m 32912/517/33429 mbufs in use (current/cache/total) 2243/203/2446/25600 mbuf clusters in use (current/cache/total/max) 2252/189 mbuf+clusters out of packet secondary zone in use (current/cache) 0/44/44/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) 12714K/711K/13425K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines >Fatal trap 12: page fault while in kernel mode >cpuid = 1; apic id = 01 >fault virtual address = 0x1378af >fault code = supervisor read, page not present >instruction pointer = 0x20:0xc5ba1190 >stack pointer = 0x28:0xe79b784c >frame pointer = 0x28:0xe79b7864 >code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 >processor eflags = interrupt enabled, resume, IOPL = 0 >current process = 33060 (mpd5) >trap number = 12 >panic: page fault >cpuid = 1 >Uptime: 2d4h49m17s >Physical memory: 2036 MB >Dumping 228 MB:panic: bufwrite: buffer is not busy??? >cpuid = 1 > 213 197 181 165 149 133 117 101 85 69 53 37 21 5 > > >(kgdb) #0 doadump () at pcpu.h:231 >#1 0xc06b0ac3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 >#2 0xc06b0d29 in panic (fmt=Variable "fmt" is not available. >) at /usr/src/sys/kern/kern_shutdown.c:590 >#3 0xc092239c in trap_fatal (frame=0xe79b780c, eva=1276079) > at /usr/src/sys/i386/i386/trap.c:938 >#4 0xc0922620 in trap_pfault (frame=0xe79b780c, usermode=0, eva=1276079) > at /usr/src/sys/i386/i386/trap.c:851 >#5 0xc0922f0c in trap (frame=0xe79b780c) at /usr/src/sys/i386/i386/trap.c:533 >#6 0xc0904a9c in calltrap () at /usr/src/sys/i386/i386/exception.s:166 >#7 0xc5ba1190 in ng_ID2noderef (ID=64998) > at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:811 >#8 0xc5ba192b in ng_path2noderef (here=0xc5f3f180, > address=0xcac2a800 "[fde6]:", destp=0xe79b7ac0, lasthook=0xe79b7abc) > at > /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:1679 >#9 0xc5ba1d40 in ng_address_path (here=0xc5f3f180, item=0xc5c597c0, > address=0xcac2a800 "[fde6]:", retaddr=0) > at > /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3536 >#10 0xc5b9c662 in ngc_send (so=0xc56e380c, flags=0, m=0xcaee6800, > addr=0xca8baca0, control=0x0, td=0xc6298780) > at /usr/src/sys/modules/netgraph/socket/../../../netgraph/ng_socket.c:296 >#11 0xc07111aa in sosend_generic (so=0xc56e380c, addr=0xca8baca0, > uio=0xe79b7be8, top=0xcaee6800, control=0x0, flags=0, td=0xc6298780) > at /usr/src/sys/kern/uipc_socket.c:1260 >#12 0xc070d40f in sosend (so=0xc56e380c, addr=0xca8baca0, uio=0xe79b7be8, > top=0x0, control=0x0, flags=0, td=0xc6298780) > at /usr/src/sys/kern/uipc_socket.c:1304 >#13 0xc0714a47 in kern_sendit (td=0xc6298780, s=24, mp=0xe79b7c5c, flags=0, > control=0x0, segflg=UIO_USERSPACE) > at /usr/src/sys/kern/uipc_syscalls.c:788 >#14 0xc0714c81 in sendit (td=0xc6298780, s=24, mp=0xe79b7c5c, flags=0) > at /usr/src/sys/kern/uipc_syscalls.c:724 >#15 0xc0714d98 in sendto (td=0xc6298780, uap=0xe79b7cf8) > at /usr/src/sys/kern/uipc_syscalls.c:840 >#16 0xc09228fa in syscall (frame=0xe79b7d38) > at /usr/src/sys/i386/i386/trap.c:1111 >#17 0xc0904b01 in Xint0x80_syscall () > at /usr/src/sys/i386/i386/exception.s:264 >#18 0x00000033 in ?? () >Previous frame inner to this frame (corrupt stack?) >(kgdb) > > > > ---Mike > >_______________________________________________ >freebsd-stable@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-stable >To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike