From owner-freebsd-stable Mon May 13 13:56:19 2002 Delivered-To: freebsd-stable@freebsd.org Received: from host217-41-47-151.in-addr.btopenworld.com (host217-41-47-151.in-addr.btopenworld.com [217.41.47.151]) by hub.freebsd.org (Postfix) with ESMTP id B447137B408; Mon, 13 May 2002 13:55:44 -0700 (PDT) Received: by host217-41-47-151.in-addr.btopenworld.com (Postfix, from userid 1001) id 41B19615; Mon, 13 May 2002 21:55:40 +0100 (BST) Date: Mon, 13 May 2002 21:55:40 +0100 From: Dominic Marks To: freebsd-hackers@freebsd.org Cc: freebsd-stable@freebsd.org Subject: backtrace of a 4.6-PRE panic Message-ID: <20020513215540.A345@host217-41-47-151.in-addr.btope> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hey, I've been having panics with RELENG_4 for a few months now at 3 to 4 day intervals. I haven't been able to get a meaning debugging data from the machine until today. Any clues on what might be the clue of this would be very gratefully received! If it helps the machine wasn't doing very much at the time, just NAT for a few hosts on a DSL line with a few ssh logins running various low-strain applications. # uname -smr FreeBSD 4.6-PRERELEASE i386 # gdb -k /usr/crash/kernel.1 /usr/crash/vmcore.1 IdlePTD at phsyical address 0x002f3000 initial pcb at physical address 0x0021d4e0 panicstr: page fault panic messages: --- Fatal trap 12: page fault while in kernel mode fault virtual address = 0xc0aac37f fault code = supervisor write, page not present instruction pointer = 0x8:0xc015e763 stack pointer = 0x10:0xca66de68 frame pointer = 0x10:0xca66de8c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 284 (pptp) interrupt mask = net trap number = 12 panic: page fault syncing disks... 24 Fatal trap 12: page fault while in kernel mode fault virtual address = 0xc0aac37f fault code = supervisor write, page not present instruction pointer = 0x8:0xc027a573 stack pointer = 0x10:0xca66db54 frame pointer = 0x10:0xca66db60 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 284 (pptp) interrupt mask = net trap number = 12 panic: page fault Uptime: 3d21h11m12s dumping to dev #ad/0x20001, offset 221184 dump ata0: resetting devices .. done 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 --- #0 0xc013ff7e in dumpsys () (kgdb) bt #0 0xc013ff7e in dumpsys () #1 0xc013fd48 in boot () #2 0xc014017c in poweroff_wait () #3 0xc01da9af in trap_fatal () #4 0xc01da669 in trap_pfault () #5 0xc01da23f in trap () #6 0xc027a573 in ?? () #7 0xc027a736 in ?? () #8 0xc027ae68 in ?? () #9 0xc01dd81b in intr_mux () #10 0xc01d0016 in vec11 () #11 0xc01992ca in interlocked_sleep () #12 0xc019ec92 in drain_output () #13 0xc019da56 in softdep_fsync_mountdev () #14 0xc01a1c1a in ffs_fsync () #15 0xc01a089c in ffs_sync () #16 0xc016fd33 in sync () #17 0xc013fbd3 in boot () #18 0xc014017c in poweroff_wait () #19 0xc01da9af in trap_fatal () #20 0xc01da669 in trap_pfault () #21 0xc01da23f in trap () #22 0xc015e763 in sosend () #23 0xc015220a in soo_write () #24 0xc014ef84 in dofilewrite () #25 0xc014ee73 in write () #26 0xc01dabb2 in syscall2 () #27 0xc01ceb15 in Xint0x80_syscall () #28 0x804aa57 in ?? () #29 0x804a883 in ?? () #30 0x80494dc in ?? () #31 0x8049109 in ?? () # netstat -m -M /usr/crash/vmcore.1 206/944/4096 mbufs in use (current/peak/max): 206 mbufs allocated to data 49/290/1024 mbuf clusters in use (current/peak/max) 816 Kbytes allocated to network (26% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines # vmstat -i -M /usr/crash/vmcore.1 interrupt total rate ata0 irq14 4212446 12 ata1 irq15 24 0 uhci0 irq9 278310 0 mux irq11 25139819 74 atkbd0 irq1 185681 0 clk irq0 33546748 99 Total 63363028 187 # vmstat -m -M /usr/crash/vmcore.1 Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 438 586 967543 1280 0 32 565 11339 7687041 640 232264 64 32281 58663 34070085 320 98727 128 8200 14936 5394095 160 270089 256 35898 2534 5138130 80 2167 512 351 97 384108 40 645 1K 16 1460 595080 20 439367 2K 40 180 10888 10 8830 4K 34 3 4275266 5 0 8K 5 35 3643 5 3557 16K 18 0 248 5 0 32K 5 0 13 5 0 64K 1 0 1 5 0 128K 5 0 6267 5 0 512K 1 0 1 5 0 Memory usage type by bucket size Size Type(s) 16 NFSV3 srvdesc, USB, feeder, ac97, UFS dirhash, routetbl, ether_multi, BPF, vnodes, mount, pcb, soname, shm, rman, bus, sysctloid, sysctl, ATA generic, temp, devbuf, proc-args, kld 32 linux, USBHC, USB, atkbddev, memdesc, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag, indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl, ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname, ATAPI generic, taskqueue, SWAP, eventhandler, bus, sysctl, uidinfo, subproc, pgrp, temp, devbuf, proc-args, sigio, kld 64 NFS daemon, USBHC, USBdev, USB, feeder, isadev, UFS dirhash, allocindir, allocdirect, pagedep, routetbl, ether_multi, ifaddr, vnodes, cluster_save buffer, vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc, session, temp, devbuf, lockf, proc-args, file, AR driver, AD driver 128 USBdev, USB, ZONE, UFS dirhash, freeblks, inodedep, routetbl, BPF, vnodes, mount, cluster_save buffer, vfscache, soname, ATAPI generic, ttys, iov, bus, uidinfo, cred, temp, devbuf, zombie, proc-args, dev_t, timecounter, kld 256 NFSV3 srvdesc, NFS daemon, NFS srvsock, tun, ac97, UFS dirhash, FFS node, newblk, routetbl, ifaddr, Export Host, vnodes, vfscache, ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue, file desc 512 NFS daemon, USBdev, UFS dirhash, UFS mount, BPF, mount, vfscache, BIO buffer, ptys, msg, ioctlops, bus, proc, ATA generic, temp, devbuf, file desc, AR driver 1K NQNFS Lease, mixer, UFS dirhash, BIO buffer, shm, sem, ioctlops, bus, temp, devbuf, kqueue, file desc, kld, AD driver 2K mbuf, UFS dirhash, UFS mount, ifaddr, BPF, BIO buffer, ACD driver, pcb, ioctlops, bus, devbuf, file desc, kld, AR driver 4K USB, memdesc, UFS mount, sem, msg, ioctlops, kobj, bus, temp, devbuf, kld 8K indirdep, pagedep, syncache, bus, temp, kld 16K UFS mount, indirdep, shm, msg, temp, devbuf, kld 32K NFS hash, VM pgdata, UFS ihash, inodedep, BPF, temp 64K vfscache 128K SWAP, devbuf, kld 512K kld Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) linux 7 1K 1K 31792K 7 0 0 32 NFS hash 1 32K 32K 31792K 1 0 0 32K NQNFS Lease 1 1K 1K 31792K 1 0 0 1K NFSV3 srvdesc 0 0K 1K 31792K 827968 0 0 16,256 NFS daemon 69 7K 7K 31792K 69 0 0 64,256,512 NFS srvsock 1 1K 1K 31792K 1 0 0 256 USBHC 3 1K 1K 31792K 3 0 0 32,64 USBdev 4 2K 2K 31792K 4 0 0 64,128,512 USB 25 22K 22K 31792K 59 0 0 16,32,64,128,4K tun 1 1K 1K 31792K 1 0 0 256 feeder 51 1K 1K 31792K 3543 0 0 16,64 ac97 2 1K 1K 31792K 2 0 0 16,256 mixer 1 1K 1K 31792K 1 0 0 1K atkbddev 2 1K 1K 31792K 2 0 0 32 memdesc 1 4K 8K 31792K 145 0 0 32,4K mbuf 1 2K 2K 31792K 1 0 0 2K isadev 4 1K 1K 31792K 4 0 0 64 ZONE 16 2K 2K 31792K 16 0 0 128 VM pgdata 1 32K 32K 31792K 1 0 0 32K UFS dirhash 825 170K 212K 31792K 10689 0 0 16,32,64,128,256,512,1K,2K UFS mount 21 64K 64K 31792K 57 0 0 512,2K,4K,16K UFS ihash 1 32K 32K 31792K 1 0 0 32K FFS node 35503 8876K 8877K 31792K 2693502 0 0 256 dirrem 0 0K 305K 31792K 345839 0 0 32 mkdir 0 0K 18K 31792K 51408 0 0 32 diradd 0 0K 79K 31792K 402127 0 0 32 freefile 0 0K 276K 31792K 293769 0 0 32 freeblks 0 0K 1153K 31792K 293550 0 0 128 freefrag 0 0K 27K 31792K 270753 0 0 32 allocindir 0 0K 5495K 31792K 3649542 0 0 64 indirdep 0 0K 282K 31792K 11383 0 0 32,8K,16K allocdirect 0 0K 243K 31792K 862516 0 0 64 bmsafemap 0 0K 3K 31792K 11420 0 0 32 newblk 1 1K 1K 31792K 4512059 0 0 32,256 inodedep 1 32K 1234K 31792K 794717 0 0 128,32K pagedep 1 8K 44K 31792K 68509 0 0 64,8K syncache 1 8K 8K 31792K 1 0 0 8K tseg_qent 16 1K 8K 31792K 148203 0 0 32 in_multi 3 1K 1K 31792K 3 0 0 32 routetbl 249 35K 52K 31792K 3239 0 0 16,32,64,128,256 ether_multi 9 1K 1K 31792K 9 0 0 16,32,64 ifaddr 15 4K 4K 31792K 15 0 0 32,64,256,2K BPF 3 1K 67K 31792K 19 0 0 16,32,128,512,2K,32K Export Host 3 1K 1K 31792K 11 0 0 256 vnodes 21 5K 5K 31792K 242 0 0 16,32,64,128,256 mount 7 4K 4K 31792K 26 0 0 16,128,512 cluster_save buffer 0 0K 1K 31792K 64350 0 0 32,64,128 vfscache 37531 2791K 3340K 31792K 3951905 0 0 64,128,256,512,64K BIO buffer 22 43K 1458K 31792K 148291 0 0 512,1K,2K ACD driver 2 3K 3K 31792K 2 0 0 256,2K pcb 31 5K 5K 31792K 23407 0 0 16,32,64,2K soname 20 3K 5K 31792K 516756 0 0 16,32,64,128 ATAPI generic 1 1K 1K 31792K 14 0 0 32,128 ptys 11 6K 6K 31792K 11 0 0 512 ttys 684 87K 112K 31792K 5676 0 0 128,256 shm 1 12K 17K 31792K 502 0 0 16,1K,16K sem 3 6K 6K 31792K 3 0 0 1K,4K msg 4 25K 25K 31792K 4 0 0 512,4K,16K rman 45 3K 3K 31792K 339 0 0 16,64 iov 0 0K 1K 31792K 2821 0 0 128 ioctlops 0 0K 4K 31792K 88 0 0 512,1K,2K,4K taskqueue 1 1K 1K 31792K 1 0 0 32 SWAP 2 105K 105K 31792K 2 0 0 32,128K kobj 5 20K 20K 31792K 5 0 0 4K eventhandler 13 1K 1K 31792K 13 0 0 32,64 bus 335 36K 36K 31792K 604 0 0 16,32,64,128,256,512,1K,2K,4K,8K sysctloid 10 1K 1K 31792K 10 0 0 16,64 sysctl 0 0K 1K 31792K 8415 0 0 16,32 uidinfo 7 1K 1K 31792K 219 0 0 32,128 cred 75 10K 13K 31792K 2175785 0 0 128 subproc 133 11K 19K 31792K 3142783 0 0 32,64,256 proc 2 1K 1K 31792K 2 0 0 512 session 35 3K 3K 31792K 1608 0 0 64 pgrp 37 2K 2K 31792K 3067 0 0 32 ATA generic 8 3K 3K 31792K 9 0 0 16,512 temp 589 131K 746K 31792K 2622731 0 0 16,32,64,128,256,512,1K,4K,8K,16K,32K devbuf 114 472K 480K 31792K 4222367 0 0 16,32,64,128,256,512,1K,2K,4K,16K,128K lockf 9 1K 2K 31792K 199227 0 0 64 zombie 0 0K 5K 31792K 1498405 0 0 128 proc-args 46 3K 6K 31792K 834785 0 0 16,32,64,128,256 kqueue 7 7K 17K 31792K 13633 0 0 256,1K sigio 4 1K 1K 31792K 451 0 0 32 file 229 15K 30K 31792K 18122993 0 0 64 file desc 59 16K 51K 31792K 1502159 0 0 256,512,1K,2K dev_t 861 108K 108K 31792K 861 0 0 128 timecounter 5 1K 1K 31792K 5 0 0 128 kld 43 559K 565K 31792K 240 0 0 16,32,128,1K,2K,4K,8K,16K,128K,512K AR driver 1 1K 3K 31792K 5 0 0 64,512,2K AD driver 2 2K 3K 31792K 4212417 0 0 64,1K Memory Totals: In Use Free Requests 13822K 8691K 58532409 # cat /var/run/dmesg.boot Copyright (c) 1992-2002 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.6-PRERELEASE #2: Thu May 2 19:01:45 BST 2002 dom@host217-39-140-82.in-addr.btopenworld.com:/usr/obj/usr/src/sys/GALLIUM Timecounter "i8254" frequency 1193182 Hz CPU: Pentium II/Pentium II Xeon/Celeron (447.69-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x652 Stepping = 2 Features=0x183f9ff real memory = 201326592 (196608K bytes) avail memory = 193048576 (188524K bytes) Preloaded elf kernel "kernel" at 0xc02d4000. Preloaded elf module "if_tun.ko" at 0xc02d409c. Preloaded elf module "if_xl.ko" at 0xc02d413c. Preloaded elf module "miibus.ko" at 0xc02d41dc. Preloaded elf module "snd_es137x.ko" at 0xc02d427c. Preloaded elf module "snd_pcm.ko" at 0xc02d4320. Preloaded elf module "usb.ko" at 0xc02d43c0. Preloaded elf module "ums.ko" at 0xc02d445c. Pentium Pro MTRR support enabled Using $PIR table, 11 entries at 0xc00fdf10 apm0: on motherboard apm: found APM BIOS v1.2, connected at v1.2 npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 pci1: at 0.0 irq 11 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x10e0-0x10ef at device 7.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata1: at 0x170 irq 15 on atapci0 uhci0: port 0x10c0-0x10df irq 9 at device 7.2 on pci0 usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered ums0: Microsoft Microsoft Wheel Mouse Optical\M-., rev 1.10/1.21, addr 2, iclass 3/1 ums0: 3 buttons and Z dir. chip1: port 0x7000-0x700f at device 7.3 on pci0 pcm0: port 0x1080-0x10bf irq 11 at device 12.0 on pci0 xl0: <3Com 3c905B-TX Fast Etherlink XL> port 0x1000-0x107f mem 0xf4000000-0xf400007f irq 11 at device 14.0 on pci0 xl0: Ethernet address: 00:01:02:45:63:95 miibus0: on xl0 xlphy0: <3Com internal media interface> on miibus0 xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto orm0: