From owner-freebsd-stable Wed May 3 0:49:55 2000 Delivered-To: freebsd-stable@freebsd.org Received: from account.abs.net (account.abs.net [207.114.5.70]) by hub.freebsd.org (Postfix) with ESMTP id DA86D37B92B; Wed, 3 May 2000 00:49:31 -0700 (PDT) (envelope-from howardl@account.abs.net) Received: (from howardl@localhost) by account.abs.net (8.9.3/8.9.3+RBL+DUL+RSS+ORBS) id DAA84934; Wed, 3 May 2000 03:48:42 -0400 (EDT) (envelope-from howardl) From: Howard Leadmon Message-Id: <200005030748.DAA84934@account.abs.net> Subject: Debugging Kernel/System Crashes, can anyone help?? To: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Date: Wed, 3 May 2000 03:48:42 -0400 (EDT) X-Mailer: ELM [version 2.4ME+ PL72 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hello, I know I posted a few messages here in the past, but maybe someone who is good at tracking kernel problems can step up and lend a hand. I have a machine running FBSD 4.0-STABLE, and have been experiencing almost daily kernel panics or reboots on the machine. I have replaced ALL of the hardware, and reloaded the OS, but still having troubles. I am at a bit of a loss as to what is going on. From one panic, I thought well maybe this is an SMP issue, but removed one of the CPU's and still the box crashes. As I have basically replaced everything, I am at a loss as to where to go from here, so looking for some type of pointers or help with this.. The other day I was there, and got the following from one of the crashes, as many times I am gone and luckally in some ways the box will just panicboot and go on it's way. Here is what I was able to copy down: Fatal trap 12: page fault while in kernel mode mp_lock=01000002; cpuid=1; lapic.id=01000000 fault virtual address= 0x30 fault code= supervisor read, page not present instruction pointer= 0x8:0xC01CAF71 stack pointer= 0x10:0xFF80DE48 frame pointer= 0x10:0xFF80DE4C code segment= base 0x0, limit 0xFFFFF, type 0x1B = DPL 0, pres 1, def 32, gran 1 processor eflags= interrupt enabled, resume, IOPL=0 current process = idle interupt mask= bio <- SMP: XXX trap number= 12 panic: page fault The formatting of it may not be perfect, but the information should be accurate, as I tried to be precise on what I wrote down. Also here are a few previous messages I had posted a while back when I thought this might be network related, but after trying several different NIC's I still have the same issues. I will include the info below, as maybe it will have some value in trying to debunk this problem. ==== Hello, I am running a 4.0-STABLE machine which is being used to host an Undernet IRC server, and the machine keeps dying at times, or should I say the networking side of it is at least dying. At first I thought it might have been related to the dc (DEC Chip) based drivers, so I replaced it with a EEpro board using the fxp driver, but the same results. I have also set MNBCLUSTERS to 20480, and when I do a netstat I see in general plenty of free clusters, but suspect I must be running out of some other resource. If I do a netstat -m, I see info like this: u2$ netstat -m 1697/2144/81920 mbufs in use (current/peak/max): 498 mbufs allocated to data 1199 mbufs allocated to packet headers 221/514/20480 mbuf clusters in use (current/peak/max) 1296 Kbytes allocated to network (50% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines When the machine last froze, the Kbytes allocated to the network was 99% in use, and the mbufs were up to about 20K of the 80K allocated, but I saw no calls for memory denied or delayed. Still after a few hours of uptime, I start seeing errors like this, and then the machines network interface dies and I have to reboot to get everything back in operation: Mar 27 12:39:00 u2 /kernel: fxp0: device timeout Mar 27 12:39:00 u2 syslogd: sendto: No buffer space available Mar 27 12:39:38 u2 last message repeated 2 times Mar 27 12:41:32 u2 last message repeated 6 times Mar 27 12:44:15 u2 syslogd: sendto: No buffer space available Mar 27 12:44:04 u2 last message repeated 8 times Not sure what other information to send, but here is a dmesg on the machine, and if anyone has any ideas, or needs more info please let me know. It's very annoying to have to reboot this machine daily, sometimes more often.. :( Copyright (c) 1992-2000 The FreeBSD Project. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 4.0-STABLE #8: Wed Mar 22 18:31:51 EST 2000 howardl@u2.abs.net:/usr/src/sys/compile/U2 Timecounter "i8254" frequency 1193182 Hz CPU: Pentium II/Pentium II Xeon/Celeron (551.25-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x665 Stepping = 5 Features=0x183fbff real memory = 402587648 (393152K bytes) config> q avail memory = 387022848 (377952K bytes) Programming 24 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 0, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 1, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 2, version: 0x00170011, at 0xfec00000 Preloaded elf kernel "kernel" at 0xc02c8000. Preloaded userconfig_script "/boot/kernel.conf" at 0xc02c809c. Pentium Pro MTRR support enabled md0: Malloc disk npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0xf000-0xf00f at device 7.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 pci0: at 7.2 irq 5 Timecounter "PIIX" frequency 3579545 Hz chip1: port 0x5000-0x500f at device 7.3 on pci0 fxp0: port 0xd400-0xd41f mem 0xd5400000-0xd54fffff,0xd5510000-0xd5510fff irq 17 at device 13.0 on pci0 fxp0: Ethernet address 00:a0:c9:c7:fb:ff pci0: at 15.0 irq 16 atapci1: port 0xe000-0xe0ff,0xdc00-0xdc03,0xd800-0xd807 irq 18 at device 19.0 on pci0 atapci2: port 0xec00-0xecff,0xe800-0xe803,0xe400-0xe407 irq 18 at device 19.1 on pci0 fdc0: at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> on fdc0 drive 0 atkbdc0: at port 0x60-0x6f on isa0 atkbd0: irq 1 on atkbdc0 vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 sc0: on isa0 sc0: VGA <16 virtual consoles, flags=0x200> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode ppi0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port plip0: on ppbus0 APIC_IO: Testing 8254 interrupt delivery APIC_IO: routing 8254 via IOAPIC #0 intpin 2 SMP: AP CPU #1 Launched! ad0: 8693MB [17662/16/63] at ata0-master using UDMA33 Mounting root from ufs:/dev/ad0s1a ===== First as mentioned previously I had an Intel EEpro card in the box running to my Cisco Catalyst switch, and on the console when everything fell apart and I lost connectivity, I see the following: fxp0: device timeout syslogd: sendto: No buffer space available Here is some of the requested debugging information: ifconfig: fxp0: flags=8843 mtu 1500 inet 207.114.4.35 netmask 0xfffffff0 broadcast 207.114.4.47 inet 207.114.4.36 netmask 0xffffffff broadcast 207.114.4.36 inet 207.114.4.45 netmask 0xffffffff broadcast 207.114.4.45 inet 207.114.4.46 netmask 0xffffffff broadcast 207.114.4.46 ether 00:a0:c9:c7:fb:ff media: autoselect (100baseTX ) status: active supported media: autoselect 100baseTX 100baseTX 10baseT/UTP 10baseT/UTP netstat -m: 403/21472/81920 mbufs in use (current/peak/max): 259 mbufs allocated to data 144 mbufs allocated to packet headers 124/10652/20480 mbuf clusters in use (current/peak/max) 23988 Kbytes allocated to network (1% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines vmstat -m: Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 286 994 4581005 0 1280 32 179 36685 518690 0 640 64 14436 4380 1373135 0 320 128 1096 88 9883 0 160 256 13335 30569 317127 0 80 512 18 6 74806 0 40 1K 107 949 12272 0 20 2K 12 6 18478 0 10 4K 13 2 98260 0 5 8K 2 2 384331 0 5 16K 8 0 2689062 0 5 32K 3 0 1321506 0 5 64K 3 0 3 0 5 128K 3 0 3 0 5 256K 1 0 1 0 5 Memory usage type by bucket size Size Type(s) 16 MD disk, kld, proc-args, atexit, temp, sysctl, bus, rman, soname, pcb, mount, vnodes, ether_multi, routetbl, p1003.1b, devbuf, isa_devlist, atkbddev 32 kld, sigio, proc-args, temp, pgrp, proc, subproc, sysctl, bus, eventhandler, SWAP, pcb, cluster_save buffer, vnodes, BPF, ifaddr, ether_multi, routetbl, in_multi, tseg_qent, devbuf 64 file, proc-args, lockf, temp, session, subproc, bus, eventhandler, rman, pcb, vfscache, cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, isadev, AD driver 128 ppbusdev, kld, timecounter, dev_t, proc-args, zombie, temp, cred, bus, ttys, soname, vfscache, cluster_save buffer, mount, vnodes, ifaddr, routetbl, ZONE, devbuf 256 file desc, proc-args, temp, subproc, bus, ttys, vnodes, ifaddr, routetbl, NFS daemon, FFS node, devbuf 512 kld, file desc, temp, bus, ioctlops, ptys, BIO buffer, mount, UFS mount, ATA generic, devbuf, isa_devlist 1K MD disk, kld, file desc, temp, proc, bus, ioctlops, BIO buffer, NQNFS Lease, AD driver, devbuf, isa_devlist 2K file desc, temp, bus, pcb, BIO buffer, UFS mount, devbuf 4K kld, file desc, temp, proc, devbuf, memdesc 8K kld, file desc, temp, UFS mount 16K file desc, temp, devbuf 32K file desc, temp, devbuf, mbuf 64K ISOFS mount, NFS hash, UFS ihash 128K temp, vfscache, VM pgdata 256K SWAP Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) MD disk 2 2K 2K 64194K 2 0 0 16,1K ppbusdev 3 1K 1K 64194K 3 0 0 128 ISOFS mount 1 64K 64K 64194K 1 0 0 64K kld 10 11K 16K 64194K 53 0 0 16,32,128,512,1K,4K,8K timecounter 10 2K 2K 64194K 10 0 0 128 dev_t 540 68K 68K 64194K 540 0 0 128 file desc 35 46K 60K 64194K 6800 0 0 256,512,1K,2K,4K,8K,16K,32K file 109 7K 283K 64194K 1135473 0 0 64 sigio 1 1K 1K 64194K 1 0 0 32 proc-args 23 1K 2K 64194K 5559 0 0 16,32,64,128,256 zombie 0 0K 1K 64194K 6753 0 0 128 atexit 1 1K 1K 64194K 1 0 0 16 lockf 1 1K 1K 64194K 23 0 0 64 temp 177 82K 115K 64194K 4596159 0 0 16,32,64,128,256,512,1K,2K,4K,8K,16K,32K,128K pgrp 22 1K 1K 64194K 1233 0 0 32 session 20 2K 2K 64194K 949 0 0 64 proc 7 10K 10K 64194K 11 0 0 32,1K,4K subproc 72 7K 10K 64194K 14795 0 0 32,64,256 cred 9 2K 2K 64194K 1082 0 0 128 sysctl 0 0K 1K 64194K 646 0 0 16,32 bus 358 29K 29K 64194K 476 0 0 16,32,64,128,256,512,1K,2K eventhandler 11 1K 1K 64194K 11 0 0 32,64 SWAP 2 141K 141K 64194K 2 0 0 32,256K ioctlops 0 0K 1K 64194K 5 0 0 512,1K rman 50 3K 3K 64194K 79 0 0 16,64 ttys 410 53K 63K 64194K 1229 0 0 128,256 ptys 3 2K 2K 64194K 3 0 0 512 soname 1 1K 1K 64194K 3940726 0 0 16,128 pcb 45 5K 20K 64194K 639691 0 0 16,32,64,2K BIO buffer 100 102K 1048K 64194K 9950 0 0 512,1K,2K vfscache 14044 1007K 1007K 64194K 17344 0 0 64,128,128K cluster_save buffer 0 0K 1K 64194K 694 0 0 32,64,128 mount 4 2K 2K 64194K 6 0 0 16,128,512 vnodes 24 6K 6K 64194K 327 0 0 16,32,64,128,256 BPF 3 1K 1K 64194K 3 0 0 32 ifaddr 15 2K 2K 64194K 15 0 0 32,64,128,256 ether_multi 7 1K 1K 64194K 7 0 0 16,32,64 routetbl 61 9K 10295K 64194K 585667 0 0 16,32,64,128,256 in_multi 2 1K 1K 64194K 2 0 0 32 tseg_qent 0 0K 5K 64194K 212819 0 0 32 NFS daemon 1 1K 1K 64194K 1 0 0 256 NQNFS Lease 1 1K 1K 64194K 1 0 0 1K NFS hash 1 64K 64K 64194K 1 0 0 64K p1003.1b 1 1K 1K 64194K 1 0 0 16 FFS node 13187 3297K 3297K 64194K 14242 0 0 256 UFS ihash 1 64K 64K 64194K 1 0 0 64K UFS mount 9 20K 20K 64194K 9 0 0 512,2K,8K VM pgdata 1 128K 128K 64194K 1 0 0 128K ZONE 18 3K 3K 64194K 18 0 0 128 isadev 11 1K 1K 64194K 11 0 0 64 ATA generic 0 1K 1K 64194K 1 0 0 512 AD driver 2 2K 2K 64194K 204988 0 0 64,1K devbuf 82 207K 207K 64194K 114 0 0 16,32,128,256,512,1K,2K,4K,16K,32K mbuf 1 28K 28K 64194K 1 0 0 32K memdesc 1 4K 4K 64194K 1 0 0 4K isa_devlist 0 0K 2K 64194K 19 0 0 16,512,1K atkbddev 2 1K 1K 64194K 2 0 0 16 Memory Totals: In Use Free Requests 5472K 10077K 11398562 Now to probably complicate things more, I replaced the EEpro card with a DEC 21143 based board using the dc driver, and with that card the machine dies a little less often, but when it does the machine usually hangs hard, or reboots. Catching the console before it's totally dead, I can see the following message scrolling on the screen: dc0: watchdog timeout Different than the error from the EEpro card, but still network related, so again I dumped the above information for comparison, and here it is: ifconfig: dc0: flags=8843 mtu 1500 inet 207.114.4.35 netmask 0xfffffff0 broadcast 207.114.4.47 inet 207.114.4.36 netmask 0xffffffff broadcast 207.114.4.36 inet 207.114.4.45 netmask 0xffffffff broadcast 207.114.4.45 inet 207.114.4.46 netmask 0xffffffff broadcast 207.114.4.46 ether 00:c0:f0:3b:a7:eb media: autoselect (100baseTX ) status: active supported media: autoselect 100baseTX 100baseTX 10baseT/UTP 10baseT/UTP none netstat -m: 7526/15744/81920 mbufs in use (current/peak/max): 6064 mbufs allocated to data 1462 mbufs allocated to packet headers 3948/7874/20480 mbuf clusters in use (current/peak/max) 17716 Kbytes allocated to network (49% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines vmstat -m: Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 296 984 4706660 0 1280 32 3254 21706 599940 0 640 64 14657 3199 1669347 0 320 128 1099 53 15587 0 160 256 16537 14743 357609 0 80 512 14 2 30928 0 40 1K 33 743 13704 0 20 2K 13 5 40824 0 10 4K 13 2 348612 0 5 8K 2 4 1255714 0 5 16K 10 0 2479452 0 5 32K 1 0 1485462 0 5 64K 4 0 4 0 5 128K 3 0 3 0 5 256K 1 0 1 0 5 Memory usage type by bucket size Size Type(s) 16 MD disk, kld, proc-args, atexit, temp, sysctl, bus, rman, soname, pcb, mount, vnodes, ether_multi, routetbl, p1003.1b, devbuf, isa_devlist, atkbddev 32 kld, sigio, proc-args, temp, pgrp, proc, subproc, sysctl, bus, eventhandler, SWAP, pcb, cluster_save buffer, vnodes, BPF, ifaddr, ether_multi, routetbl, in_multi, tseg_qent, newblk, bmsafemap, indirdep, freefrag, freefile, diradd, dirrem, devbuf 64 file, proc-args, lockf, temp, session, subproc, bus, eventhandler, rman, pcb, vfscache, cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, pagedep, allocdirect, allocindir, isadev, AD driver 128 ppbusdev, kld, timecounter, dev_t, proc-args, zombie, temp, cred, bus, ttys, soname, vfscache, cluster_save buffer, mount, vnodes, ifaddr, routetbl, inodedep, freeblks, ZONE, devbuf 256 file desc, proc-args, temp, subproc, bus, ttys, vnodes, ifaddr, routetbl, NFS daemon, newblk, FFS node, devbuf 512 kld, file desc, temp, bus, ioctlops, ptys, BIO buffer, mount, UFS mount, ATA generic, devbuf, isa_devlist 1K MD disk, kld, file desc, temp, proc, bus, ioctlops, BIO buffer, NQNFS Lease, AD driver, devbuf, isa_devlist 2K file desc, temp, bus, pcb, BIO buffer, UFS mount, devbuf 4K kld, file desc, temp, proc, devbuf, memdesc 8K kld, file desc, temp, indirdep, UFS mount 16K file desc, temp, pagedep, devbuf 32K temp, mbuf 64K ISOFS mount, NFS hash, inodedep, UFS ihash 128K temp, vfscache, VM pgdata 256K SWAP Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) MD disk 2 2K 2K 64189K 2 0 0 16,1K ppbusdev 3 1K 1K 64189K 3 0 0 128 ISOFS mount 1 64K 64K 64189K 1 0 0 64K kld 10 11K 16K 64189K 53 0 0 16,32,128,512,1K,4K,8K timecounter 10 2K 2K 64189K 10 0 0 128 dev_t 540 68K 68K 64189K 540 0 0 128 file desc 37 30K 36K 64189K 7923 0 0 256,512,1K,2K,4K,8K,16K file 174 11K 208K 64189K 1332169 0 0 64 sigio 1 1K 1K 64189K 1 0 0 32 proc-args 24 2K 2K 64189K 6468 0 0 16,32,64,128,256 zombie 0 0K 1K 64189K 7875 0 0 128 atexit 1 1K 1K 64189K 1 0 0 16 lockf 1 1K 1K 64189K 3 0 0 64 temp 169 113K 138K 64189K 5648678 0 0 16,32,64,128,256,512,1K,2K,4K,8K,16K,32K,128K pgrp 23 1K 1K 64189K 1400 0 0 32 session 21 2K 2K 64189K 1115 0 0 64 proc 7 10K 10K 64189K 7 0 0 32,1K,4K subproc 77 7K 9K 64189K 17254 0 0 32,64,256 cred 9 2K 2K 64189K 1264 0 0 128 sysctl 0 0K 1K 64189K 738 0 0 16,32 bus 367 31K 31K 64189K 503 0 0 16,32,64,128,256,512,1K,2K eventhandler 11 1K 1K 64189K 11 0 0 32,64 SWAP 2 141K 141K 64189K 2 0 0 32,256K ioctlops 0 0K 1K 64189K 5 0 0 512,1K rman 50 3K 3K 64189K 79 0 0 16,64 ttys 410 53K 58K 64189K 1307 0 0 128,256 ptys 2 1K 1K 64189K 2 0 0 512 soname 1 1K 1K 64189K 3967614 0 0 16,128 pcb 50 5K 19K 64189K 738447 0 0 16,32,64,2K BIO buffer 26 28K 769K 64189K 12308 0 0 512,1K,2K vfscache 14194 1016K 1016K 64189K 18278 0 0 64,128,128K cluster_save buffer 0 0K 1K 64189K 981 0 0 32,64,128 mount 4 2K 2K 64189K 6 0 0 16,128,512 vnodes 24 6K 6K 64189K 327 0 0 16,32,64,128,256 BPF 3 1K 1K 64189K 3 0 0 32 ifaddr 16 2K 2K 64189K 16 0 0 32,64,128,256 ether_multi 7 1K 1K 64189K 7 0 0 16,32,64 routetbl 6193 871K 6957K 64189K 663076 0 0 16,32,64,128,256 in_multi 2 1K 1K 64189K 2 0 0 32 tseg_qent 0 0K 2K 64189K 220376 0 0 32 NFS daemon 1 1K 1K 64189K 1 0 0 256 NQNFS Lease 1 1K 1K 64189K 1 0 0 1K NFS hash 1 64K 64K 64189K 1 0 0 64K p1003.1b 1 1K 1K 64189K 1 0 0 16 pagedep 2 17K 17K 64189K 32 0 0 64,16K inodedep 4 65K 68K 64189K 2813 0 0 128,64K newblk 1 1K 1K 64189K 23834 0 0 32,256 bmsafemap 3 1K 1K 64189K 4690 0 0 32 allocdirect 1 1K 2K 64189K 8555 0 0 64 indirdep 1 1K 25K 64189K 2822 0 0 32,8K allocindir 1 1K 26K 64189K 15278 0 0 64 freefrag 0 0K 4K 64189K 3464 0 0 32 freeblks 0 0K 4K 64189K 1520 0 0 128 freefile 0 0K 1K 64189K 40 0 0 32 diradd 2 1K 1K 64189K 61 0 0 32 dirrem 0 0K 1K 64189K 64 0 0 32 FFS node 13320 3330K 3331K 64189K 14373 0 0 256 UFS ihash 1 64K 64K 64189K 1 0 0 64K UFS mount 9 20K 20K 64189K 9 0 0 512,2K,8K VM pgdata 1 128K 128K 64189K 1 0 0 128K ZONE 18 3K 3K 64189K 18 0 0 128 isadev 11 1K 1K 64189K 11 0 0 64 ATA generic 0 1K 1K 64189K 1 0 0 512 AD driver 1 1K 2K 64189K 277266 0 0 64,1K devbuf 81 175K 175K 64189K 113 0 0 16,32,128,256,512,1K,2K,4K,16K mbuf 1 28K 28K 64189K 1 0 0 32K memdesc 1 4K 4K 64189K 1 0 0 4K isa_devlist 0 0K 2K 64189K 18 0 0 16,512,1K atkbddev 2 1K 1K 64189K 2 0 0 16 Memory Totals: In Use Free Requests 6372K 5380K 13003847 All of the above stats were taken while the network card was spitting out errors prior to performing a reboot which brings the box back online. I also tried unplugging the nic and plugging it back in without out any change. I also over time have replaced everything in the box except the case, but still the problem persists, and in fact took the old hardware and built a different machine that works fine. So something related to the heavy use by the IRC programs is killing this thing almost daily, and I am at a loss as to what. If you or anyone here on the list has any ideas, I would sure love to hear them, as it would be nice to get to the bottom of this issue... ===== Guess thats about it for now, and hopefully someone can give me some pointers, or lend a hand in trying to get this issue resolved as it's driving me nutz.. :) --- Howard Leadmon - howardl@abs.net - http://www.abs.net ABSnet Internet Services - Phone: 410-361-8160 - FAX: 410-361-8162 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message