From owner-freebsd-stable@FreeBSD.ORG Wed Nov 22 20:43:45 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9EEE516A492; Wed, 22 Nov 2006 20:43:45 +0000 (UTC) (envelope-from vulture@netvulture.com) Received: from rackman.netvulture.com (adsl-63-197-17-60.dsl.snfc21.pacbell.net [63.197.17.60]) by mx1.FreeBSD.org (Postfix) with ESMTP id 008AE43D5C; Wed, 22 Nov 2006 20:43:11 +0000 (GMT) (envelope-from vulture@netvulture.com) Received: from [192.168.2.249] (coolman [208.201.244.79]) (authenticated bits=0) by rackman.netvulture.com (8.13.5/8.13.5) with ESMTP id kAMKfR3U040361; Wed, 22 Nov 2006 12:41:28 -0800 (PST) Message-ID: <4564B5E0.3000404@netvulture.com> Date: Wed, 22 Nov 2006 12:41:04 -0800 From: Jonathan Feally User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-stable@freebsd.org, freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-netvulture_com-MailScanner-Information: Please contact the ISP for more information X-netvulture_com-MailScanner: Found to be clean X-netvulture_com-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (timed out) X-netvulture_com-MailScanner-From: vulture@netvulture.com X-Spam-Status: No Cc: Subject: 6-STABLE (6.2-PRE) and applications (named natd dhcpd) getting stuck in state zoneli (zone limit) - dynamic ipfw rules not working after time - vlans on em X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2006 20:43:45 -0000 Sorry to cross post, but the net list didn't help a couple weeks back on this. names, natd, and dhcpd have all been getting stuck in zoneli (zone limit) since I upgraded to the box to stable about a month ago. It was running a 6.1-STABLE before with out difficulty. Very little has changed on the box. All the same applications, same ipfw rules for the most part (just more rules for new customers). Most of the time the processes cannot be killed. I did get lucky yesterday with dhcpd. It finally died about an hour later. I was compiling the latest stable at the time. But got up again today to find dhcpd in zoneli. Can someone please point me in the correct direction to trouble shoot this problem. I don't really know how to get a full dump of what a process is doing, so a quick what to do and post back would be great. More Info On the Setup: This box is acting as a all-in-one router with traffic shaping. It has a single em card with 10 tagged vlans on it routing the majority of traffic from the customer vlan (vlan125) to the isp vlan (vlan901). It also runs dhcpd and named to service the customers on vlan125. All of the customer's have a set of ipfw rules like the following. # ipfw queue 11206 show q11206: weight 40 pipe 1206 384 KB 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 tcp x.y.125.136/1925 a.b.247.137/9580 264461 111550268 0 0 0 # ipfw queue 21206 show q21206: weight 60 pipe 1206 384 KB 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 udp c.d.55.125/36049 x.y.125.136/17307 239325 38512205 1 76 0 # ipfw queue 31206 show q31206: weight 21 pipe 10000 384 KB 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 tcp x.y.125.136/1925 a.b.247.137/9580 265850 112136314 0 0 0 # ipfw queue 41206 show q41206: weight 21 pipe 20000 384 KB 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 udp c.d.55.125/36049 x.y.125.136/17307 239447 38467203 0 0 0 # ipfw pipe 1206 show 01206: 2.090 Mbit/s 0 ms 768 KB 0 queues (1 buckets) droptail q11206: weight 40 pipe 1206 384 KB 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 tcp x.y.125.136/1925 a.b.247.137/9580 267269 113411268 0 0 0 q21206: weight 60 pipe 1206 384 KB 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 udp c.d.55.125/36049 x.y.125.136/17307 241121 38680715 0 0 0 # ipfw pipe 10000 show 10000: 20.224 Mbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail q31206: weight 21 pipe 10000 384 KB 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 tcp x.y.125.136/1925 a.b.247.137/9580 268208 113672991 0 0 0 # ipfw pipe 20000 show 20000: 20.224 Mbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail q41206: weight 21 pipe 20000 384 KB 1 queues (1 buckets) droptail mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 udp c.d.55.125/36049 x.y.125.136/17307 241392 38632811 0 0 0 # The 10000 and 20000 pipes are for the whole link, while the 1206 pipe is the customers alloted bandwidth. Customer cannot exceed his bandwidth, and if the whole connection is maxed out he would be weighted at 21 against other customers competing for isp bandwidth. The more bandwidth a customer pays for the bigger his weight in the main pipes. #Customer rules (#10000-#49999) 19000 0 0 reset tcp from any 25,135,139,445 to x.y.125.136 19000 4 176 reset tcp from any to x.y.125.136 dst-port 25,135,139,445 19000 0 0 reset tcp from x.y.125.136 to any dst-port 25,135,139,445 19001 216130 87769342 queue 11206 ip from x.y.125.136 to any out 19002 197604 34923736 queue 21206 ip from any to x.y.125.136 out 19003 216635 87807692 queue 31206 ip from x.y.125.136 to any in 19004 197283 34831273 queue 41206 ip from any to x.y.125.136 in 19005 432765 175577034 allow ip from x.y.125.136 to any 19006 394887 69755009 allow ip from any to x.y.125.136 #Other keep state rules - the keep-state rules will stop working after a while. 50000 3325 258473 allow udp from any to any dst-port 53 50000 3020 938086 allow udp from any 53 to any 50001 41462 28661128 divert 8668 ip from any to any via vlan901 50002 0 0 check-state # Used to forward all un-provisioned customer machines to the local httpd so their mac can be recorded. 51004 358 23712 fwd 127.0.0.1,8080 tcp from x.y.125.0/24 to any dst-port 80 in keep-state # Drop anybody not allowed above 51099 1158 141144 deny ip from x.y.125.0/24 to any # Allow admin vlan1 to go anywhere 60003 257 50939 allow ip from 10.255.1.0/24 to any keep-state $ Final drop 65000 430 44547 reset tcp from any to any 65001 1887 424765 deny ip from any to any Below are some dumps of process states when it locks up #NATD zoneli UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 96 0 0 0 - WLs ?? 0:00.00 [swapper] 0 1 0 0 8 0 772 368 wait ILs ?? 0:00.80 /sbin/init -- 0 2 0 0 8 0 0 8 crypto DL ?? 0:00.00 [crypto] 0 3 0 0 8 0 0 8 crypto DL ?? 0:00.00 [crypto returns] 0 4 0 0 -8 0 0 8 - DL ?? 0:12.55 [g_event] 0 5 0 0 -8 0 0 8 - DL ?? 0:17.83 [g_up] 0 6 0 0 -8 0 0 8 - DL ?? 0:18.84 [g_down] 0 7 0 0 8 0 0 8 - DL ?? 0:00.00 [kqueue taskq] 0 8 0 0 8 0 0 8 - DL ?? 0:00.00 [acpi_task_0] 0 9 0 0 8 0 0 8 - DL ?? 0:00.00 [acpi_task_1] 0 10 0 133 171 0 0 8 - RL ?? 4032:07.99 [idle: cpu0] 0 11 0 5 -44 0 0 8 - WL ?? 215:09.39 [swi1: net] 0 12 0 0 -32 0 0 8 - WL ?? 195:44.05 [swi4: clock sio] 0 13 0 0 -36 0 0 8 - WL ?? 0:00.00 [swi3: vm] 0 14 0 0 -16 0 0 8 - DL ?? 11:00.18 [yarrow] 0 15 0 0 -40 0 0 8 - WL ?? 0:00.00 [swi2: cambio] 0 16 0 0 -24 0 0 8 - WL ?? 0:00.00 [swi6: task queue] 0 17 0 0 8 0 0 8 - DL ?? 0:00.00 [acpi_task_2] 0 18 0 0 -24 0 0 8 - WL ?? 0:00.00 [swi6: Giant taskq] 0 19 0 0 8 0 0 8 - DL ?? 0:00.00 [thread taskq] 0 20 0 0 -28 0 0 8 - WL ?? 0:00.00 [swi5: +] 0 21 0 0 -52 0 0 8 - WL ?? 0:00.00 [irq9: acpi0] 0 22 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq16: uhci0 uhci3] 0 23 0 0 8 0 0 8 usbevt DL ?? 0:00.02 [usb0] 0 24 0 0 8 0 0 8 usbtsk DL ?? 0:00.00 [usbtask] 0 25 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq19: uhci1] 0 26 0 0 8 0 0 8 usbevt DL ?? 0:00.01 [usb1] 0 27 0 0 -64 0 0 8 - WL ?? 0:08.08 [irq18: uhci2+] 0 28 0 0 8 0 0 8 usbevt DL ?? 0:00.02 [usb2] 0 29 0 0 8 0 0 8 usbevt DL ?? 0:00.01 [usb3] 0 30 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq23: ehci0] 0 31 0 0 8 0 0 8 usbevt DL ?? 0:00.02 [usb4] 0 32 0 1 -68 0 0 8 - DL ?? 46:33.63 [em0 taskq] 0 33 0 0 -68 0 0 8 - WL ?? 0:00.00 [irq20: fxp0] 0 34 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq14: ata0] 0 35 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq15: ata1] 0 36 0 0 -60 0 0 8 - WL ?? 0:00.00 [irq1: atkbd0] 0 37 0 0 -48 0 0 8 - WL ?? 0:00.00 [swi0: sio] 0 38 0 0 -60 0 0 8 - WL ?? 0:00.00 [irq7: ppc0] 0 39 0 0 -16 0 0 8 psleep DL ?? 0:00.29 [pagedaemon] 0 40 0 0 20 0 0 8 psleep DL ?? 0:00.00 [vmdaemon] 0 41 0 0 171 0 0 8 pgzero DL ?? 1:07.23 [pagezero] 0 42 0 0 -16 0 0 8 psleep DL ?? 0:01.50 [bufdaemon] 0 43 0 0 -4 0 0 8 vlruwt DL ?? 0:01.90 [vnlru] 0 44 0 0 20 0 0 8 syncer DL ?? 2:28.29 [syncer] 0 45 0 0 -16 0 0 8 sdflus DL ?? 0:03.68 [softdepflush] 0 46 0 0 -64 0 0 8 - DL ?? 0:19.45 [schedcpu] 0 163 1 167 20 0 1252 728 pause Is ?? 0:00.00 adjkerntz -i 0 1242 1 167 116 0 528 388 select Is ?? 0:00.00 /sbin/devd 0 1486 1 0 -16 0 1572 1112 zoneli Ds ?? 1:47.61 /sbin/natd -f /etc/natd.conf -n vlan901 0 1645 1 0 96 0 1404 1112 select Ss ?? 0:18.82 /usr/sbin/syslogd -l /var/run/log -l /var/named/var/run/log -c 53 1707 1 0 96 0 13552 12700 select Ss ?? 7:38.07 /usr/sbin/named -t /var/named -u bind 0 1717 1 0 96 0 1504 1216 select Is ?? 0:00.26 /usr/sbin/rpcbind 0 1772 1 176 118 0 1492 1256 select Is ?? 0:00.00 /usr/sbin/mountd -r 0 1774 1 176 118 0 1372 1104 select Is ?? 0:00.03 nfsd: master (nfsd) 0 1775 1774 176 4 0 1280 888 - I ?? 0:00.00 nfsd: server (nfsd) 0 1776 1774 176 4 0 1280 888 - I ?? 0:00.00 nfsd: server (nfsd) 0 1777 1774 176 4 0 1280 888 - I ?? 0:00.00 nfsd: server (nfsd) 0 1778 1774 176 4 0 1280 888 - I ?? 0:00.00 nfsd: server (nfsd) 0 1784 1 0 96 0 263584 1112 select Is ?? 0:00.20 /usr/sbin/rpc.statd 0 1789 1 0 96 0 1596 1364 select Is ?? 0:00.24 rpc.lockd: server (rpc.lockd) 1 1796 1789 176 4 0 1596 1364 nfsloc I ?? 0:00.00 rpc.lockd: client (rpc.lockd) 0 1808 1 0 96 0 3204 2508 select Ss ?? 0:01.08 /usr/local/sbin/racoon 0 1853 1 0 96 0 2924 1824 select Ss ?? 0:14.87 /usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db 0 1873 1 0 96 0 1288 824 select Is ?? 0:00.36 /usr/sbin/usbd 0 1902 1 0 96 0 3520 2648 select Is ?? 0:13.45 /usr/sbin/sshd 0 1907 1 0 96 0 3508 2948 select Ss ?? 0:05.87 sendmail: accepting connections (sendmail) 25 1911 1 0 20 0 3384 2804 pause Is ?? 0:00.10 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (se 0 1917 1 0 8 0 1388 1100 nanslp Is ?? 0:30.23 /usr/sbin/cron -s 0 6535 1 0 8 0 11708 9140 nanslp Ss ?? 0:10.04 /usr/local/apache2/bin/httpd -k start 0 6565 1 0 96 0 1468 1220 select Is ?? 0:00.00 /usr/sbin/inetd -wW -C 60 65534 18455 6535 0 20 0 12040 9488 lockf I ?? 0:00.02 /usr/local/apache2/bin/httpd -k start 65534 51018 6535 0 20 0 12136 9632 lockf I ?? 0:00.09 /usr/local/apache2/bin/httpd -k start 65534 51041 6535 0 4 0 12128 9624 kqread I ?? 0:00.12 /usr/local/apache2/bin/httpd -k start 65534 51042 6535 0 20 0 12064 9560 lockf I ?? 0:00.03 /usr/local/apache2/bin/httpd -k start 65534 51439 6535 0 20 0 12140 9632 lockf I ?? 0:00.08 /usr/local/apache2/bin/httpd -k start 65534 51462 6535 0 20 0 12196 9700 lockf I ?? 0:00.13 /usr/local/apache2/bin/httpd -k start 65534 51463 6535 0 20 0 12064 9552 lockf I ?? 0:00.10 /usr/local/apache2/bin/httpd -k start 65534 51476 6535 0 20 0 12060 9512 lockf I ?? 0:00.02 /usr/local/apache2/bin/httpd -k start 65534 51477 6535 0 20 0 12152 9648 lockf I ?? 0:00.11 /usr/local/apache2/bin/httpd -k start 65534 51479 6535 0 20 0 12048 9540 lockf I ?? 0:00.07 /usr/local/apache2/bin/httpd -k start 1001 82596 1 0 96 0 2732 2452 select Ss ?? 0:03.93 /usr/local/sbin/dhcpd -cf /usr/local/etc/dhcpd.conf -lf /var/db 0 88474 1902 0 4 0 6272 3276 sbwait Is ?? 0:00.04 sshd: vulture [priv] (sshd) 1002 88653 88474 0 96 0 6256 3280 select S ?? 0:00.23 sshd: vulture@ttyp0 (sshd) 0 6583 1 172 5 0 1344 952 ttyin Is+ v0 0:00.00 /usr/libexec/getty Pc ttyv0 0 6584 1 172 5 0 1344 952 ttyin Is+ v1 0:00.00 /usr/libexec/getty Pc ttyv1 0 6585 1 172 5 0 1344 952 ttyin Is+ v2 0:00.00 /usr/libexec/getty Pc ttyv2 0 6586 1 172 5 0 1344 952 ttyin Is+ v3 0:00.00 /usr/libexec/getty Pc ttyv3 0 6587 1 172 5 0 1344 952 ttyin Is+ v4 0:00.00 /usr/libexec/getty Pc ttyv4 0 6588 1 172 5 0 1344 952 ttyin Is+ v5 0:00.00 /usr/libexec/getty Pc ttyv5 0 6589 1 172 5 0 1344 952 ttyin Is+ v6 0:00.00 /usr/libexec/getty Pc ttyv6 0 6590 1 172 5 0 1344 952 ttyin Is+ v7 0:00.00 /usr/libexec/getty Pc ttyv7 0 1882 1 0 -58 0 3204 2832 bpf S con- 0:06.62 /usr/local/sbin/arpwatch -i vlan125 -f arp.vlan125.dat 1002 88657 88653 0 8 0 3196 2052 wait Is p0 0:00.01 -bash (bash) 0 88661 88657 0 8 0 1776 1416 wait I p0 0:00.01 su - 0 88662 88661 1 8 0 3232 2092 wait S p0 0:00.06 -su (bash) 0 90806 88662 0 96 0 1500 988 - R+ p0 0:00.00 ps -axl #Named zoneli UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND USER %CPU %MEM STARTED 0 10 0 269 171 0 0 8 - RL ?? 3841:42.30 [idle: cpu0] root 79.2 0.0 Sun05PM 0 57074 56659 87 106 0 4892 4472 select S+ v1 0:10.89 cvsup system/sta root 9.6 0.9 5:00PM 0 11 0 42 -44 0 0 8 - WL ?? 190:27.13 [swi1: net] root 4.8 0.0 Sun05PM 0 0 0 0 96 0 0 0 - WLs ?? 0:00.00 [swapper] root 0.0 0.0 Sun05PM 0 1 0 0 8 0 772 368 wait ILs ?? 0:00.08 /sbin/init -- root 0.0 0.1 Sun05PM 0 2 0 0 8 0 0 8 crypto DL ?? 0:00.00 [crypto] root 0.0 0.0 Sun05PM 0 3 0 0 8 0 0 8 crypto DL ?? 0:00.00 [crypto returns] root 0.0 0.0 Sun05PM 0 4 0 0 -8 0 0 8 - DL ?? 0:11.47 [g_event] root 0.0 0.0 Sun05PM 0 5 0 0 -8 0 0 8 - DL ?? 0:14.95 [g_up] root 0.0 0.0 Sun05PM 0 6 0 1 -8 0 0 8 - DL ?? 0:17.02 [g_down] root 0.0 0.0 Sun05PM 0 7 0 0 8 0 0 8 - DL ?? 0:00.00 [kqueue taskq] root 0.0 0.0 Sun05PM 0 8 0 0 8 0 0 8 - DL ?? 0:00.00 [acpi_task_0] root 0.0 0.0 Sun05PM 0 9 0 0 8 0 0 8 - DL ?? 0:00.00 [acpi_task_1] root 0.0 0.0 Sun05PM 0 12 0 1 -32 0 0 8 - WL ?? 148:47.86 [swi4: clock sio root 0.0 0.0 Sun05PM 0 13 0 0 -36 0 0 8 - WL ?? 0:00.00 [swi3: vm] root 0.0 0.0 Sun05PM 0 14 0 0 -16 0 0 8 - DL ?? 8:07.19 [yarrow] root 0.0 0.0 Sun05PM 0 15 0 0 -40 0 0 8 - WL ?? 0:00.00 [swi2: cambio] root 0.0 0.0 Sun05PM 0 16 0 0 -24 0 0 8 - WL ?? 0:00.00 [swi6: task queu root 0.0 0.0 Sun05PM 0 17 0 0 8 0 0 8 - DL ?? 0:00.00 [acpi_task_2] root 0.0 0.0 Sun05PM 0 18 0 1 -24 0 0 8 - WL ?? 0:00.56 [swi6: Giant tas root 0.0 0.0 Sun05PM 0 19 0 0 8 0 0 8 - DL ?? 0:00.00 [thread taskq] root 0.0 0.0 Sun05PM 0 20 0 0 -28 0 0 8 - WL ?? 0:00.00 [swi5: +] root 0.0 0.0 Sun05PM 0 21 0 0 -52 0 0 8 - WL ?? 0:00.00 [irq9: acpi0] root 0.0 0.0 Sun05PM 0 22 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq16: uhci0 uh root 0.0 0.0 Sun05PM 0 23 0 0 8 0 0 8 usbevt DL ?? 0:00.01 [usb0] root 0.0 0.0 Sun05PM 0 24 0 0 8 0 0 8 usbtsk DL ?? 0:00.00 [usbtask] root 0.0 0.0 Sun05PM 0 25 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq19: uhci1] root 0.0 0.0 Sun05PM 0 26 0 0 8 0 0 8 usbevt DL ?? 0:00.01 [usb1] root 0.0 0.0 Sun05PM 0 27 0 1 -64 0 0 8 - WL ?? 0:06.87 [irq18: uhci2+] root 0.0 0.0 Sun05PM 0 28 0 0 8 0 0 8 usbevt DL ?? 0:00.01 [usb2] root 0.0 0.0 Sun05PM 0 29 0 0 8 0 0 8 usbevt DL ?? 0:00.01 [usb3] root 0.0 0.0 Sun05PM 0 30 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq23: ehci0] root 0.0 0.0 Sun05PM 0 31 0 0 8 0 0 8 usbevt DL ?? 0:00.02 [usb4] root 0.0 0.0 Sun05PM 0 32 0 4 -68 0 0 8 - DL ?? 36:30.75 [em0 taskq] root 0.0 0.0 Sun05PM 0 33 0 0 -68 0 0 8 - WL ?? 0:00.00 [irq20: fxp0] root 0.0 0.0 Sun05PM 0 34 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq14: ata0] root 0.0 0.0 Sun05PM 0 35 0 0 -64 0 0 8 - WL ?? 0:00.00 [irq15: ata1] root 0.0 0.0 Sun05PM 0 36 0 0 -60 0 0 8 - WL ?? 0:00.04 [irq1: atkbd0] root 0.0 0.0 Sun05PM 0 37 0 0 -48 0 0 8 - WL ?? 0:00.00 [swi0: sio] root 0.0 0.0 Sun05PM 0 38 0 0 -60 0 0 8 - WL ?? 0:00.00 [irq7: ppc0] root 0.0 0.0 Sun05PM 0 39 0 0 -16 0 0 8 psleep DL ?? 0:00.26 [pagedaemon] root 0.0 0.0 Sun05PM 0 40 0 0 20 0 0 8 psleep DL ?? 0:00.00 [vmdaemon] root 0.0 0.0 Sun05PM 0 41 0 0 171 0 0 8 pgzero DL ?? 0:58.03 [pagezero] root 0.0 0.0 Sun05PM 0 42 0 0 -16 0 0 8 psleep DL ?? 0:01.37 [bufdaemon] root 0.0 0.0 Sun05PM 0 43 0 0 -4 0 0 8 vlruwt DL ?? 0:01.78 [vnlru] root 0.0 0.0 Sun05PM 0 44 0 0 20 0 0 8 syncer DL ?? 2:53.26 [syncer] root 0.0 0.0 Sun05PM 0 45 0 0 -16 0 0 8 sdflus DL ?? 0:03.31 [softdepflush] root 0.0 0.0 Sun05PM 0 46 0 0 -64 0 0 8 - DL ?? 0:17.67 [schedcpu] root 0.0 0.0 Sun05PM 0 163 1 167 20 0 1252 728 pause Is ?? 0:00.00 adjkerntz -i root 0.0 0.1 Sun05PM 0 1242 1 165 116 0 528 388 select Is ?? 0:00.00 /sbin/devd root 0.0 0.1 Sun05PM 0 1486 1 0 96 0 1568 1104 select Ss ?? 0:19.48 /sbin/natd -f /e root 0.0 0.2 Sun05PM 65534 1590 6647 0 20 0 11788 9236 lockf S ?? 0:00.04 /usr/local/apach nobody 0.0 1.8 3:14AM 0 1645 1 0 96 0 1408 1144 select Ss ?? 0:16.44 /usr/sbin/syslog root 0.0 0.2 Sun05PM 53 1707 1 0 -16 0 9296 8436 zoneli Ds ?? 5:48.13 /usr/sbin/named bind 0.0 1.6 Sun05PM 0 1717 1 0 96 0 1504 1216 select Is ?? 0:00.23 /usr/sbin/rpcbin root 0.0 0.2 Sun05PM 0 1772 1 164 116 0 1492 1256 select Is ?? 0:00.00 /usr/sbin/mountd root 0.0 0.2 Sun05PM 0 1774 1 164 116 0 1372 1104 select Is ?? 0:00.03 nfsd: master (nf root 0.0 0.2 Sun05PM 0 1775 1774 164 4 0 1280 888 - I ?? 0:00.00 nfsd: server (nf root 0.0 0.2 Sun05PM 0 1776 1774 164 4 0 1280 888 - I ?? 0:00.00 nfsd: server (nf root 0.0 0.2 Sun05PM 0 1777 1774 164 4 0 1280 888 - I ?? 0:00.00 nfsd: server (nf root 0.0 0.2 Sun05PM 0 1778 1774 164 4 0 1280 888 - I ?? 0:00.00 nfsd: server (nf root 0.0 0.2 Sun05PM 0 1784 1 0 96 0 263584 1112 select Is ?? 0:00.19 /usr/sbin/rpc.st root 0.0 0.2 Sun05PM 0 1789 1 0 96 0 1596 1364 select Is ?? 0:00.22 rpc.lockd: serve root 0.0 0.3 Sun05PM 1 1796 1789 164 4 0 1596 1364 nfsloc I ?? 0:00.00 rpc.lockd: clien daemon 0.0 0.3 Sun05PM 0 1808 1 0 96 0 3204 2436 select Ss ?? 0:01.30 /usr/local/sbin/ root 0.0 0.5 Sun05PM 0 2101 1 0 96 0 2924 1824 select Ss ?? 0:13.51 /usr/sbin/ntpd - root 0.0 0.4 Sun05PM 0 2121 1 0 96 0 1288 824 select Is ?? 0:00.33 /usr/sbin/usbd root 0.0 0.2 Sun05PM 0 2150 1 40 101 0 3520 2648 select Is ?? 0:01.11 /usr/sbin/sshd root 0.0 0.5 Sun05PM 0 2157 1 0 96 0 3508 2944 select Ss ?? 0:05.36 sendmail: accept root 0.0 0.6 Sun05PM 25 2161 1 0 20 0 3384 2804 pause Is ?? 0:00.09 sendmail: Queue smmsp 0.0 0.5 Sun05PM 0 2167 1 9 8 0 1388 1100 nanslp Ss ?? 0:27.45 /usr/sbin/cron - root 0.0 0.2 Sun05PM 65534 3605 6647 0 20 0 12148 9644 lockf S ?? 0:00.38 /usr/local/apach nobody 0.0 1.9 9:06PM 0 6647 1 0 8 0 11708 9140 nanslp Ss ?? 0:09.25 /usr/local/apach root 0.0 1.8 Sun05PM 0 6672 1 158 115 0 1468 1168 select Is ?? 0:00.00 /usr/sbin/inetd root 0.0 0.2 Sun05PM 65534 34952 6647 0 20 0 12340 9820 lockf S ?? 0:01.05 /usr/local/apach nobody 0.0 1.9 9:48PM 1001 43220 1 0 96 0 2708 2428 select Ss ?? 0:00.42 /usr/local/sbin/ dhcpd 0.0 0.5 3:46PM 65534 49660 6647 0 20 0 11788 9236 lockf S ?? 0:00.03 /usr/local/apach nobody 0.0 1.8 7:32AM 65534 49710 6647 0 20 0 11788 9236 lockf S ?? 0:00.02 /usr/local/apach nobody 0.0 1.8 7:32AM 65534 50579 6647 0 4 0 12132 9624 kqread S ?? 0:00.14 /usr/local/apach nobody 0.0 1.9 11:07PM 65534 51202 6647 0 20 0 12092 9596 lockf S ?? 0:00.13 /usr/local/apach nobody 0.0 1.9 11:10PM 0 57451 2167 0 -8 0 1388 1164 piperd S ?? 0:00.00 cron: running jo root 0.0 0.2 5:03PM 0 57452 2167 0 -8 0 1388 1164 piperd S ?? 0:00.00 cron: running jo root 0.0 0.2 5:03PM 0 57454 2167 0 -8 0 1388 1164 piperd S ?? 0:00.00 cron: running jo root 0.0 0.2 5:03PM 0 57457 57451 0 8 0 1708 1224 wait Ss ?? 0:00.00 /bin/sh -c /etc/ root 0.0 0.2 5:03PM 0 57458 57454 0 8 0 1708 1224 wait Ss ?? 0:00.00 /bin/sh -c /etc/ root 0.0 0.2 5:03PM 0 57466 57452 1 8 0 1708 1224 wait Ss ?? 0:00.00 /bin/sh -c /etc/ root 0.0 0.2 5:03PM 0 57484 57457 0 8 0 1704 1256 wait S ?? 0:00.00 /bin/sh /etc/pin root 0.0 0.2 5:03PM 0 57495 57466 2 8 0 1704 1256 wait S ?? 0:00.00 /bin/sh /etc/pin root 0.0 0.2 5:03PM 0 57509 57458 0 8 0 1704 1256 wait S ?? 0:00.00 /bin/sh /etc/pin root 0.0 0.2 5:03PM 0 57521 57495 2 8 0 1248 472 nanslp S ?? 0:00.00 sleep 21 root 0.0 0.1 5:03PM 0 57523 57484 0 8 0 1248 472 nanslp S ?? 0:00.00 sleep 22 root 0.0 0.1 5:03PM 0 57525 57509 0 8 0 1248 472 nanslp S ?? 0:00.00 sleep 20 root 0.0 0.1 5:03PM 65534 97950 6647 0 20 0 12128 9624 lockf S ?? 0:00.38 /usr/local/apach nobody 0.0 1.9 2:57AM 65534 98135 6647 0 20 0 12064 9568 lockf S ?? 0:00.22 /usr/local/apach nobody 0.0 1.9 2:58AM 65534 98319 6647 0 96 0 12260 9744 select S ?? 0:00.15 /usr/local/apach nobody 0.0 1.9 2:59AM 0 6690 1 0 8 0 1760 1512 wait Is v0 0:00.03 login [pam] (log root 0.0 0.3 Sun05PM 0 56648 6690 0 8 0 3216 2072 wait S v0 0:00.03 -bash (bash) root 0.0 0.4 4:58PM 0 57620 56648 1 96 0 1508 1032 - R+ v0 0:00.00 ps -axeluwww root 0.0 0.2 5:03PM 0 6691 1 0 8 0 1740 1488 wait Is v1 0:00.02 login [pam] (log root 0.0 0.3 Sun05PM 0 56659 6691 0 8 0 3216 2072 wait I v1 0:00.04 -bash (bash) root 0.0 0.4 4:58PM 0 6692 1 0 8 0 1736 1460 wait Is v2 0:00.02 login [pam] (log root 0.0 0.3 Sun05PM 0 57198 6692 0 5 0 3212 2068 ttyin S+ v2 0:00.01 -bash (bash) root 0.0 0.4 5:01PM 0 6693 1 158 5 0 1344 952 ttyin Is+ v3 0:00.00 /usr/libexec/get root 0.0 0.2 Sun05PM 0 6694 1 158 5 0 1344 952 ttyin Is+ v4 0:00.00 /usr/libexec/get root 0.0 0.2 Sun05PM 0 6695 1 158 5 0 1344 952 ttyin Is+ v5 0:00.00 /usr/libexec/get root 0.0 0.2 Sun05PM 0 6696 1 158 5 0 1344 952 ttyin Is+ v6 0:00.00 /usr/libexec/get root 0.0 0.2 Sun05PM 0 6697 1 158 5 0 1344 952 ttyin Is+ v7 0:00.00 /usr/libexec/get root 0.0 0.2 Sun05PM 0 2130 1 0 -58 0 3216 2844 bpf S con- 0:06.37 /usr/local/sbin/ root 0.0 0.6 Sun05PM Hopefully somebody can point me in the correct direction soon. dhcpd is locking up within 12 hours now with the latest stable from yesterday. If its the way I have the vlans all on 1 card, then I can fix that. Thanks -Jon