From owner-freebsd-stable@FreeBSD.ORG Tue Jul 26 07:53:14 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B36A3106566B; Tue, 26 Jul 2011 07:53:14 +0000 (UTC) (envelope-from amon@aelita.org) Received: from aelita.org (unknown [IPv6:2001:7a8:7003::3]) by mx1.freebsd.org (Postfix) with ESMTP id B5E418FC12; Tue, 26 Jul 2011 07:53:12 +0000 (UTC) Received: from ra.aabs (localhost [127.0.0.1]) by aelita.org (8.14.4/8.14.4) with ESMTP id p6Q9nEZw090394; Tue, 26 Jul 2011 11:49:14 +0200 (CEST) (envelope-from amon@ra.aabs) Received: (from amon@localhost) by ra.aabs (8.14.4/8.14.4/Submit) id p6Q9nDx5090393; Tue, 26 Jul 2011 11:49:13 +0200 (CEST) (envelope-from amon) Date: Tue, 26 Jul 2011 11:49:13 +0200 From: Herve Boulouis To: Kostik Belousov Message-ID: <20110726094913.GF17204@ra.aabs> References: <20110725102107.GB17204@ra.aabs> <20110725085902.GM17489@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: text/plain; charset=unknown-8bit Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20110725085902.GM17489@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.4.2.3i Cc: rmacklem@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Sleeping thread owns a nonsleepable lock panic (& lor) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2011 07:53:14 -0000 Le 25/07/2011 11:59, Kostik Belousov a écrit: Ok the patched server crashed this morning strangely : all httpd processes were stuck in nfs or vmopar and were unkillable. Below is the full ps. When tried to reboot it got stuck after "All buffers synced" with LOR : lock order reversal: 1st 0xffffff00094a4270 ufs (ufs) @ kern/vfs_mount.c:1204 2nd 0xffffff000960f7f8 syncer (syncer) @ kern/vfs_subr.c:2232 lock order reversal: 1st 0xffffff000960f9d0 nfs (nfs) @ kern/vfs_subr.c:2120 2nd 0xffffffff80b567e0 allproc (allproc) @ kern/kern_descrip.c:2518 In my original post I forgot to add that this box (Dell M610) has 24 cores inside (2 physicals CPUs with 6 cores each and HT enabled). Full ps before reboot : UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 8 0 0 96 - DLs ?? 0:45.66 [kernel] 0 1 0 0 60 0 3204 120 wait ILs ?? 0:00.78 /sbin/init -- 0 2 0 0 -8 0 0 16 - DL ?? 0:03.70 [g_event] 0 3 0 0 -8 0 0 16 - DL ?? 3:18.53 [g_up] 0 4 0 0 -8 0 0 16 - DL ?? 1:01.57 [g_down] 0 5 0 0 -16 0 0 16 idle DL ?? 0:00.00 [mpt_recovery0] 0 6 0 0 -16 0 0 16 idle DL ?? 0:00.00 [mpt_raid0] 0 7 0 0 -16 0 0 16 waitin DL ?? 0:00.00 [sctp_iterator] 0 8 0 0 -16 0 0 16 pftm DL ?? 0:00.38 [pfpurge] 0 9 0 0 -16 0 0 16 ccb_sc DL ?? 0:00.00 [xpt_thrd] 0 10 0 0 -16 0 0 16 audit_ DL ?? 0:00.00 [audit] 0 11 0 0 171 0 0 384 - RL ?? 21204:47.83 [idle] 0 12 0 0 -60 0 0 640 - WL ?? 21:01.69 [intr] 0 13 0 0 44 0 0 16 - DL ?? 0:42.12 [yarrow] 0 14 0 0 -64 0 0 448 - DL ?? 0:01.85 [usb] 0 15 0 0 -8 0 0 16 mdwait DL ?? 0:00.00 [md0] 0 16 0 0 52 0 0 16 psleep DL ?? 25:15.74 [pagedaemon] 0 17 0 0 53 0 0 16 psleep DL ?? 1:54.70 [vmdaemon] 0 18 0 0 76 0 0 16 pgzero DL ?? 0:00.00 [pagezero] 0 19 0 0 -16 0 0 16 psleep DL ?? 0:00.35 [bufdaemon] 0 20 0 0 46 0 0 16 syncer DL ?? 9:22.33 [syncer] 0 21 0 0 46 0 0 16 vlruwt DL ?? 1:38.87 [vnlru] 0 22 0 0 44 0 0 16 sdflus DL ?? 1:31.12 [softdepflush] 0 23 0 0 -16 0 0 16 flowcl DL ?? 0:01.61 [flowcleaner] 80 169 1 0 46 0 132088 10752 nfs D ?? 0:00.05 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 465 1 0 44 -20 175352 10652 nfs D< ?? 0:13.53 /usr/local/sbin/httpd -DNOHTTPACCEPT 0 715 1 0 44 0 3200 32 select Is ?? 0:00.00 /sbin/devd 0 888 1 0 44 0 5864 900 select Ss ?? 3:47.20 /usr/sbin/syslogd -ss 0 914 1 0 44 0 6924 936 select Ss ?? 0:11.19 /usr/sbin/rpcbind 0 1020 1 0 44 0 267976 800 select Ss ?? 0:08.92 /usr/sbin/rpc.statd 0 1028 1 0 76 0 6916 760 rpcsvc Ss ?? 0:00.09 /usr/sbin/rpc.lockd 0 1043 1 0 47 0 15200 2660 nanslp I ?? 6:01.08 /usr/local/bin/monit -c /usr/local/etc/monitrc 122 1052 1 0 76 5 11988 0 wait IWN ?? 0:00.00 zabbix_agentd: main process (zabbix_agentd) 122 1061 1052 0 49 5 11988 1244 nanslp SN ?? 2:49.93 zabbix_agentd: collector [sleeping for 1 seconds] (zabbix_agentd 122 1062 1052 0 49 5 11988 800 select SN ?? 1:44.39 zabbix_agentd: listener [waiting for connection] (zabbix_agentd) 122 1063 1052 0 49 5 11988 800 accept SN ?? 1:43.88 zabbix_agentd: listener [waiting for connection] (zabbix_agentd) 122 1064 1052 0 49 5 11988 808 accept SN ?? 1:35.01 zabbix_agentd: listener [waiting for connection] (zabbix_agentd) 122 1065 1052 0 49 5 11988 1192 nanslp SN ?? 3:00.97 zabbix_agentd: poller [sleeping for 1 second(s)] (zabbix_agentd) 0 1077 1 0 45 0 28064 2316 select S ?? 15:20.13 /usr/local/sbin/snmpd -p /var/run/snmpd.pid 0 1269 1 0 44 0 5992 536 kqread Ss ?? 0:06.57 /usr/local/libexec/postfix/master 125 1282 1269 0 44 0 5996 792 kqread S ?? 0:02.04 qmgr -l -t fifo -u 0 1288 1 0 44 0 43724 3212 select S ?? 0:40.03 /usr/local/bin/python2.6 /usr/local/bin/fail2ban-server -b -s /v 0 1339 1 0 44 0 36936 1824 select Ss ?? 0:00.08 sshd: root@pts/0 (sshd) 0 1340 1 0 76 0 6920 480 nanslp Is ?? 0:05.94 /usr/sbin/cron -s 80 1721 1 0 44 -20 212216 10664 nfs D< ?? 0:06.65 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 1746 1 0 60 -20 208120 10640 nfs D< ?? 0:29.05 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 1747 1 0 44 -20 156920 10692 nfs D< ?? 0:04.32 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 2324 1 0 76 -20 212216 10704 nfs D< ?? 0:01.95 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 3400 1 0 44 -20 163064 10660 nfs D< ?? 0:05.01 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 3401 1 0 44 -20 181496 10688 nfs D< ?? 0:00.80 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 3942 1 0 47 -20 181496 10620 nfs D< ?? 0:29.44 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 3978 1 0 44 -20 179448 10672 nfs D< ?? 0:03.49 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 3979 1 0 44 -20 181496 10648 nfs D< ?? 0:09.13 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 4494 1 0 44 -20 175352 10640 nfs D< ?? 0:25.07 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 4495 1 0 68 -20 183544 10624 nfs D< ?? 0:33.51 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 5110 1 0 76 -20 208120 10688 nfs D< ?? 0:01.41 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 5675 1 0 44 -20 197880 10592 nfs D< ?? 0:42.38 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 6943 1 0 68 -20 208120 10656 nfs D< ?? 0:15.50 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 6944 1 0 44 -20 183544 10644 nfs D< ?? 0:18.46 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 7350 1 0 44 0 154744 15180 nfs D ?? 0:01.61 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 7436 1 0 44 0 177400 15952 nfs D ?? 0:19.34 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 7496 1 0 44 -20 175352 10648 nfs D< ?? 0:12.21 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 7647 1 0 44 -20 152824 10660 nfs D< ?? 0:02.36 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 8127 1 0 47 -20 173304 10616 nfs D< ?? 0:46.18 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 8713 1 0 76 -20 181496 10656 nfs D< ?? 0:10.67 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 8961 1 0 44 -20 154872 10692 nfs D< ?? 0:02.43 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 9219 1 0 54 -20 173304 10612 nfs D< ?? 0:18.19 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 9236 1 0 44 0 171256 15400 nfs D ?? 0:07.30 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 9238 1 0 44 -20 199928 79656 nfs D< ?? 0:01.20 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 9267 1 0 44 -20 199928 79988 nfs D< ?? 0:34.80 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 9273 1 0 76 -20 199928 81060 nfs D< ?? 0:10.44 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 9693 1 0 76 -20 181496 10656 nfs D< ?? 0:17.64 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 9831 1 0 44 -20 173304 10672 nfs D< ?? 0:07.07 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 9871 1 0 44 -20 169208 10684 nfs D< ?? 0:05.17 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 10223 1 0 44 -20 181496 10636 nfs D< ?? 0:25.70 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 10279 1 0 63 0 175352 15724 nfs D ?? 0:16.30 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 10361 1 0 44 -20 191736 10672 nfs D< ?? 0:08.45 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 10655 1 0 59 -20 181496 61448 nfs D< ?? 0:01.71 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 10656 1 0 71 -20 199928 79648 nfs D< ?? 0:05.74 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 10697 1 0 59 -20 199928 79980 nfs D< ?? 0:12.17 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 10994 1 0 62 0 171256 15868 nfs D ?? 0:11.99 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11044 1 0 72 -20 199928 79952 nfs D< ?? 0:27.56 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11078 1 0 59 -20 181496 61992 nfs D< ?? 0:57.98 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11093 1 0 59 -20 199928 79968 nfs D< ?? 0:12.68 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11245 1 0 44 -20 161016 10696 nfs D< ?? 0:01.55 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11274 1 0 64 -20 181496 10684 nfs D< ?? 0:09.52 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11300 1 0 76 -20 208120 10708 nfs D< ?? 0:01.58 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11301 1 0 76 -20 201976 10688 nfs D< ?? 0:08.91 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11303 1 0 56 -20 181496 11256 nfs D< ?? 0:48.15 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11307 1 0 70 -20 201976 11072 nfs D< ?? 0:23.23 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11661 1 0 76 -20 201976 82356 nfs D< ?? 0:38.99 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 11978 1 0 73 -20 201976 11028 nfs D< ?? 0:23.06 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 12423 1 0 65 -20 210168 12328 nfs D< ?? 0:20.99 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 12441 1 0 56 -20 201976 81556 nfs D< ?? 1:23.33 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 12757 1 0 65 0 158968 14772 nfs D ?? 0:00.98 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 13281 1 0 67 -20 201848 11304 nfs D< ?? 0:02.11 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 13855 1 0 53 -20 173304 10680 nfs D< ?? 0:09.40 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 13874 1 0 44 -20 175352 10640 nfs D< ?? 0:15.75 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14008 1 0 58 -20 185720 14796 nfs D< ?? 0:48.99 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14041 1 0 54 -20 179448 10664 nfs D< ?? 0:16.97 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14043 1 0 44 -20 173304 10648 nfs D< ?? 0:15.35 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14082 1 0 44 -20 171256 10672 nfs D< ?? 0:07.50 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14264 1 0 60 0 173304 15328 nfs D ?? 0:04.78 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14327 1 0 44 -20 169208 10656 nfs D< ?? 0:14.71 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14510 1 0 58 0 173304 15800 nfs D ?? 0:17.82 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14701 1 0 64 -20 199928 79836 nfs D< ?? 0:04.98 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14891 1 0 44 -20 200056 80060 nfs D< ?? 0:26.68 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 14930 1 0 46 0 132088 10760 nfs D ?? 0:00.02 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 15102 1 0 44 -20 183544 10648 nfs D< ?? 0:36.67 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 15118 1 0 46 -20 201976 80812 nfs D< ?? 0:42.98 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 15119 1 0 44 -20 165112 10664 nfs D< ?? 0:05.73 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 15275 1 0 76 -20 201976 81900 nfs D< ?? 0:06.18 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 15583 1 0 44 -20 136184 10688 nfs D< ?? 0:00.12 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 15617 1 0 52 0 165112 15532 nfs D ?? 0:03.31 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 15956 1 0 56 -20 181496 10636 nfs D< ?? 0:28.79 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 15957 1 0 44 -20 136184 10700 nfs D< ?? 0:00.06 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 16395 1 0 59 0 175352 15604 nfs D ?? 0:06.21 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 16635 1 0 44 -20 156920 10680 nfs D< ?? 0:00.76 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 16680 1 0 58 -20 212216 10680 nfs D< ?? 0:07.00 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 16937 1 0 52 -20 199928 80224 nfs D< ?? 0:18.39 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17044 1 0 44 -20 181496 10748 nfs D< ?? 0:39.98 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17173 1 0 44 -20 167160 11100 nfs D< ?? 0:11.24 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17250 1 0 45 0 163064 15148 nfs D ?? 0:03.63 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17251 1 0 44 0 169208 15564 nfs D ?? 0:05.59 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17466 1 0 57 -20 199928 79908 nfs D< ?? 0:04.42 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17483 1 0 62 -20 181496 61652 nfs D< ?? 0:12.78 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17489 1 0 58 0 187640 15816 nfs D ?? 0:20.67 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17506 1 0 44 -20 204024 78260 nfs D< ?? 0:03.26 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17656 1 0 46 -20 201976 10804 nfs D< ?? 0:16.42 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 17657 1 0 72 -20 202104 10704 nfs D< ?? 0:19.79 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 18163 1 0 45 0 132088 10756 nfs D ?? 0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 18900 1 0 65 -20 201976 81880 nfs D< ?? 0:12.15 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 18956 1 0 44 -20 181496 61316 nfs D< ?? 0:15.83 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 18986 1 0 65 -20 212216 31548 nfs D< ?? 0:07.71 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19127 1 0 76 -20 201976 10696 nfs D< ?? 0:01.33 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19307 1 0 44 0 185592 15512 nfs D ?? 0:28.67 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19489 1 0 61 -20 181496 33452 nfs D< ?? 0:05.68 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19517 1 0 44 0 163064 13824 nfs D ?? 0:03.39 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19566 1 0 60 -20 199928 52336 nfs D< ?? 0:53.69 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19580 1 0 44 -20 171256 10932 nfs D< ?? 0:08.18 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19593 1 0 76 -20 201976 10888 nfs D< ?? 0:09.98 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19595 1 0 44 -20 167160 10760 nfs D< ?? 0:07.23 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 19596 1 0 51 -20 173304 10700 nfs D< ?? 0:08.23 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 20109 1 0 76 -20 201848 11664 nfs D< ?? 0:01.64 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 20293 1 0 65 -20 199928 79584 nfs D< ?? 0:01.05 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 20570 1 0 76 -20 201976 10680 nfs D< ?? 0:02.63 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 20773 1 0 60 -20 199928 10684 nfs D< ?? 0:02.42 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 20774 1 0 44 -20 167160 10860 nfs D< ?? 0:20.15 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 20913 1 0 44 -20 204024 84968 nfs D< ?? 0:22.02 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 21013 1 0 72 -20 210168 42436 nfs D< ?? 0:16.85 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 21091 1 0 54 -20 201976 81764 nfs D< ?? 0:21.27 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 21551 1 0 45 -20 201976 81896 nfs D< ?? 0:07.26 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 21581 1 0 61 -20 201976 81884 nfs D< ?? 0:32.97 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 21692 1 0 66 -20 199928 80132 nfs D< ?? 0:44.93 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 22467 1 0 48 -20 200056 80068 vmopar D< ?? 0:23.91 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 22473 1 0 44 -20 181496 58076 nfs D< ?? 0:06.67 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 22474 1 0 49 -20 181496 60980 nfs D< ?? 0:06.15 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 22722 1 0 46 -20 161016 10676 nfs D< ?? 0:01.12 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 22751 1 0 54 -20 212216 10896 nfs D< ?? 0:24.68 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 22808 1 0 44 -20 179448 10892 nfs D< ?? 0:26.90 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 22874 1 0 49 -20 181496 10648 nfs D< ?? 0:13.46 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 22949 1 0 48 -20 181368 60744 nfs D< ?? 0:00.73 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 23342 1 0 45 0 132088 10900 nfs D ?? 0:00.03 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 23353 1 0 51 -20 193784 10680 nfs D< ?? 0:05.29 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 23758 1 0 72 -20 201976 81968 nfs D< ?? 0:36.37 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 23779 1 0 53 0 175352 15680 nfs D ?? 0:15.83 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 23906 1 0 44 -20 158968 10676 nfs D< ?? 0:03.77 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 23948 1 0 76 -20 199928 79984 nfs D< ?? 0:25.26 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 23956 1 0 62 0 154872 15204 nfs D ?? 0:05.36 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 23976 1 0 67 -20 181368 10692 nfs D< ?? 0:00.79 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 24713 1 0 44 -20 175352 10680 nfs D< ?? 0:04.47 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 25264 1 0 59 0 163064 15208 nfs D ?? 0:05.28 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 25290 1 0 51 0 163064 15716 nfs D ?? 0:07.25 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 25731 1 0 59 0 163064 15600 nfs D ?? 0:02.72 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 25739 1 0 76 -20 201976 81956 nfs D< ?? 0:05.77 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 25986 1 0 45 -20 181496 10692 nfs D< ?? 0:03.28 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 26873 1 0 44 -20 199928 78284 nfs D< ?? 0:36.38 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 26877 1 0 76 -20 181496 61292 nfs D< ?? 0:05.44 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 27215 1 0 49 -20 181496 10676 nfs D< ?? 0:03.17 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 27435 1 0 63 -20 201976 81992 nfs D< ?? 0:20.09 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 27543 1 0 44 0 154872 14984 vmopar D ?? 0:03.65 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 27562 1 0 60 -20 201848 81788 nfs D< ?? 0:03.04 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 27648 1 0 51 0 163064 13392 nfs D ?? 0:04.92 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 27667 1 0 44 -20 183544 14120 nfs D< ?? 0:18.14 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 27840 1 0 44 0 156920 14824 nfs D ?? 0:06.36 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 28078 1 0 63 -20 181496 15416 nfs D< ?? 0:11.50 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 28090 1 0 46 -20 181496 10672 vmopar D< ?? 0:09.44 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 28155 1 0 50 0 169208 15200 nfs D ?? 0:04.77 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 28162 1 0 54 0 167160 5548 nfs D ?? 0:06.27 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 28625 1 0 46 -20 201976 81752 nfs D< ?? 0:08.19 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 28895 1 0 76 0 163064 14828 nfs D ?? 0:02.60 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 28953 1 0 45 0 132088 12660 nfs D ?? 0:00.02 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 29015 1 0 76 -20 199928 79512 nfs D< ?? 0:11.14 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 29210 1 0 45 -20 181496 10796 nfs D< ?? 0:08.80 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 29429 1 0 57 0 136184 13440 nfs D ?? 0:01.68 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 29473 1 0 47 0 136184 13636 nfs D ?? 0:01.19 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 29688 1 0 59 0 173304 8336 nfs D ?? 0:13.10 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 29879 1 0 58 0 132088 12752 nfs D ?? 0:00.39 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 30280 1 0 76 0 132088 12752 vmopar D ?? 0:00.03 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 30439 1 0 76 0 132088 12820 nfs D ?? 0:00.10 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 30595 1 0 44 -20 201976 81104 nfs D< ?? 0:33.13 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 31237 1 0 44 -20 181368 61032 nfs D< ?? 0:01.83 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 31238 1 0 76 -20 201976 82036 nfs D< ?? 1:15.62 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 31312 1 0 76 -20 181496 21012 nfs D< ?? 0:06.26 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 31438 1 0 64 -20 201976 81716 nfs D< ?? 0:14.72 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 31572 1 0 57 0 169208 15444 nfs D ?? 0:21.23 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 31617 1 0 76 0 175352 11000 nfs D ?? 0:22.85 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 31938 1 0 46 -20 181496 61108 nfs D< ?? 0:03.33 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 32070 1 0 57 0 173304 15600 nfs D ?? 0:15.80 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 32071 1 0 65 -20 201976 81628 nfs D< ?? 0:27.76 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 32451 1 0 59 0 158968 14864 nfs D ?? 0:07.64 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 32572 1 0 44 0 161016 14104 nfs D ?? 0:06.11 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 32980 1 0 46 -20 181496 54796 nfs D< ?? 0:08.39 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 33042 1 0 61 0 169208 15384 nfs D ?? 0:03.58 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 33333 1 0 76 -20 199928 75412 nfs D< ?? 0:13.56 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 33353 1 0 53 -20 199928 62204 nfs D< ?? 0:40.47 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 33682 1 0 76 0 136184 13664 nfs D ?? 0:03.65 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 33794 1 0 55 0 165112 14860 nfs D ?? 0:01.62 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 34043 1 0 76 0 136184 9184 nfs D ?? 0:06.89 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 34373 1 0 51 0 132088 12772 nfs D ?? 0:00.03 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 34833 1 0 55 0 132088 12676 nfs D ?? 0:00.88 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 35460 1 0 49 0 132088 12900 nfs D ?? 0:00.20 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 35508 1 0 57 0 136184 12836 nfs D ?? 0:00.70 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 35573 1 0 49 -20 189688 59980 nfs D< ?? 0:19.03 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 35646 1 0 64 0 136184 14216 nfs D ?? 0:03.65 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 35921 1 0 51 0 132088 12828 nfs D ?? 0:00.17 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 36368 1 0 63 0 181496 16032 nfs D ?? 0:14.35 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 36391 1 0 57 -20 181496 60416 nfs D< ?? 0:12.79 /usr/local/sbin/httpd -DNOHTTPACCEPT 0 36410 1 0 76 0 25004 3212 select Ss ?? 0:05.12 /usr/sbin/sshd 80 36431 1 0 44 0 163064 15404 nfs D ?? 0:22.64 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 36497 1 0 70 0 136184 13352 nfs D ?? 0:02.78 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 37113 1 0 69 -20 199928 79680 nfs D< ?? 0:18.34 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 37358 1 0 44 0 132088 12680 nfs D ?? 0:00.02 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 38182 1 0 76 0 185592 14604 nfs D ?? 0:51.43 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 38907 1 0 57 -20 199928 79748 nfs D< ?? 0:12.44 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 40607 1 0 57 -20 181496 50724 nfs D< ?? 0:09.08 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 41682 1 0 64 0 175352 14088 nfs D ?? 0:07.51 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 41705 1 0 76 0 181496 14436 nfs D ?? 0:27.59 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 43380 1 0 44 -20 181496 33816 nfs D< ?? 0:02.26 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 43381 1 0 45 0 154872 13536 nfs D ?? 0:01.87 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 43671 1 0 46 0 132088 12600 nfs D ?? 0:00.02 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 43794 1 0 65 0 177400 13604 nfs D ?? 0:11.35 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 44522 1 0 57 0 169208 14128 nfs D ?? 0:06.29 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 45082 1 0 45 0 181496 15528 nfs D ?? 0:24.88 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 45421 1 0 58 -20 199928 55868 nfs D< ?? 0:02.67 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 46241 1 0 70 -20 201976 59196 nfs D< ?? 0:10.51 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 46243 1 0 56 -20 199928 70824 nfs D< ?? 0:11.06 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 46604 1 0 71 -20 181496 49116 nfs D< ?? 0:02.93 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 46649 1 0 44 -20 181496 20004 nfs D< ?? 0:07.93 /usr/local/sbin/httpd -DNOHTTPACCEPT 0 46791 1 0 44 0 36936 1652 select Ss ?? 0:29.76 sshd: root@pts/1 (sshd) 80 47119 1 0 71 -20 201976 57668 nfs D< ?? 0:48.74 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 47124 1 0 44 -20 185720 54048 nfs D< ?? 0:42.99 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 47542 1 0 55 -20 201976 71656 nfs D< ?? 0:44.89 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 48148 1 0 71 -20 201976 42480 nfs D< ?? 1:24.99 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 48657 1 0 54 0 173304 15340 nfs D ?? 0:03.34 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 49373 1 0 44 -20 204024 68580 nfs D< ?? 0:07.56 /usr/local/sbin/httpd -DNOHTTPACCEPT 0 49523 1 0 44 0 36936 1636 select Is ?? 0:22.79 sshd: root@pts/2 (sshd) 80 49624 1 0 44 -20 201848 81272 nfs D< ?? 0:02.94 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 49641 1 0 45 0 156920 14616 nfs D ?? 0:08.58 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 49726 1 0 53 0 165112 15344 nfs D ?? 0:10.71 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 49866 1 0 44 0 161016 8728 nfs D ?? 0:05.48 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 49880 1 0 63 0 169208 13820 nfs D ?? 0:51.78 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 51064 1 0 49 -20 199928 73000 nfs D< ?? 0:08.01 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 52192 1 0 76 -20 191736 25312 nfs D< ?? 0:22.50 /usr/local/sbin/httpd -DNOHTTPACCEPT 80 52481 1 0 44 -20 207208 10492 vmopar D