From owner-freebsd-stable@FreeBSD.ORG Wed Nov 23 03:48:09 2005 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E93D116A41F for ; Wed, 23 Nov 2005 03:48:09 +0000 (GMT) (envelope-from gcr+freebsd-stable@tharned.org) Received: from nc8000.tharned.org (rrcs-24-56-87-26.ma.biz.rr.com [24.56.87.26]) by mx1.FreeBSD.org (Postfix) with ESMTP id E057243D55 for ; Wed, 23 Nov 2005 03:48:06 +0000 (GMT) (envelope-from gcr+freebsd-stable@tharned.org) Received: from nc8000.tharned.org (localhost [127.0.0.1]) by nc8000.tharned.org (8.13.4/8.13.4) with ESMTP id jAN3m5t0013772; Tue, 22 Nov 2005 21:48:05 -0600 (CST) (envelope-from gcr+freebsd-stable@tharned.org) Received: from localhost (gcr@localhost) by nc8000.tharned.org (8.13.4/8.13.4/Submit) with ESMTP id jAN3m5td013769; Tue, 22 Nov 2005 21:48:05 -0600 (CST) (envelope-from gcr+freebsd-stable@tharned.org) Date: Tue, 22 Nov 2005 21:48:04 -0600 (CST) From: Greg Rivers Sender: gcr@tharned.org To: Kris Kennaway In-Reply-To: <20051122043952.GA14168@xor.obsecurity.org> Message-ID: <20051122211507.P32523@nc8000.tharned.org> References: <20051121164139.T48994@w10.sac.fedex.com> <20051122021224.GA12402@xor.obsecurity.org> <20051121205535.W32523@nc8000.tharned.org> <20051122043952.GA14168@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-stable@freebsd.org, Kirk McKusick Subject: Re: Recurring problem: processes block accessing UFS file system X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Nov 2005 03:48:10 -0000 On Mon, 21 Nov 2005, Kris Kennaway wrote: > It may not be the same problem. You should also try to obtain a trace > when snapshots are not implicated. > Agreed. I'll do so at the first opportunity. > 'show lockedvnods' is very important for diagnosing filesystem > deadlocks, and 'alltrace' is the easiest way to obtain stack traces for > the resulting processes that are holding locks. You'll want to 'set > $lines=0' and capture the output from the serial console to a file. > Thanks Kris, these are exactly the clues I needed. Since the deadlock during a snapshot is fairly easy to reproduce, I did so and collected this information below. "alltrace" didn't work as I expected (didn't produce a trace), so I traced each pid associated with a locked vnode separately. -- Greg # sync;sync; date; uname -a Tue Nov 22 20:31:07 CST 2005 FreeBSD mx17.sac.fedex.com 6.0-STABLE FreeBSD 6.0-STABLE #0: Mon Nov 21 23:46:18 CST 2005 root@mx17.sac.fedex.com:/usr/obj/usr/src/sys/MX i386 # KDB: enter: Line break on console [thread pid 12 tid 100004 ] Stopped at kdb_enter+0x30: leave db> set $lines=0 db> ps pid proc uid ppid pgrp flag stat wmesg wchan cmd 99774 ced88000 0 673 673 0000100 new [INACTIVE] sendmail 99773 cecc7624 100 673 673 0000100 [SLPQ connec 0xced3c302][SLP] sendmail 99772 cecc7830 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 99771 cecc7a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99770 cecc7c48 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 99769 ced8820c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99768 ced88418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99767 ced44a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99766 ced88624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99765 ceb5aa3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99764 ceb5a830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99763 ced45c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99762 cec30624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99761 c8ad820c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99760 ce685000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99759 cecc8000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99758 cecc820c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99757 cecc8418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99756 c8941624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99755 cecc8830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99750 ced44c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99743 ced45000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99742 ced4520c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99741 ced45418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99740 ced45624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99739 ced45830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99738 ced45a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99737 cec3020c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99736 cec30418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99733 cecc9624 11521 99729 99726 0004000 [SLPQ piperd 0xc8e98660][SLP] awk 99731 cecc9830 11521 99729 99726 0004100 [SLPQ ufs 0xc8991d18][SLP] sendmail 99730 c896a830 11521 99728 99728 0004000 [SLPQ nanslp 0xc06596ac][SLP] perl5.8.7 99729 cec30830 11521 99726 99726 0004000 [SLPQ piperd 0xc8e47cc0][SLP] ksh 99728 cecc9a3c 11521 99725 99728 0004000 [SLPQ wait 0xcecc9a3c][SLP] sh 99726 cec30a3c 11521 99723 99726 0004000 [SLPQ wait 0xcec30a3c][SLP] sh 99725 c890920c 0 689 689 0000000 [SLPQ piperd 0xcaae0660][SLP] cron 99723 ce68d000 0 689 689 0000000 [SLPQ piperd 0xc89604c8][SLP] cron 99722 cec30c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99721 cec31000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99720 cecc9c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99719 ced1a000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99718 ced1a20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99717 ced1a418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99716 ced1a624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99715 c8e91624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99714 ced1a830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99713 ced1aa3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99712 ced1ac48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99711 cec3120c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99710 ced1e000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99709 ced1e20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99708 ced1e418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99707 ced1e624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99706 ced1e830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99705 ceb92c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99704 cebb9000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99703 cebb920c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99702 cec31418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99701 cec31624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99700 cec31830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99699 cec31a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99698 cec31c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99697 cecc9000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99696 cecc920c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99695 cecc9418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99694 cec37418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99693 cec37624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99692 cec37830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99691 cec37a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99690 cec37c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99689 cec69000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99688 cec6920c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99687 cec69418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99686 cec69624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99685 cec69830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99684 cec69a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99683 cec69c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99682 cec6e000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99681 cec6e20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99680 cec6e418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99679 cec6e624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99678 cec6e830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99677 cec6ea3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99676 cec34000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99675 cec6ec48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99674 cec6f000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99673 cec6f20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99672 cec6f418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99671 cec6f624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99670 cec6f830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99669 cec6fa3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99668 cec6fc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99667 cec70000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99666 cec7020c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99665 cec70418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99664 cec70624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99663 cec70830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99662 cec70a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99661 cebbb20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99660 cebbb418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99659 cebbb624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99657 cebbb830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99656 cebbba3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99655 cebbbc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99654 cec3420c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99653 cebdd000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99652 cebdd20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99651 cebdd418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99650 cec34418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99649 cebdd624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99648 cebdd830 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 99647 cebdda3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99646 cebddc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99645 cebdf000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99644 cebdf20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99643 cebdf418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99642 cebdf624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99641 cebdf830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99640 cebdfa3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99639 cebdfc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99638 cebe0000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99637 cebe020c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99636 cebe0418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99635 cec34624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99634 cec34830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99633 cebb9418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99632 cec34a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99631 cebb9624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99630 cebb9830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99629 cebb9a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99628 cebb9c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99627 cebba000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99626 cec34c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99625 cec35000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99624 cec3520c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99623 cec35418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99622 cec35624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99621 cec35830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99620 cec35a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99619 cec35c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99618 cec37000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99617 cebe0624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99616 cebe0830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99615 cebba20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99614 cec3720c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99613 cebe0a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99612 cebe0c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99611 cebe1000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99610 cebe120c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99609 ce68a000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99608 cebe1418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99607 cebe1624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99606 ceaca624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99605 cebe1830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99604 ceb5a000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99603 ceb5a20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99602 ceb5a418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99601 ceaca830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99600 ceb5a624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99597 ceb5ac48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99596 ceb5e000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99595 ceb5e20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99594 ceb5e418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99593 ceb5e624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99592 ceb5e830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99591 ceb5ea3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99590 ceb5ec48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99587 ceb5f000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99586 ceb5f20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99585 ceacaa3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99584 ceacac48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99583 ceb5f418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99582 ceb40000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99581 ceb5f624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99580 ceb5f830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99579 ceb5fa3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99578 cebba418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99577 ceb5fc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99576 cebba624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99575 ceb60000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99574 ceb6020c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99573 ceb60418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99572 ceb4020c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99571 cebba830 11521 99568 99566 0004000 [SLPQ piperd 0xc8e48198][SLP] awk 99570 cebbaa3c 11521 99568 99566 0004100 [SLPQ ufs 0xc8991d18][SLP] sendmail 99569 ceb60624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99568 cebbac48 11521 99566 99566 0004000 [SLPQ piperd 0xc8960660][SLP] ksh 99566 cebbb000 11521 99564 99566 0004000 [SLPQ wait 0xcebbb000][SLP] sh 99564 ceb40418 0 689 689 0000000 [SLPQ piperd 0xce752330][SLP] cron 99562 ceb40830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99561 ceb40a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99560 ceb40c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99559 ceb43000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99558 ceb4320c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99557 ceb43418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99556 ceb43624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99555 ceb60830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99554 ceb43830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99553 ceb61830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99552 ceb61a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99551 ceb61c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99550 ceb8e000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99549 ceb60a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99548 ceb8e20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99547 ce68520c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99546 ceb43a3c 25 677 677 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99545 ceb43c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99544 ceb8e418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99543 ceb8e624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99542 ceb8e830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99541 ceb8ea3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99540 ceb8ec48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99539 ceb44000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99538 ceb92000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99537 ceb9220c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99536 ceb60c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99535 ceb92418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99534 ceb92624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99532 ceb92830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99531 ceb92a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99530 ceb4420c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99529 ceb44418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99528 ceb44624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99527 ceb44830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99526 ceb44a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99525 ceb61000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99524 ceb6120c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99523 ceb61418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99522 ceb61624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99521 c901b624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99520 ce681418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99519 ccb1b000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99518 ceb44c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99517 ceb46000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99516 ceb4620c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99515 c8ad020c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99514 c8969830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99513 ce681a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99512 ceb46418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99511 c8e90a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99510 ceb46624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99509 cc3f9a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99508 ceb46830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99507 cea68830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99506 ceb46a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99505 ceb46c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99504 cea8fc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99503 ceac2000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99502 ceac220c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99501 ceac2418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99500 ceac2624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99499 ceac2830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99498 c901dc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99497 c9022418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99496 c8e91000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99495 c901da3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99494 ce689000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99493 c894720c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99492 ceac2a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99491 ceac2c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99490 ceac4000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99489 ceac420c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99488 ceac4418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99487 ceac4624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99486 cea68a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99485 cea68c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99484 ceac4830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99483 ceac4a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99482 ceac4c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99481 ceac7000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99480 ceac720c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99479 ceac7418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99478 ceac7624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99477 cea8e000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99476 ceac7830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99475 ceac7a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99474 ceac7c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99473 ceac8000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99472 ceac820c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99471 ceac8418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99470 ceac8624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99469 ceac8830 11521 99467 99466 0004000 [SLPQ piperd 0xc895f660][SLP] awk 99468 ceac8a3c 11521 99467 99466 0004100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99467 ce68d20c 11521 99466 99466 0004000 [SLPQ piperd 0xc8971cc0][SLP] ksh 99466 ce689c48 11521 99465 99466 0004000 [SLPQ wait 0xce689c48][SLP] sh 99465 ceac8c48 0 689 689 0000000 [SLPQ piperd 0xc8cdf198][SLP] cron 99464 ceaca000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99463 ceaca20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99462 cea8e20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99461 cea8e418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99460 cea8e624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99459 ceaca418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99458 ce682830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99457 cea8e830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99456 cea8ea3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99455 cea8ec48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99454 cea61624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99453 cea61000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99452 cea6120c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99449 cea61830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99448 cea61a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99447 cea8f000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99446 cea61c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99445 cea62000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99444 cea6220c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99443 cea62418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99442 cea8f20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99441 cea8f418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99440 cea8f624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99439 cea62624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99438 cea8f830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99437 cea8fa3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99436 cea62830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99435 ce68da3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99434 cea62a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99433 cea62c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99432 ce68dc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99431 ce9c7000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99430 ce9c720c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99429 cea65000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99428 ce9c7418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99427 ce9c7624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99426 cea6520c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99425 ce9c7830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99424 cea65418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99423 cea65624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99422 cc3f4a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99421 ce68920c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99420 cea65830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99419 cea65a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99418 cea65c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99417 cc3f4000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99416 cea66000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99415 cea6620c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99414 ce9c7a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99413 cea66418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99412 cea66624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99411 cea66830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99410 cea66a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99409 cea66c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99408 cea68000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99407 cea6820c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99406 ce68aa3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99405 cea68418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99404 cea68624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99403 c8e9220c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99402 cc3f9c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99401 ce682c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99400 ccb1ba3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99399 c8ad6c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99398 c894120c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99397 ce685830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99396 ce682a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99395 c86ed830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99394 ccb17418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99393 ce9c7c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99392 ce685a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99391 ccb20c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99390 ce9cb000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99389 ce68ac48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99388 c9a59a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99387 c9a59418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99386 c901b418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99384 ce9cb20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99383 ce9cb418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99382 ce9cb624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99381 c8ad8a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99380 ce9cb830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99379 ce9cba3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99378 c894620c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99377 ce688c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99376 ccb2220c 100 673 673 0000100 [SLPQ ufs 0xc8d30498][SLP] sendmail 99375 ccb20a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99374 c901bc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99373 c8ad620c 100 673 673 0000100 [SLPQ ufs 0xc8dc1af8][SLP] sendmail 99372 c8ad0418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99371 c8e92418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99370 c8ad0830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99369 c8e59c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99368 ccb16418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99367 c9022c48 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 99366 ce681830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99365 ce68420c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99364 ccb21a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99363 c8e8ca3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99362 c8e58c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99361 ce682624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99360 c9022a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99359 c896a000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99358 ce67b20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99356 ce9cbc48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99355 ce67bc48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99354 ce688000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99353 c9a5a20c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99352 c901d000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99351 c8947000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99350 cc3fa20c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99349 ce684c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99348 ce68120c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99347 c9021830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99345 ccb21624 11008 4703 99345 0004002 [SLPQ ttyin 0xcc8b3810][SLP] ksh 99344 cc3f8624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99343 c8904830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99342 c9a5a624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99341 c901ba3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99340 ccb16830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99339 c890820c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99338 c8969418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99337 c8e8c624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99334 c901b20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99333 ccb22624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99332 c8e8c000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99331 ce689a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99330 cc3f8c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99329 ccb22000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99328 c9a5220c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99327 c8e8c20c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99326 ccb1620c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99325 c8e59830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99324 ccb20624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99323 c902220c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99322 ce68a830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99321 ce684a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99320 ccb16000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99319 c8ad0000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99318 c896a20c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99317 c8e90418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99316 c9021624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99315 c8969000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99308 c8e8cc48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99307 ce67ba3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99306 c8946418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99305 c8e9020c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99304 ce67b830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99303 c9a59000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99302 c9021000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99301 ce67b000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99299 c896aa3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99297 ce68220c 11521 99294 99290 0004000 [SLPQ piperd 0xc895f198][SLP] awk 99296 ce689830 11521 99294 99290 0004100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99294 ccb16a3c 11521 99290 99290 0004000 [SLPQ piperd 0xce423330][SLP] ksh 99290 c901d830 11521 99286 99290 0004000 [SLPQ wait 0xc901d830][SLP] sh 99286 ce685624 0 689 689 0000000 [SLPQ piperd 0xc8960198][SLP] cron 99285 c9a5920c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99284 c86e9000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99283 c8941830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99282 c8947624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99281 ce68a20c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99280 ce67b418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99279 c9a52000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99278 c86e920c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99277 cc3f4418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99276 c896920c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99275 c890420c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99274 c8e59418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99273 c9a52624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99272 c9021418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99271 ccb22c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99270 ccb1b418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99269 cc3f420c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99268 ce682000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99267 ccb1b624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99266 ce689624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99265 c8908c48 100 673 673 0000100 [SLPQ ufs 0xc8d30278][SLP] sendmail 99263 c9a58000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99262 ce68d418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99261 ce685c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99260 c8908000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99259 ccb1b830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99258 c8947c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99257 c8946c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99256 c9a5a418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99255 ccb20830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99254 c8e9120c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99253 c8e92c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99252 ccb22418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99251 ce681000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99250 c9a52a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99249 c8965418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99248 ce684000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99247 c86ed624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99246 ce681c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99245 ce688830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99244 c8947418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99243 ce68a624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99242 ccb22830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99241 c8ad8830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99240 ccb20000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99239 c901d20c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99238 c86edc48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99237 c9021c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99236 c8e58624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99235 c8e90c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99234 c9a58418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99233 ccb17a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99232 c8e52830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99231 c8e90830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99230 c8904c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99229 c9a52830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99227 c8e8c418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99226 c9022624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99225 c8e92a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99224 ce68a418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99223 c8e90624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99222 ce684830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99221 ccb17000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99220 c8e5920c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99219 c8947a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99218 ccb16c48 100 1 99218 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99215 c8e59000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99214 c9a59830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99213 ce684418 100 1 99213 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99211 c9a58a3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99210 c9a5a000 100 1 99210 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99206 c8904000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99204 ccb1b20c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99200 ce684624 100 1 99200 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 99183 cc3fa418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99182 c901d624 100 1 99182 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99176 c8e92624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99165 c9a52418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99117 c9a5a830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99099 c901d418 100 1 99099 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99086 c896520c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 99078 cc3fac48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 98956 c8e52624 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 98921 ce68d830 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 98854 ce68820c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 98665 c9a52c48 0 98282 98665 0004002 [SLPQ suspfs 0xc86d706c][SLP] ls 98639 c8e91418 11008 98637 97958 0004102 [SLPQ ufs 0xc8991d18][SLP] mksnap_ffs 98637 c8e5820c 11008 97958 97958 0004002 [SLPQ wait 0xc8e5820c][SLP] sh 98282 c8e58418 0 98261 98282 0004002 [SLPQ pause 0xc8e5844c][SLP] ksh 98261 c8965624 11008 97022 98261 0004102 [SLPQ wait 0xc8965624][SLP] su 97958 c8941000 11008 4705 97958 0004002 [SLPQ wait 0xc8941000][SLP] dump 97439 ce68d624 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 97310 c8ad0c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 97158 c896a418 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 97022 c8e91a3c 11008 4703 97022 0004002 [SLPQ pause 0xc8e91a70][SLP] ksh 97004 cc3f8830 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 96699 ccb21418 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 96109 ccb21830 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 95970 c8904418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 95957 cc3f4830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 95932 c8ad6418 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 95907 cc3faa3c 100 673 673 0000100 [SLPQ sbwait 0xc8f5479c][SLP] sendmail 94322 cc3f9418 100 673 673 0000100 [SLPQ sbwait 0xcb4800a8][SLP] sendmail 93567 c9a58c48 100 673 673 0000100 [SLPQ pause 0xc9a58c7c][SLP] sendmail 93563 cc3f9000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 91307 c8e58a3c 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 89069 c9022000 100 673 673 0000100 [SLPQ sbwait 0xce675d2c][SLP] sendmail 87575 ccb17624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 85547 ccb21000 100 673 673 0000100 [SLPQ sbwait 0xce7cb0a8][SLP] sendmail 85524 c8946624 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 85431 c8908624 100 673 673 0000100 [SLPQ sbwait 0xcc809a64][SLP] sendmail 85005 c8909624 100 673 673 0000100 [SLPQ sbwait 0xcc809638][SLP] sendmail 83829 c8e5220c 100 673 673 0000100 [SLPQ sbwait 0xce67420c][SLP] sendmail 83436 c8e59624 100 673 673 0000100 [SLPQ sbwait 0xc94e70a8][SLP] sendmail 83282 cc3f8a3c 100 673 673 0000100 [SLPQ sbwait 0xcc40c370][SLP] sendmail 81449 ccb2020c 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 76255 c8ad8000 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 74272 ce688a3c 100 673 673 0000100 [SLPQ sbwait 0xccbb10a8][SLP] sendmail 73313 ce67b624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 71860 c9a5ac48 100 673 673 0000100 [SLPQ sbwait 0xce644e90][SLP] sendmail 69304 c9a58624 100 673 673 0000100 [SLPQ sbwait 0xce64ce90][SLP] sendmail 61830 ccb1bc48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 59550 cc3f8418 100 673 673 0000100 [SLPQ sbwait 0xce6e54d4][SLP] sendmail 56973 cc3f4c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 55659 cc3fa000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 54198 ce688418 100 673 673 0000100 [SLPQ sbwait 0xc8f540a8][SLP] sendmail 53424 c896ac48 100 673 673 0000100 [SLPQ sbwait 0xccbb1638][SLP] sendmail 50232 c8908a3c 100 673 673 0000100 [SLPQ sbwait 0xce64c900][SLP] sendmail 46440 ccb1720c 100 673 673 0000100 [SLPQ pause 0xccb17240][SLP] sendmail 45756 ccb22a3c 100 673 673 0000100 [SLPQ sbwait 0xce7abbc8][SLP] sendmail 44967 c8965a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 42202 cc3f4624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 41899 c9a59c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 35265 ccb21c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 22008 c9a5aa3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 21146 c8909830 100 673 673 0000100 [SLPQ pause 0xc8909864][SLP] sendmail 4993 cc3f8000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 81603 cc3fa624 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 70166 cc3f820c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 64399 c86eda3c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 63092 c8ad6000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 62406 c8965000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 60658 c8946830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 59668 c8e58000 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 58412 c9a58830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 57591 c8965830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 55626 ccb16624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 54177 ccb17830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 49657 ccb17c48 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 35742 ccb2120c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 20979 cc3f9830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 6030 c8909418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 91041 ce681624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 76748 ce689418 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 62737 c8e91830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 52019 c901b830 100 673 673 0000100 [SLPQ select 0xc0660244][SLP] sendmail 41885 c8e90000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 33339 c8965c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 25301 cc3fa830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 15180 c8946000 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 5670 c8e52000 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 93621 c8e52c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 76951 cc3f9624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 68197 c896a624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 61840 c9022830 100 61839 61839 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 61839 c8941c48 100 1 61839 0000100 [SLPQ wait 0xc8941c48][SLP] sendmail 59681 c8ad0624 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 51464 c9a5820c 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 40069 c8e8c830 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 30519 c8909c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 22495 c8e92830 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 13893 c8909a3c 100 673 673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail 4772 c9a59624 0 4753 4772 0025002 stop[SUSP] ksh 4753 c8e92000 11008 4705 4753 0025102 stop[SUSP] su 4705 c902120c 11008 4703 4705 0004002 [SLPQ pause 0xc9021240][SLP] ksh 4703 c8908830 11008 4700 4703 0000100 [SLPQ select 0xc0660244][SLP] screen 4700 c9021a3c 11008 4609 4700 0004102 [SLPQ pause 0xc9021a70][SLP] screen 4609 c8ad6a3c 11008 4600 4609 0004002 [SLPQ pause 0xc8ad6a70][SLP] ksh 4600 c8904624 11008 4588 4588 0000100 [SLPQ select 0xc0660244][SLP] sshd 4588 c901b000 0 668 4588 0004100 [SLPQ sbwait 0xc94ef638][SLP] sshd 1860 c8e52418 0 1857 1860 0004002 [SLPQ ttyin 0xc8714810][SLP] ksh 1857 c8e58830 0 1 1857 0004102 [SLPQ wait 0xc8e58830][SLP] login 1856 c8941a3c 0 1 1856 0004002 [SLPQ ttyin 0xc870a010][SLP] getty 1855 c8e59a3c 0 1 1855 0004002 [SLPQ ttyin 0xc8714010][SLP] getty 1854 c8969a3c 0 1 1854 0004002 [SLPQ ttyin 0xc8716810][SLP] getty 1853 c8e52a3c 0 1 1853 0004002 [SLPQ ttyin 0xc8714410][SLP] getty 1844 c8e91c48 0 1 1844 0000000 [SLPQ select 0xc0660244][SLP] inetd 689 c8ad8624 0 1 689 0000000 [SLPQ nanslp 0xc06596ac][SLP] cron 677 c8ad8418 25 1 677 0000100 [SLPQ pause 0xc8ad844c][SLP] sendmail 676 c8969c48 100 673 673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail 673 c8941418 0 1 673 0000100 [CPU 0] sendmail 668 c86ed418 0 1 668 0000100 [SLPQ select 0xc0660244][SLP] sshd 650 c8ad0a3c 0 1 650 0000000 [SLPQ select 0xc0660244][SLP] ntpd 558 c8969624 53 1 558 0000100 [SLPQ select 0xc0660244][SLP] named 493 c8908418 0 1 493 0000000 [SLPQ select 0xc0660244][SLP] syslogd 460 c8947830 0 1 460 0000000 [SLPQ select 0xc0660244][SLP] devd 223 c8ad6830 0 0 0 0000204 [SLPQ mdwait 0xc8857000][SLP] md1 214 c8904a3c 0 0 0 0000204 [SLPQ mdwait 0xc892e000][SLP] md0 155 c8909000 0 0 0 0000204 [SLPQ m:w2 0xc88b2300][SLP] g_mirror gm1 154 c86e9418 0 0 0 0000204 [SLPQ m:w2 0xc88eda00][SLP] g_mirror gm0 153 c86e9624 0 0 0 0000204 [SLPQ m:w2 0xc8683200][SLP] g_mirror gm2 152 c86e9830 0 0 0 0000204 [SLPQ - 0xe8c56cfc][SLP] schedcpu 151 c86e9a3c 0 0 0 0000204 [SLPQ vlruwt 0xc86e9a3c][SLP] vnlru 150 c86e9c48 0 0 0 0000204 [SLPQ syncer 0xc065941c][SLP] syncer 149 c86ec000 0 0 0 0000204 [SLPQ psleep 0xc066078c][SLP] bufdaemon 148 c86ec20c 0 0 0 0000204 [SLPQ pollid 0xc0658a60][SLP] idlepoll 147 c86ec418 0 0 0 000020c [SLPQ pgzero 0xc0663f24][SLP] pagezero 146 c86ec624 0 0 0 0000204 [SLPQ psleep 0xc0663a74][SLP] vmdaemon 145 c86ec830 0 0 0 0000204 [SLPQ psleep 0xc0663a30][SLP] pagedaemon 144 c86eca3c 0 0 0 0000204 [IWAIT] swi0: sio 143 c86ecc48 0 0 0 0000204 [SLPQ cooling 0xc86f58d4][SLP] acpi_cooling0 142 c86ed000 0 0 0 0000204 [SLPQ tzpoll 0xc07924f4][SLP] acpi_thermal 141 c86ed20c 0 0 0 0000204 [SLPQ idle 0xc86abc74][SLP] ciss_notify0 9 c8610a3c 0 0 0 0000204 [SLPQ - 0xc8634500][SLP] kqueue taskq 140 c8610c48 0 0 0 0000204 [IWAIT] swi5:+ 8 c8623000 0 0 0 0000204 [SLPQ - 0xc8634680][SLP] acpi_task2 7 c862320c 0 0 0 0000204 [SLPQ - 0xc8634680][SLP] acpi_task1 6 c8623418 0 0 0 0000204 [SLPQ - 0xc8634680][SLP] acpi_task0 5 c8623624 0 0 0 0000204 [SLPQ - 0xc8634700][SLP] thread taskq 139 c8623830 0 0 0 0000204 [IWAIT] swi6:+ 138 c8623a3c 0 0 0 0000204 [IWAIT] swi6: task queue 137 c8623c48 0 0 0 0000204 [IWAIT] swi2: cambio 136 c8627000 0 0 0 0000204 [SLPQ - 0xc0655de0][SLP] yarrow 4 c862720c 0 0 0 0000204 [SLPQ - 0xc0656748][SLP] g_down 3 c8627418 0 0 0 0000204 [SLPQ - 0xc0656744][SLP] g_up 2 c8627624 0 0 0 0000204 [SLPQ - 0xc065673c][SLP] g_event 135 c8627830 0 0 0 0000204 [IWAIT] swi3: vm 134 c8627a3c 0 0 0 000020c [IWAIT] swi4: clock sio 133 c8627c48 0 0 0 0000204 [IWAIT] swi1: net 132 c85f9624 0 0 0 0000204 [IWAIT] irq119: 131 c85f9830 0 0 0 0000204 [IWAIT] irq118: 130 c85f9a3c 0 0 0 0000204 [IWAIT] irq117: 129 c85f9c48 0 0 0 0000204 [IWAIT] irq116: 128 c860c000 0 0 0 0000204 [IWAIT] irq115: 127 c860c20c 0 0 0 0000204 [IWAIT] irq114: 126 c860c418 0 0 0 0000204 [IWAIT] irq113: 125 c860c624 0 0 0 0000204 [IWAIT] irq112: 124 c860c830 0 0 0 0000204 [IWAIT] irq111: 123 c860ca3c 0 0 0 0000204 [IWAIT] irq110: 122 c860cc48 0 0 0 0000204 [IWAIT] irq109: 121 c8610000 0 0 0 0000204 [IWAIT] irq108: 120 c861020c 0 0 0 0000204 [IWAIT] irq107: 119 c8610418 0 0 0 0000204 [IWAIT] irq106: 118 c8610624 0 0 0 0000204 [IWAIT] irq105: 117 c8610830 0 0 0 0000204 [IWAIT] irq104: 116 c85e520c 0 0 0 0000204 [IWAIT] irq103: 115 c85e5418 0 0 0 0000204 [IWAIT] irq102: 114 c85e5624 0 0 0 0000204 [IWAIT] irq101: 113 c85e5830 0 0 0 0000204 [IWAIT] irq100: 112 c85e5a3c 0 0 0 0000204 [IWAIT] irq99: 111 c85e5c48 0 0 0 0000204 [IWAIT] irq98: em1 110 c85f8000 0 0 0 0000204 [IWAIT] irq97: em0 109 c85f820c 0 0 0 0000204 [IWAIT] irq96: 108 c85f8418 0 0 0 0000204 [IWAIT] irq95: 107 c85f8624 0 0 0 0000204 [IWAIT] irq94: 106 c85f8830 0 0 0 0000204 [IWAIT] irq93: 105 c85f8a3c 0 0 0 0000204 [IWAIT] irq92: 104 c85f8c48 0 0 0 0000204 [IWAIT] irq91: 103 c85f9000 0 0 0 0000204 [IWAIT] irq90: 102 c85f920c 0 0 0 0000204 [IWAIT] irq89: 101 c85f9418 0 0 0 0000204 [IWAIT] irq88: 100 c85e1000 0 0 0 0000204 [IWAIT] irq87: 99 c85e120c 0 0 0 0000204 [IWAIT] irq86: 98 c85e1418 0 0 0 0000204 [IWAIT] irq85: 97 c85e1624 0 0 0 0000204 [IWAIT] irq84: 96 c85e1830 0 0 0 0000204 [IWAIT] irq83: 95 c85e1a3c 0 0 0 0000204 [IWAIT] irq82: 94 c85e1c48 0 0 0 0000204 [IWAIT] irq81: 93 c85e4000 0 0 0 0000204 [IWAIT] irq80: 92 c85e420c 0 0 0 0000204 [IWAIT] irq79: 91 c85e4418 0 0 0 0000204 [IWAIT] irq78: 90 c85e4624 0 0 0 0000204 [IWAIT] irq77: 89 c85e4830 0 0 0 0000204 [IWAIT] irq76: 88 c85e4a3c 0 0 0 0000204 [IWAIT] irq75: 87 c85e4c48 0 0 0 0000204 [IWAIT] irq74: 86 c85e5000 0 0 0 0000204 [IWAIT] irq73: 85 c85cd000 0 0 0 0000204 [IWAIT] irq72: 84 c85cd20c 0 0 0 0000204 [IWAIT] irq71: 83 c85cd418 0 0 0 0000204 [IWAIT] irq70: 82 c85cd624 0 0 0 0000204 [IWAIT] irq69: 81 c85cd830 0 0 0 0000204 [IWAIT] irq68: 80 c85cda3c 0 0 0 0000204 [IWAIT] irq67: 79 c85cdc48 0 0 0 0000204 [IWAIT] irq66: 78 c85d0000 0 0 0 0000204 [IWAIT] irq65: 77 c85d020c 0 0 0 0000204 [IWAIT] irq64: 76 c85d0418 0 0 0 0000204 [IWAIT] irq63: 75 c85d0624 0 0 0 0000204 [IWAIT] irq62: 74 c85d0830 0 0 0 0000204 [IWAIT] irq61: 73 c85d0a3c 0 0 0 0000204 [IWAIT] irq60: 72 c85d0c48 0 0 0 0000204 [IWAIT] irq59: 71 c85a220c 0 0 0 0000204 [IWAIT] irq58: 70 c85a2418 0 0 0 0000204 [IWAIT] irq57: 69 c85a2624 0 0 0 0000204 [IWAIT] irq56: 68 c85a2830 0 0 0 0000204 [IWAIT] irq55: 67 c85a2a3c 0 0 0 0000204 [IWAIT] irq54: 66 c85a2c48 0 0 0 0000204 [IWAIT] irq53: 65 c85be000 0 0 0 0000204 [IWAIT] irq52: 64 c85be20c 0 0 0 0000204 [IWAIT] irq51: ciss0 63 c85be418 0 0 0 0000204 [IWAIT] irq50: 62 c85be624 0 0 0 0000204 [IWAIT] irq49: 61 c85be830 0 0 0 0000204 [IWAIT] irq48: 60 c85bea3c 0 0 0 0000204 [IWAIT] irq47: 59 c85bec48 0 0 0 0000204 [IWAIT] irq46: 58 c8593624 0 0 0 0000204 [IWAIT] irq45: 57 c8593830 0 0 0 0000204 [IWAIT] irq44: 56 c8593a3c 0 0 0 0000204 [IWAIT] irq43: 55 c8593c48 0 0 0 0000204 [IWAIT] irq42: 54 c85a1000 0 0 0 0000204 [IWAIT] irq41: 53 c85a120c 0 0 0 0000204 [IWAIT] irq40: 52 c85a1418 0 0 0 0000204 [IWAIT] irq39: 51 c85a1624 0 0 0 0000204 [IWAIT] irq38: 50 c85a1830 0 0 0 0000204 [IWAIT] irq37: 49 c85a1a3c 0 0 0 0000204 [IWAIT] irq36: 48 c85a1c48 0 0 0 0000204 [IWAIT] irq35: 47 c85a2000 0 0 0 0000204 [IWAIT] irq34: 46 c8586c48 0 0 0 0000204 [IWAIT] irq33: 45 c8592000 0 0 0 0000204 [IWAIT] irq32: 44 c859220c 0 0 0 0000204 [IWAIT] irq31: 43 c8592418 0 0 0 0000204 [IWAIT] irq30: 42 c8592624 0 0 0 0000204 [IWAIT] irq29: 41 c8592830 0 0 0 0000204 [IWAIT] irq28: 40 c8592a3c 0 0 0 0000204 [IWAIT] irq27: 39 c8592c48 0 0 0 0000204 [IWAIT] irq26: bge1 38 c8593000 0 0 0 0000204 [IWAIT] irq25: bge0 37 c859320c 0 0 0 0000204 [IWAIT] irq24: 36 c8593418 0 0 0 0000204 [IWAIT] irq23: 35 c8578624 0 0 0 0000204 [IWAIT] irq22: 34 c8578830 0 0 0 0000204 [IWAIT] irq21: 33 c8578a3c 0 0 0 0000204 [IWAIT] irq20: 32 c8578c48 0 0 0 0000204 [IWAIT] irq19: 31 c8586000 0 0 0 0000204 [IWAIT] irq18: 30 c858620c 0 0 0 0000204 [IWAIT] irq17: 29 c8586418 0 0 0 0000204 [IWAIT] irq16: 28 c8586624 0 0 0 0000204 [IWAIT] irq15: ata1 27 c8586830 0 0 0 0000204 [IWAIT] irq14: ata0 26 c8586a3c 0 0 0 0000204 [IWAIT] irq13: 25 c852620c 0 0 0 0000204 [IWAIT] irq12: 24 c8526418 0 0 0 0000204 [IWAIT] irq11: 23 c8526624 0 0 0 0000204 [IWAIT] irq10: 22 c8526830 0 0 0 0000204 [IWAIT] irq9: acpi0 21 c8526a3c 0 0 0 0000204 [IWAIT] irq8: 20 c8526c48 0 0 0 0000204 [IWAIT] irq7: 19 c8578000 0 0 0 0000204 [IWAIT] irq6: 18 c857820c 0 0 0 0000204 [IWAIT] irq5: 17 c8578418 0 0 0 0000204 [IWAIT] irq4: sio0 16 c8521000 0 0 0 0000204 [IWAIT] irq3: sio1 15 c852120c 0 0 0 0000204 [IWAIT] irq0: 14 c8521418 0 0 0 0000204 [IWAIT] irq1: atkbd0 13 c8521624 0 0 0 000020c [Can run] idle: cpu0 12 c8521830 0 0 0 000020c [CPU 1] idle: cpu1 11 c8521a3c 0 0 0 000020c [CPU 2] idle: cpu2 10 c8521c48 0 0 0 000020c [CPU 3] idle: cpu3 1 c8526000 0 0 1 0004200 [SLPQ wait 0xc8526000][SLP] init 0 c0656840 0 0 0 0000200 [IWAIT] swapper db> show lockedvnods Locked vnodes 0xc8991cc0: tag ufs, type VDIR usecount 4, writecount 0, refcount 6 mountedhere 0 flags (VV_ROOT) v_object 0xc922f7bc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xceac9900 (pid 99468) with 3 pending ino 2, on dev stripe/gs0 0xc8acf330: tag ufs, type VDIR usecount 972, writecount 0, refcount 974 mountedhere 0 flags () v_object 0xc8d476b4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xccb1d180 (pid 99376) with 384 pending ino 2221504, on dev stripe/gs0 0xc8d30440: tag ufs, type VDIR usecount 2, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xca292b58 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc8ad1a80 (pid 99373) with 1 pending ino 2221506, on dev stripe/gs0 0xc8dc1aa0: tag ufs, type VDIR usecount 1, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xca127ad4 ref 0 pages 40 lock type ufs: EXCL (count 1) by thread 0xce67cd80 (pid 99361) with 1 pending ino 2221507, on dev stripe/gs0 0xc8d38990: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xc96709cc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc86ebd80 (pid 99265) ino 2221514, on dev stripe/gs0 0xc8d30220: tag ufs, type VDIR usecount 1, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xcce0c318 ref 0 pages 38 lock type ufs: EXCL (count 1) by thread 0xce680000 (pid 99355) with 1 pending ino 2221515, on dev stripe/gs0 0xc8d11dd0: tag ufs, type VDIR usecount 0, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xcb51ad68 ref 0 pages 40 lock type ufs: EXCL (count 1) by thread 0xce67d900 (pid 91041) ino 2221527, on dev stripe/gs0 0xc8dd7110: tag ufs, type VDIR usecount 0, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xc8e3f420 ref 0 pages 39 lock type ufs: EXCL (count 1) by thread 0xccb1e180 (pid 99375) ino 2221545, on dev stripe/gs0 0xcd2ae990: tag ufs, type VREG usecount 1, writecount 0, refcount 6493 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xc8e8e300 (pid 98639) ino 4, on dev stripe/gs0 db> alltrace Tracing command sendmail pid 99774 tid 100861 td 0xced87000 db> alltrace 99468 99376 99373 99361 99265 99355 91041 99375 98639 Tracing command sendmail pid 99774 tid 100861 td 0xced87000 db> trace 99468 Tracing pid 99468 tid 100572 td 0xceac9900 sched_switch(ceac9900,0,1,c6dcade6,9d4675e8) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c8acf388,ebc8f930) at mi_switch+0x1d5 sleepq_switch(c8acf388,ebc8f964,c04bb9ce,c8acf388,50) at sleepq_switch+0x16f sleepq_wait(c8acf388,50,c06186b3,0,ebc8f978) at sleepq_wait+0x11 msleep(c8acf388,c06576d4,50,c06186b3,0) at msleep+0x3d7 acquire(ebc8f9cc,40,60000,c05104f8,c8991d18) at acquire+0x89 lockmgr(c8acf388,2002,c8acf3ac,ceac9900,ebc8f9f4) at lockmgr+0x45f vop_stdlock(ebc8fa4c,c05f9f5b,c0646cc0,ebc8fa4c,ebc8fa04) at vop_stdlock+0x2f VOP_LOCK_APV(c0647200,ebc8fa4c,ebc8fa1c,c05f9f43,ebc8fa4c) at VOP_LOCK_APV+0x44 ffs_lock(ebc8fa4c,0,2002,c8acf330,ebc8fa68) at ffs_lock+0x19 VOP_LOCK_APV(c0646cc0,ebc8fa4c,c0653620,c8991cc0,2002) at VOP_LOCK_APV+0x44 vn_lock(c8acf330,2002,ceac9900,c8991cc0,4) at vn_lock+0x132 vget(c8acf330,2002,ceac9900,40,cc557900) at vget+0xee cache_lookup(c8991cc0,ebc8fc74,ebc8fc88,ceac9900,cc557900) at cache_lookup+0x501 vfs_cache_lookup(ebc8fba8,ebc8fb50,c8991cc0,0,ebc8fbc4) at vfs_cache_lookup+0xa1 VOP_LOOKUP_APV(c0646cc0,ebc8fba8,ebc8fb94,ceac9900,ce6436f4) at VOP_LOOKUP_APV+0x44 lookup(ebc8fc60,c8910c00,400,ebc8fc7c,0) at lookup+0x4a8 namei(ebc8fc60,ebc8fc68,c04b4bb1,ebc8fc78,c05a4108) at namei+0x442 kern_chdir(ceac9900,bfbfcc00,0,ebc8fd30,c05ef1c1) at kern_chdir+0x42 chdir(ceac9900,ebc8fd04,4,ceac9900,8146000) at chdir+0x22 syscall(3b,3b,3b,8113ca0,0) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (12, FreeBSD ELF32, chdir), eip = 0x88297ba3, esp = 0xbfbfbb3c, ebp = 0xbfbfd018 --- db> trace 99376 Tracing pid 99376 tid 100414 td 0xccb1d180 sched_switch(ccb1d180,0,1,997d638c,601ad0fb) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c8d30498,eba4d878) at mi_switch+0x1d5 sleepq_switch(c8d30498,eba4d8ac,c04bb9ce,c8d30498,50) at sleepq_switch+0x16f sleepq_wait(c8d30498,50,c06186b3,0,0) at sleepq_wait+0x11 msleep(c8d30498,c065852c,50,c06186b3,0) at msleep+0x3d7 acquire(eba4d914,40,60000,c0597494,1000) at acquire+0x89 lockmgr(c8d30498,2002,c8d304bc,ccb1d180,eba4d93c) at lockmgr+0x45f vop_stdlock(eba4d994,0,c0646cc0,eba4d994,eba4d94c) at vop_stdlock+0x2f VOP_LOCK_APV(c0647200,eba4d994,eba4d964,c05f9f43,eba4d994) at VOP_LOCK_APV+0x44 ffs_lock(eba4d994,0,2002,c8d30440,eba4d9b0) at ffs_lock+0x19 VOP_LOCK_APV(c0646cc0,eba4d994,eba4dc1c,eba4dc08,3fff) at VOP_LOCK_APV+0x44 vn_lock(c8d30440,2002,ccb1d180,eba4d9f4,c04a4957) at vn_lock+0x132 vget(c8d30440,2002,ccb1d180,40,ce72d380) at vget+0xee cache_lookup(c8acf330,eba4dc08,eba4dc1c,ccb1d180,ce72d380) at cache_lookup+0x501 vfs_cache_lookup(eba4daf0,eba4da98,c8acf330,ce6eda00,eba4db0c) at vfs_cache_lookup+0xa1 VOP_LOOKUP_APV(c0646cc0,eba4daf0,ccb1d180,c88a5250,0) at VOP_LOOKUP_APV+0x44 lookup(eba4dbf4,c8910000,400,eba4dc10,1) at lookup+0x4a8 namei(eba4dbf4,100,1,2ec,c926f730) at namei+0x442 kern_stat(ccb1d180,80eec20,0,eba4dc68,eba4dc94) at kern_stat+0x3d stat(ccb1d180,eba4dd04,8,ccb1d180,80eb000) at stat+0x2f syscall(3b,bfbf003b,bfbf003b,2000,4000) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x88305907, esp = 0xbfbfa52c, ebp = 0xbfbfa5c8 --- db> trace 99373 Tracing pid 99373 tid 100232 td 0xc8ad1a80 sched_switch(c8ad1a80,0,1,d6bbcf42,9f2c12b3) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c8dc1af8,eb722878) at mi_switch+0x1d5 sleepq_switch(c8dc1af8,eb7228ac,c04bb9ce,c8dc1af8,50) at sleepq_switch+0x16f sleepq_wait(c8dc1af8,50,c06186b3,0,c88f9000) at sleepq_wait+0x11 msleep(c8dc1af8,c0658574,50,c06186b3,0) at msleep+0x3d7 acquire(eb722914,40,60000,c0597494,0) at acquire+0x89 lockmgr(c8dc1af8,2002,c8dc1b1c,c8ad1a80,eb72293c) at lockmgr+0x45f vop_stdlock(eb722994,0,c0646cc0,eb722994,eb72294c) at vop_stdlock+0x2f VOP_LOCK_APV(c0647200,eb722994,eb722964,c05f9f43,eb722994) at VOP_LOCK_APV+0x44 ffs_lock(eb722994,0,2002,c8dc1aa0,eb7229b0) at ffs_lock+0x19 VOP_LOCK_APV(c0646cc0,eb722994,ce9f2c78,23,0) at VOP_LOCK_APV+0x44 vn_lock(c8dc1aa0,2002,c8ad1a80,eb7229f4,c04a4957) at vn_lock+0x132 vget(c8dc1aa0,2002,c8ad1a80,40,ce7b9c00) at vget+0xee cache_lookup(c8d30440,eb722c08,eb722c1c,c8ad1a80,ce7b9c00) at cache_lookup+0x501 vfs_cache_lookup(eb722af0,eb722a98,c8d30440,0,eb722b0c) at vfs_cache_lookup+0xa1 VOP_LOOKUP_APV(c0646cc0,eb722af0,c8ad1a80,c88a5250,0) at VOP_LOOKUP_APV+0x44 lookup(eb722bf4,c8e1b000,400,eb722c10,1) at lookup+0x4a8 namei(eb722bf4,eb722b9c,c05a1789,ce9f2bfc,0) at namei+0x442 kern_stat(c8ad1a80,8141520,0,eb722c68,0) at kern_stat+0x3d stat(c8ad1a80,eb722d04,8,c8ad1a80,8154000) at stat+0x2f syscall(3b,bfbf003b,bfbf003b,0,814f3c0) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x88305907, esp = 0xbfbfa6dc, ebp = 0xbfbfa778 --- db> trace 99361 Tracing pid 99361 tid 100437 td 0xce67cd80 sched_switch(ce67cd80,0,1,3d4e53a2,95c40548) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c86d706c,ebb00908) at mi_switch+0x1d5 sleepq_switch(c86d706c,ebb0093c,c04bb9ce,c86d706c,9f) at sleepq_switch+0x16f sleepq_wait(c86d706c,9f,c061a026,0,c0647200) at sleepq_wait+0x11 msleep(c86d706c,c86d7044,29f,c061a026,0,c0653720,c8dc1aa0,ebb00978,ce67cd80) at msleep+0x3d7 vn_write_suspend_wait(c8dc1aa0,c86d7000,1,0,ca07f817) at vn_write_suspend_wait+0x181 ufs_inactive(ebb009c8,ebb009dc,c051b330,c0646cc0,ebb009c8) at ufs_inactive+0x1b4 VOP_INACTIVE_APV(c0646cc0,ebb009c8,a03,ebb009d0,c051054f) at VOP_INACTIVE_APV+0x3a vinactive(c8dc1aa0,ce67cd80,ffffffdf,ebb00a24,c0513592) at vinactive+0x82 vput(c8dc1aa0,ffffffdf,2,ebb00a50,0) at vput+0x187 vn_open_cred(ebb00bc0,ebb00cc0,180,c9430580,5) at vn_open_cred+0xfb vn_open(ebb00bc0,ebb00cc0,180,5,4b5fcfad) at vn_open+0x33 kern_open(ce67cd80,81414e0,0,a03,180) at kern_open+0xca open(ce67cd80,ebb00d04,c,ce67cd80,8155000) at open+0x36 syscall(3b,3b,3b,0,a02) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbfa40c, ebp = 0xbfbfa498 --- db> trace 99265 Tracing pid 99265 tid 100174 td 0xc86ebd80 sched_switch(c86ebd80,0,1,86db4e3a,c0bf63cf) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c8d30278,e93ee878) at mi_switch+0x1d5 sleepq_switch(c8d30278,e93ee8ac,c04bb9ce,c8d30278,50) at sleepq_switch+0x16f sleepq_wait(c8d30278,50,c06186b3,0,0) at sleepq_wait+0x11 msleep(c8d30278,c0658724,50,c06186b3,0) at msleep+0x3d7 acquire(e93ee914,40,60000,c0597494,1000) at acquire+0x89 lockmgr(c8d30278,2002,c8d3029c,c86ebd80,e93ee93c) at lockmgr+0x45f vop_stdlock(e93ee994,0,c0646cc0,e93ee994,e93ee94c) at vop_stdlock+0x2f VOP_LOCK_APV(c0647200,e93ee994,e93ee964,c05f9f43,e93ee994) at VOP_LOCK_APV+0x44 ffs_lock(e93ee994,0,2002,c8d30220,e93ee9b0) at ffs_lock+0x19 VOP_LOCK_APV(c0646cc0,e93ee994,e93eec1c,e93eec08,3fff) at VOP_LOCK_APV+0x44 vn_lock(c8d30220,2002,c86ebd80,e93ee9f4,c04a4957) at vn_lock+0x132 vget(c8d30220,2002,c86ebd80,40,ce6d6d80) at vget+0xee cache_lookup(c8d38990,e93eec08,e93eec1c,c86ebd80,ce6d6d80) at cache_lookup+0x501 vfs_cache_lookup(e93eeaf0,e93eea98,c8d38990,0,e93eeb0c) at vfs_cache_lookup+0xa1 VOP_LOOKUP_APV(c0646cc0,e93eeaf0,c86ebd80,c065b0a0,40) at VOP_LOOKUP_APV+0x44 lookup(e93eebf4,ce89b800,400,e93eec10,1) at lookup+0x4a8 namei(e93eebf4,100,e93eeb9c,5b4,c9236730) at namei+0x442 kern_stat(c86ebd80,80eec20,0,e93eec68,e93eec94) at kern_stat+0x3d stat(c86ebd80,e93eed04,8,c86ebd80,80eb000) at stat+0x2f syscall(3b,bfbf003b,bfbf003b,2000,4000) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x88305907, esp = 0xbfbfa52c, ebp = 0xbfbfa5c8 --- db> trace 99355 Tracing pid 99355 tid 100426 td 0xce680000 sched_switch(ce680000,0,1,a8b2a004,794d29ea) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c86d706c,ebb21908) at mi_switch+0x1d5 sleepq_switch(c86d706c,ebb2193c,c04bb9ce,c86d706c,9f) at sleepq_switch+0x16f sleepq_wait(c86d706c,9f,c061a026,0,c0647200) at sleepq_wait+0x11 msleep(c86d706c,c86d7044,29f,c061a026,0,c0653720,c8d30220,ebb21978,ce680000) at msleep+0x3d7 vn_write_suspend_wait(c8d30220,c86d7000,1,0,c8e81817) at vn_write_suspend_wait+0x181 ufs_inactive(ebb219c8,ebb219dc,c051b330,c0646cc0,ebb219c8) at ufs_inactive+0x1b4 VOP_INACTIVE_APV(c0646cc0,ebb219c8,a03,ebb219d0,c051054f) at VOP_INACTIVE_APV+0x3a vinactive(c8d30220,ce680000,ffffffdf,ebb21a24,c0513592) at vinactive+0x82 vput(c8d30220,ffffffdf,2,ebb21a50,0) at vput+0x187 vn_open_cred(ebb21bc0,ebb21cc0,180,c9296280,5) at vn_open_cred+0xfb vn_open(ebb21bc0,ebb21cc0,180,5,97303610) at vn_open+0x33 kern_open(ce680000,8141500,0,a03,180) at kern_open+0xca open(ce680000,ebb21d04,c,ce680000,8151000) at open+0x36 syscall(bfbf003b,bfbf003b,bfbf003b,0,a02) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbfa5ac, ebp = 0xbfbfa638 --- db> trace 91041 Tracing pid 91041 tid 100430 td 0xce67d900 sched_switch(ce67d900,0,1,5100ec54,ba61202) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c86d706c,ebb15908) at mi_switch+0x1d5 sleepq_switch(c86d706c,ebb1593c,c04bb9ce,c86d706c,9f) at sleepq_switch+0x16f sleepq_wait(c86d706c,9f,c061a026,0,c0647200) at sleepq_wait+0x11 msleep(c86d706c,c86d7044,29f,c061a026,0,c0653720,c8d11dd0,ebb15978,ce67d900) at msleep+0x3d7 vn_write_suspend_wait(c8d11dd0,c86d7000,1,0,c8ced017) at vn_write_suspend_wait+0x181 ufs_inactive(ebb159c8,ebb159dc,c051b330,c0646cc0,ebb159c8) at ufs_inactive+0x1b4 VOP_INACTIVE_APV(c0646cc0,ebb159c8,a22,ebb159d0,c051054f) at VOP_INACTIVE_APV+0x3a vinactive(c8d11dd0,ce67d900,ffffffdf,ebb15a24,c0513592) at vinactive+0x82 vput(c8d11dd0,ffffffdf,2,0,0) at vput+0x187 vn_open_cred(ebb15bc0,ebb15cc0,180,c8fc5d80,8) at vn_open_cred+0xfb vn_open(ebb15bc0,ebb15cc0,180,8,c0653120) at vn_open+0x33 kern_open(ce67d900,bfbfa400,0,a22,180) at kern_open+0xca open(ce67d900,ebb15d04,c,ce67d900,ce67d900) at open+0x36 syscall(80e003b,3b,bfbf003b,8108d00,0) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbf976c, ebp = 0xbfbfb118 --- db> trace 99375 Tracing pid 99375 tid 100404 td 0xccb1e180 sched_switch(ccb1e180,0,1,2efe85fa,5fbad7c4) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c86d706c,eba6b908) at mi_switch+0x1d5 sleepq_switch(c86d706c,eba6b93c,c04bb9ce,c86d706c,9f) at sleepq_switch+0x16f sleepq_wait(c86d706c,9f,c061a026,0,c0647200) at sleepq_wait+0x11 msleep(c86d706c,c86d7044,29f,c061a026,0,c0653720,c8dd7110,eba6b978,ccb1e180) at msleep+0x3d7 vn_write_suspend_wait(c8dd7110,c86d7000,1,0,c8916417) at vn_write_suspend_wait+0x181 ufs_inactive(eba6b9c8,eba6b9dc,c051b330,c0646cc0,eba6b9c8) at ufs_inactive+0x1b4 VOP_INACTIVE_APV(c0646cc0,eba6b9c8,a03,eba6b9d0,c051054f) at VOP_INACTIVE_APV+0x3a vinactive(c8dd7110,ccb1e180,ffffffdf,eba6ba24,c0513592) at vinactive+0x82 vput(c8dd7110,ffffffdf,2,eba6ba50,0) at vput+0x187 vn_open_cred(eba6bbc0,eba6bcc0,180,ccb73b80,5) at vn_open_cred+0xfb vn_open(eba6bbc0,eba6bcc0,180,5,188c500) at vn_open+0x33 kern_open(ccb1e180,8141560,0,a03,180) at kern_open+0xca open(ccb1e180,eba6bd04,c,ccb1e180,8152000) at open+0x36 syscall(3b,bfbf003b,bfbf003b,0,a02) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbfa5ac, ebp = 0xbfbfa638 --- db> trace 98639 Tracing pid 98639 tid 100282 td 0xc8e8e300 sched_switch(c8e8e300,0,1,cc167362,733b6597) at sched_switch+0x158 mi_switch(1,0,c04d7b33,c8991d18,eb80c544) at mi_switch+0x1d5 sleepq_switch(c8991d18,eb80c578,c04bb9ce,c8991d18,50) at sleepq_switch+0x16f sleepq_wait(c8991d18,50,c06186b3,0,c065ae80) at sleepq_wait+0x11 msleep(c8991d18,c065888c,50,c06186b3,0) at msleep+0x3d7 acquire(eb80c5e0,40,60000,c8e8e300,0) at acquire+0x89 lockmgr(c8991d18,2002,c8991d3c,c8e8e300,eb80c608) at lockmgr+0x45f vop_stdlock(eb80c660,0,c0646cc0,eb80c660,eb80c618) at vop_stdlock+0x2f VOP_LOCK_APV(c0647200,eb80c660,eb80c630,c05f9f43,eb80c660) at VOP_LOCK_APV+0x44 ffs_lock(eb80c660,0,2002,c8991cc0,eb80c67c) at ffs_lock+0x19 VOP_LOCK_APV(c0646cc0,eb80c660,c06402e0,eb80c7d0,0) at VOP_LOCK_APV+0x44 vn_lock(c8991cc0,2002,c8e8e300,4000,c851f300) at vn_lock+0x132 ffs_snapshot(c86d7000,cc978e00,eb80c9a4,6c,eb80c964) at ffs_snapshot+0x152b ffs_mount(c86d7000,c8e8e300,0,c8e8e300,c9f71bb0) at ffs_mount+0x9af vfs_domount(c8e8e300,c88a8760,c88a8870,11211000,c9a77510) at vfs_domount+0x728 vfs_donmount(c8e8e300,11211000,eb80cbec,c9314580,e) at vfs_donmount+0x12e kernel_mount(c8737b80,11211000,eb80cc30,6c,bfbfe8c8) at kernel_mount+0x46 ffs_cmount(c8737b80,bfbfe110,11211000,c8e8e300,0) at ffs_cmount+0x85 mount(c8e8e300,eb80cd04,10,c8e8e300,8052000) at mount+0x21b syscall(3b,3b,3b,8814819c,bfbfe0b0) at syscall+0x324 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (21, FreeBSD ELF32, mount), eip = 0x880b8b03, esp = 0xbfbfe07c, ebp = 0xbfbfe778 --- db> continue # shutdown -r now Shutdown NOW! shutdown: [pid 99927] *** FINAL System shutdown message from root@mx17.sac.fedex.com *** System going down IMMEDIATELY Nov 22 20:31:48 mx17 shutdown: reboot by root: # System shutdown time has arrived ... Below are the names of the locked inodes. Note that mqueue/.hoststat is the mount point for a memory filesystem created via mdmfs(8). # cd /var/spool # find . \( -inum 2 -o -inum 2221504 -o -inum 2221506 -o -inum 2221507 -o -inum 2221514 -o -inum 2221515 -o -inum 2221527 -o -inum 2221545 -o -inum 4 \) -print | xargs ls -id 2 . 2221507 ./mqueue/q00/df 4 ./.snap/dump_snapshot 2221514 ./mqueue/q03 2221504 ./mqueue 2221515 ./mqueue/q03/df 2 ./mqueue/.hoststat 2221527 ./mqueue/q07/qf 2221506 ./mqueue/q00 2221545 ./mqueue/q09/df