From owner-freebsd-fs@FreeBSD.ORG Mon Oct 11 17:48:19 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44BBF1065670 for ; Mon, 11 Oct 2010 17:48:19 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id DB9EE8FC16 for ; Mon, 11 Oct 2010 17:48:18 +0000 (UTC) Received: by gxk4 with SMTP id 4so255404gxk.13 for ; Mon, 11 Oct 2010 10:48:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=1AdEJ9CZUV7seSEomWpKppQ5PpJHDb/E+wdlQCokCkk=; b=OLYQWOh2r8FmQnPXDiNCYZwwB0uSU1jWa2DeTckU2b2NvbUj44SqpGu9tJfz8Uspst 2IRRl/CVQZllYYQvhGNGWaCkWUKLH6p7y9Pi6bHi3UJ1HFBTeON5BGBYBYz5NYte7GsU 1cxq3AzaW1iy9bKf0Eo/lzbRWGn8Oysn8Rhic= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=kYYYrhZ8zh4uahEcY7iYTCf9eNM+wyh1PoA7yq0Eu+2evyoLa0okcwOIt/N90F5+Db MZiTjPwHI3h9LsZ7rL8qZZ6lSYlApIXNqvd6dPSSPstg5YBnY6x/iONwMCDcQfZW/E/i SYsOV6oDnZBDOmRb9OYKPU4GpaM7yah5/5okY= MIME-Version: 1.0 Received: by 10.42.191.16 with SMTP id dk16mr1878354icb.506.1286819297824; Mon, 11 Oct 2010 10:48:17 -0700 (PDT) Received: by 10.231.167.65 with HTTP; Mon, 11 Oct 2010 10:48:17 -0700 (PDT) In-Reply-To: <20101011154647.GA11532@icarus.home.lan> References: <39F05641-4E46-4BE0-81CA-4DEB175A5FBE@free.de> <20101009111241.GA58948@icarus.home.lan> <4CB17983.3020907@icyb.net.ua> <20101011151508.GA10917@icarus.home.lan> <4CB32C75.2060000@icyb.net.ua> <20101011154647.GA11532@icarus.home.lan> Date: Mon, 11 Oct 2010 12:48:17 -0500 Message-ID: From: "Sam Fourman Jr." To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Andriy Gapon Subject: Re: Locked up processes after upgrade to ZFS v15 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Oct 2010 17:48:19 -0000 On Mon, Oct 11, 2010 at 10:46 AM, Jeremy Chadwick wrote: > On Mon, Oct 11, 2010 at 10:40:11AM -0500, Sam Fourman Jr. wrote: >> > Jeremy, >> > >> > were are in a process of debugging this issue with Kai and I think tha= t we are >> > onto something. =A0I would like to ask you for some additional testing= . >> > >> > What kind of workload do your run? >> > Do you have anything using sendfile(2)? =A0E.g. Apache with EnableSend= file enabled >> > (it might be by default, without explicit options). >> > Can you try to disable sendfile(2) use, reboot and see how system beha= ves? >> > >> > If you still experience the same problem after doing the above, then I= 'd like to >> > ask you for shell access with root privileges; or establishing communi= cation via >> > IM and running some commands for me. >> > >> > Thank you! >> > -- >> > Andriy Gapon >> > _______________________________________________ >> >> >> Andriy, >> >> I am not sure if this is the same issue, but >> I have a similar problem running FreeBSD 9-CURRENT amd64 (sources as >> of 1 hour ago) >> a zfs lockup appears very frequently, it is a NFS server it locks up >> several times per day. >> >> I have been having these issues since before the ZFSv15 patch was >> committed to HEAD a few months ago. >> let me know if you need details, I have no problems giving you root >> ssh on this machine... but it does =A0not have a serial port. >> so I am not sure how useful it would be. > > Are the processes which livelock (get stuck and cannot be killed, even > with kill -9) stuck in states "zfs" or "zfsmrb"? =A0You can check with "p= s > -axl" or top. > > If so, it's probably the same problem. =A0If not, it may be a different > problem. > > -- ok so in ~3 hours I recived yet another zfs lockup I was in the middle of a scrub and zpool status -v, will not return FNFS# FNFS# zpool status -v pool: Network state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub in progress for 2h1m, 14.52% done, 11h52m to go config: NAME STATE READ WRITE CKSUM Network ONLINE 0 0 0 mirror ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 mirror ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 mirror ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 mirror ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /Network/tv/Sam-tv/Lost/Lost.S05E08.HDTV.XviD-XOR.avi /Network/jail/mirror/usr/local/man/man3/XChangeDeviceProperty.3.gz /Network/jail/mirror/usr/local/man/man3/XForceScreenSaver.3.gz pool: zFNFS state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zFNFS ONLINE 0 0 0 raidz2 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 gpt/disk2 ONLINE 0 0 0 gpt/disk3 ONLINE 0 0 0 errors: No known data errors FNFS# FNFS# FNFS# zpool status -v FNFS# ps axHwwl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMA= ND 0 0 0 0 -16 0 0 2608 sched DLs ?? 1:31.59 [kern= el] 0 0 0 0 8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 8 0 0 2608 - DLs ?? 0:00.01 [kern= el] 0 0 0 0 -68 0 0 2608 - DLs ?? 0:01.39 [kern= el] 0 0 0 0 -84 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -84 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -84 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -84 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 deadlk DLs ?? 0:00.23 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.03 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.01 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.02 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.07 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.07 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.10 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.10 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.10 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.07 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.06 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.06 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.06 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.06 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.07 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.06 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.06 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.15 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.04 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 6:04.82 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 6:04.74 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 6:04.78 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.62 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.60 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.62 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.14 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.14 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.14 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.14 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.14 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.14 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.14 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.14 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.06 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.05 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -16 0 0 2608 - DLs ?? 0:00.02 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.18 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 0 0 0 -8 0 0 2608 - DLs ?? 0:00.00 [kern= el] 0 1 0 0 44 0 7296 436 wait ILs ?? 0:00.02 /sbin/init -- 0 2 0 0 -8 0 0 16 - DL ?? 0:00.44 [g_ev= ent] 0 3 0 0 -8 0 0 16 - DL ?? 0:50.05 [g_up= ] 0 4 0 0 -8 0 0 16 - DL ?? 1:25.38 [g_do= wn] 0 5 0 0 -8 0 0 128 arc_re DL ?? 0:01.49 [zfsk= ern] 0 5 0 0 -8 0 0 128 l2arc_ DL ?? 0:00.04 [zfsk= ern] 0 5 0 0 -8 0 0 128 tx->tx DL ?? 0:00.00 [zfsk= ern] 0 5 0 0 -8 0 0 128 tx->tx DL ?? 0:00.07 [zfsk= ern] 0 5 0 0 -8 0 0 128 tx->tx DL ?? 0:00.00 [zfsk= ern] 0 5 0 0 -8 0 0 128 spa->s DL ?? 5:04.17 [zfsk= ern] 0 6 0 0 -16 0 0 16 waitin DL ?? 0:00.00 [sctp_iterator] 0 7 0 0 76 0 0 16 ccb_sc DL ?? 0:05.10 [xpt_= thrd] 0 8 0 0 -16 0 0 16 psleep DL ?? 0:00.01 [pagedaemon] 0 9 0 0 -16 0 0 16 psleep DL ?? 0:00.00 [vmda= emon] 0 10 0 0 -16 0 0 16 audit_ DL ?? 0:00.00 [audi= t] 0 11 0 0 171 0 0 64 - RL ?? 197:54.91 [idle= ] 0 11 0 0 171 0 0 64 - RL ?? 195:59.68 [idle= ] 0 11 0 0 171 0 0 64 - RL ?? 196:58.20 [idle= ] 0 11 0 0 171 0 0 64 - RL ?? 170:17.14 [idle= ] 0 12 0 0 -44 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -32 0 0 336 - WL ?? 0:07.07 [intr= ] 0 12 0 0 -32 0 0 336 - WL ?? 0:00.89 [intr= ] 0 12 0 0 -32 0 0 336 - WL ?? 0:00.42 [intr= ] 0 12 0 0 -32 0 0 336 - WL ?? 0:00.17 [intr= ] 0 12 0 0 -36 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -24 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -24 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -40 0 0 336 - WL ?? 8:54.70 [intr= ] 0 12 0 0 -28 0 0 336 - WL ?? 0:00.04 [intr= ] 0 12 0 0 -52 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -64 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -64 0 0 336 - WL ?? 31:08.03 [intr= ] 0 12 0 0 -64 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -64 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -64 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -64 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -64 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -60 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -48 0 0 336 - WL ?? 0:00.00 [intr= ] 0 12 0 0 -60 0 0 336 - WL ?? 0:00.00 [intr= ] 0 13 0 0 44 0 0 16 - DL ?? 0:06.17 [yarr= ow] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -68 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.07 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -68 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.08 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -68 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.09 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -68 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.08 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -68 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.09 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -68 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.09 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -68 0 0 448 - DL ?? 0:00.00 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.06 [usb] 0 14 0 0 -64 0 0 448 - DL ?? 0:00.00 [usb] 0 15 0 0 44 0 0 16 pgzero DL ?? 0:00.00 [page= zero] 0 16 0 0 -16 0 0 16 psleep DL ?? 0:00.04 [bufdaemon] 0 17 0 0 44 0 0 16 vlruwt DL ?? 0:00.27 [vnlr= u] 0 18 0 0 44 0 0 16 zio->i DL ?? 0:04.80 [sync= er] 0 19 0 0 -16 0 0 16 sdflus DL ?? 0:00.10 [softdepflush] 0 20 0 0 -16 0 0 16 flowcl DL ?? 0:00.04 [flowcleaner] 0 158 1 0 76 0 2892 824 pause Is ?? 0:00.00 adjkerntz -i 0 743 1 0 44 0 7296 744 select Is ?? 0:00.00 /sbin= /devd 0 938 1 0 44 0 11284 1352 select Ss ?? 0:00.03 /usr/sbin/syslogd -s 0 962 1 0 44 0 12212 1624 select Ss ?? 0:00.02 /usr/sbin/rpcbind 0 1043 1 0 76 0 10056 1288 pause Is ?? 0:00.00 nfsuserd: master (nfsuserd) 0 1044 1043 0 44 0 10056 1300 select S ?? 0:00.01 nfsuserd: slave (nfsuserd) 0 1045 1043 0 44 0 10056 1300 select S ?? 0:00.01 nfsuserd: slave (nfsuserd) 0 1046 1043 0 44 0 10056 1300 select S ?? 0:00.01 nfsuserd: slave (nfsuserd) 0 1047 1043 0 44 0 10056 1300 select S ?? 0:00.01 nfsuserd: slave (nfsuserd) 0 1066 1 0 44 0 11280 1924 select Is ?? 0:00.06 /usr/sbin/mountd -e -r -l /etc/exports /etc/zfs/exports 0 1074 1 0 50 0 10052 1564 select Is ?? 0:00.02 nfsd: master (nfsd) 0 1081 1074 0 44 0 10052 1080 rpcsvc S ?? 0:00.03 nfsd: server (nfsd) 0 1081 1074 0 44 0 10052 1080 rpcsvc S ?? 0:00.03 nfsd: server (nfsd) 0 1081 1074 0 44 0 10052 1080 rpcsvc S ?? 0:00.03 nfsd: server (nfsd) 0 1081 1074 0 44 0 10052 1080 rpcsvc S ?? 0:00.03 nfsd: server (nfsd) 0 1082 1 0 44 0 273396 1452 select Ss ?? 0:00.01 /usr/sbin/rpc.statd 0 1089 1 0 76 0 12340 1424 rpcsvc Ss ?? 0:00.01 /usr/sbin/rpc.lockd 0 1249 1 0 44 0 30548 3060 select Is ?? 0:00.00 /usr/sbin/sshd 0 1257 1 0 44 0 16408 3392 select Ss ?? 0:00.12 sendmail: accepting connections (sendmail) 25 1263 1 0 44 0 16408 3240 pause Is ?? 0:00.00 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail) 0 1270 1 0 44 0 12212 1412 nanslp Is ?? 0:00.02 /usr/sbin/cron -s 0 1469 1 0 44 0 6888 1208 select SsJ ?? 0:00.02 /usr/sbin/syslogd -s 70 1594 1 0 44 0 60780 5612 select SsJ ?? 0:00.14 /usr/local/bin/postgres -D /usr/local/pgsql/data 0 1618 1 0 76 0 26004 3036 select IsJ ?? 0:00.00 /usr/sbin/sshd 0 1625 1 0 44 0 11976 3360 select SsJ ?? 0:00.14 sendmail: accepting connections (sendmail) 25 1631 1 0 44 0 11976 3068 pause IsJ ?? 0:00.00 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail) 0 1638 1 0 44 0 7944 1276 nanslp IsJ ?? 0:00.02 /usr/sbin/cron -s 0 1801 1 0 44 0 6888 1200 select SsJ ?? 0:00.02 /usr/sbin/syslogd -s 1002 1867 1 0 76 0 28376 1852 accept IsJ ?? 0:00.00 /usr/local/bin/svnserve -d --listen-port=3D3690 --listen-host 0.0.0.0 -r /usr/local/svn 70 1916 1 0 44 0 53492 5412 select SsJ ?? 0:00.14 /usr/local/bin/postgres -D /usr/local/pgsql/data 0 1948 1 0 76 0 26004 3056 select IsJ ?? 0:00.00 /usr/sbin/sshd 0 1955 1 0 44 0 11976 3096 select SsJ ?? 0:00.14 sendmail: accepting connections (sendmail) 25 1961 1 0 44 0 11976 2980 pause IsJ ?? 0:00.00 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail) 0 1967 1 0 44 0 7944 1264 nanslp IsJ ?? 0:00.02 /usr/sbin/cron -s 0 2118 1 0 44 0 6888 1200 select IsJ ?? 0:00.02 /usr/sbin/syslogd -s 1002 2237 1 0 76 0 6704 1980 select IsJ ?? 0:00.00 /usr/local/sbin/cvsupd -e -C 8 -l @daemon -b /usr/local/etc/cvsup -s sup.client 0 2257 1 0 44 0 11976 2856 select SsJ ?? 0:00.14 sendmail: accepting connections (sendmail) 25 2263 1 0 44 0 11976 2796 pause IsJ ?? 0:00.00 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail) 0 2269 1 0 44 0 7944 1256 nanslp IsJ ?? 0:00.02 /usr/sbin/cron -s 0 2375 1 0 76 0 13268 1544 select Is ?? 0:00.00 /usr/sbin/inetd -wW -C 60 70 2419 1594 0 44 0 60780 5680 select SsJ ?? 0:01.35 postgres: writer process (postgres) 70 2420 1594 0 44 0 60780 5664 select SsJ ?? 0:00.73 postgres: wal writer process (postgres) 70 2421 1594 0 44 0 60780 5688 zio->i DsJ ?? 0:00.15 postgres: autovacuum launcher process (postgres) 70 2422 1594 0 44 0 23788 3704 select SsJ ?? 0:00.12 postgres: stats collector process (postgres) 70 2425 1916 0 44 0 53492 5520 select SsJ ?? 0:01.19 postgres: writer process (postgres) 70 2426 1916 0 44 0 53492 5512 select SsJ ?? 0:00.68 postgres: wal writer process (postgres) 70 2427 1916 0 44 0 53492 5492 zio->i DsJ ?? 0:00.16 postgres: autovacuum launcher process (postgres) 70 2428 1916 0 44 0 24900 3624 select SsJ ?? 0:00.12 postgres: stats collector process (postgres) 0 2621 1249 0 44 0 42480 4228 select Is ?? 0:00.03 sshd: root@pts/0 (sshd) 0 3711 1967 0 44 0 7944 1332 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3712 3711 0 44 0 7940 952 zio->i DsJ ?? 0:00.00 /usr/libexec/atrun 0 3713 2269 0 44 0 7944 1328 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3714 3713 0 44 0 7940 936 zio->i DsJ ?? 0:00.00 /usr/libexec/atrun 0 3717 1638 0 44 0 7944 1336 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3718 3717 0 44 0 7940 952 zio->i DsJ ?? 0:00.00 /usr/libexec/atrun 0 3719 1967 0 44 0 7944 1332 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3720 3719 0 44 0 7944 1332 zio->i DVsJ ?? 0:00.00 cron: running job (cron) 0 3721 2269 0 44 0 7944 1328 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3722 3721 0 44 0 7944 1328 zio->i DVsJ ?? 0:00.00 cron: running job (cron) 0 3735 1638 0 44 0 7944 1336 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3736 3735 0 44 0 7944 1336 zio->i DVsJ ?? 0:00.00 cron: running job (cron) 0 3737 1967 0 44 0 7944 1332 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3738 3737 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3739 2269 0 44 0 7944 1328 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3740 3739 0 44 0 4884 824 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3743 1638 0 44 0 7944 1336 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3744 3743 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3745 1967 0 44 0 7944 1332 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3746 3745 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3747 2269 0 44 0 7944 1328 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3748 3747 0 44 0 4884 824 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3751 1638 0 44 0 7944 1336 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3752 3751 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3753 1967 0 44 0 7944 1332 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3754 3753 0 44 0 7944 1332 db->db DVsJ ?? 0:00.00 cron: running job (cron) 0 3755 2269 0 44 0 7944 1328 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3756 3755 0 44 0 7944 1328 db->db DVsJ ?? 0:00.00 cron: running job (cron) 0 3769 1638 0 44 0 7944 1336 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3770 3769 0 44 0 7944 1336 db->db DVsJ ?? 0:00.00 cron: running job (cron) 0 3771 1967 0 44 0 7944 1332 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3772 3771 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3773 2269 0 44 0 7944 1328 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3774 3773 0 44 0 4884 824 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3777 1638 0 44 0 7944 1336 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3778 3777 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3779 1967 0 44 0 7944 1332 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3780 3779 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3781 2269 0 44 0 7944 1328 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3782 3781 0 44 0 4884 824 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3785 1638 0 44 0 7944 1336 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3786 3785 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3788 1249 0 44 0 42480 4344 select Ss ?? 0:00.02 sshd: root@pts/1 (sshd) 0 3796 1967 0 44 0 7944 1332 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3797 3796 0 44 0 7944 1332 db->db DVsJ ?? 0:00.00 cron: running job (cron) 0 3798 2269 0 44 0 7944 1328 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3799 3798 0 44 0 7944 1328 db->db DVsJ ?? 0:00.00 cron: running job (cron) 0 3812 1638 0 44 0 7944 1336 ppwait DJ ?? 0:00.00 cron: running job (cron) 2 3813 3812 0 44 0 7944 1336 db->db DVsJ ?? 0:00.00 cron: running job (cron) 0 3818 1967 0 44 0 7944 1332 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3819 3818 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3820 2269 0 44 0 7944 1328 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3821 3820 0 44 0 4884 824 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 3824 1638 0 44 0 7944 1336 piperd IJ ?? 0:00.00 cron: running job (cron) 1 3825 3824 0 44 0 4884 840 zfs DsJ ?? 0:00.00 /usr/libexec/atrun 0 2409 1 0 76 0 11152 1160 ttyin Is+ v0 0:00.00 /usr/libexec/getty Pc ttyv0 0 2410 1 0 76 0 11152 1160 ttyin Is+ v1 0:00.00 /usr/libexec/getty Pc ttyv1 0 2411 1 0 76 0 11152 1160 ttyin Is+ v2 0:00.00 /usr/libexec/getty Pc ttyv2 0 2412 1 0 76 0 11152 1160 ttyin Is+ v3 0:00.00 /usr/libexec/getty Pc ttyv3 0 2413 1 0 76 0 11152 1160 ttyin Is+ v4 0:00.00 /usr/libexec/getty Pc ttyv4 0 2414 1 0 76 0 11152 1160 ttyin Is+ v5 0:00.00 /usr/libexec/getty Pc ttyv5 0 2415 1 0 76 0 11152 1160 ttyin Is+ v6 0:00.00 /usr/libexec/getty Pc ttyv6 0 2416 1 0 76 0 11152 1160 ttyin Is+ v7 0:00.00 /usr/libexec/getty Pc ttyv7 0 2624 2621 0 44 0 14556 2700 pause Is 0 0:00.03 -csh = (csh) 0 3787 2624 0 44 0 19948 1600 zio->i D+ 0 0:00.00 zpool status -v 0 3792 3788 0 44 0 14556 2724 pause Ss 1 0:00.02 -csh = (csh) 0 3827 3792 0 44 0 12272 1604 - R+ 1 0:00.00 ps ax= Hwwl FNFS# --=20 Sam Fourman Jr. Fourman Networks http://www.fourmannetworks.com