From owner-freebsd-stable@FreeBSD.ORG  Tue Mar  9 10:58:12 2010
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 00C251065670
	for <freebsd-stable@freebsd.org>; Tue,  9 Mar 2010 10:58:12 +0000 (UTC)
	(envelope-from stb@lassitu.de)
Received: from gilb.zs64.net (gilb.zs64.net [212.12.50.234])
	by mx1.freebsd.org (Postfix) with ESMTP id BDDCF8FC18
	for <freebsd-stable@freebsd.org>; Tue,  9 Mar 2010 10:58:11 +0000 (UTC)
Received: by gilb.zs64.net (Postfix, from stb@lassitu.de) id 8FB0F4F117;
	Tue,  9 Mar 2010 10:58:10 +0000 (UTC)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: text/plain; charset=us-ascii
From: Stefan Bethke <stb@lassitu.de>
In-Reply-To: <20100309105355.GD3978@server.vk2pj.dyndns.org>
Date: Tue, 9 Mar 2010 11:58:10 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <D160BCD0-CBD9-492D-8512-044DC03E2D6E@lassitu.de>
References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de>
	<20100309105355.GD3978@server.vk2pj.dyndns.org>
To: Peter Jeremy <peterjeremy@acm.org>
X-Mailer: Apple Mail (2.1077)
Cc: FreeBSD Stable <freebsd-stable@freebsd.org>
Subject: Re: Many processes stuck in zfs
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 09 Mar 2010 10:58:12 -0000

Am 09.03.2010 um 11:53 schrieb Peter Jeremy:

> On 2010-Mar-09 10:15:53 +0100, Stefan Bethke <stb@lassitu.de> wrote:
>> Over the past couple of months, I've more or less regularly observed =
machines having more and more processes stuck in the zfs wchan.  The =
processes never recover from that,
>=20
> How long have you waited?

Many hours, sometimes up to 48 hours (when I didn't notice the stuck =
processes at first).

> There seems to be a problem with low free memory handling that causes =
ZFS
> to turn into cold molasses.  The work-around is to run a program that
> allocates a decent size chunk of memory and then exits.  The original
> suggestion was something like:
> 	perl -e '@x =3D (0) x 1000000;'
> I've written a short program that allocates and dirties ~100MB and =
then
> exits and run it from cron.

I'll try that the next time I encounter the stuck processes.

I'm recording ZFS ARC stats with munin, would I be able to identify such =
a low memory situation from there?  Would it make sense to monitor other =
stats?


Thanks,
Stefan

--=20
Stefan Bethke <stb@lassitu.de>   Fon +49 151 14070811