From owner-freebsd-questions@FreeBSD.ORG  Wed Mar 24 22:27:29 2010
Return-Path: <owner-freebsd-questions@FreeBSD.ORG>
Delivered-To: freebsd-questions@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id F00F31065672
	for <freebsd-questions@freebsd.org>;
	Wed, 24 Mar 2010 22:27:29 +0000 (UTC)
	(envelope-from freebsd-questions@m.gmane.org)
Received: from lo.gmane.org (lo.gmane.org [80.91.229.12])
	by mx1.freebsd.org (Postfix) with ESMTP id A97B38FC0A
	for <freebsd-questions@freebsd.org>;
	Wed, 24 Mar 2010 22:27:29 +0000 (UTC)
Received: from list by lo.gmane.org with local (Exim 4.69)
	(envelope-from <freebsd-questions@m.gmane.org>) id 1NuZ2t-00081Q-V5
	for freebsd-questions@freebsd.org; Wed, 24 Mar 2010 23:27:28 +0100
Received: from 78-1-170-106.adsl.net.t-com.hr ([78.1.170.106])
	by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
	id 1AlnuQ-0007hv-00
	for <freebsd-questions@freebsd.org>; Wed, 24 Mar 2010 23:27:27 +0100
Received: from ivoras by 78-1-170-106.adsl.net.t-com.hr with local (Gmexim 0.1
	(Debian)) id 1AlnuQ-0007hv-00
	for <freebsd-questions@freebsd.org>; Wed, 24 Mar 2010 23:27:27 +0100
X-Injected-Via-Gmane: http://gmane.org/
To: freebsd-questions@freebsd.org
From: Ivan Voras <ivoras@freebsd.org>
Date: Wed, 24 Mar 2010 23:27:12 +0100
Lines: 25
Message-ID: <hoe3k0$tuk$2@dough.gmane.org>
References: <cf9b1ee01003240918g3a7d61adqaeccb609090a09ea@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Complaints-To: usenet@dough.gmane.org
X-Gmane-NNTP-Posting-Host: 78-1-170-106.adsl.net.t-com.hr
User-Agent: Thunderbird 2.0.0.21 (X11/20090612)
In-Reply-To: <cf9b1ee01003240918g3a7d61adqaeccb609090a09ea@mail.gmail.com>
Cc: freebsd-fs@freebsd.org
Subject: Re: tuning vfs.zfs.vdev.max_pending and solving the issue of ZFS
 writes choking read IO
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions>
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 24 Mar 2010 22:27:30 -0000

Dan Naumov wrote:
> Hello
> 
> I am having a slight issue (and judging by Google results, similar
> issues have been seen by other FreeBSD and Solaris/OpenSolaris users)
> with writes choking the read IO. The issue I am having is described
> pretty well here:
> http://opensolaris.org/jive/thread.jspa?threadID=106453 It seems that
> under heavy write load, ZFS likes to aggregate a really huge amount of
> data before actually writing it to disks, resulting in sudden 10+
> second stalls where it frantically tries to commit everything,
> completely choking read IO in the process and sometimes even the
> network (with a large enough write to a mirror pool using DD, I can
> cause my SSH sessions to drop dead, without actually running out of
> RAM. As soon as the data is committed, I can reconnect back).

Mostly a wild guess, but can you test if this patch will help with 
choking your network and ssh:

http://people.freebsd.org/~ivoras/diffs/spa.c.diff

?

You can then fiddle with the vfs.zfs.zio_worker_threads_count loader 
tunable to see if it helps more.