From owner-freebsd-stable@FreeBSD.ORG  Tue Oct 16 09:03:07 2007
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id AD00916A419;
	Tue, 16 Oct 2007 09:03:07 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36])
	by mx1.freebsd.org (Postfix) with ESMTP id EAA1E13C4C5;
	Tue, 16 Oct 2007 09:03:06 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Message-ID: <47147E49.9020301@FreeBSD.org>
Date: Tue, 16 Oct 2007 11:03:05 +0200
From: Kris Kennaway <kris@FreeBSD.org>
User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728)
MIME-Version: 1.0
To: Alexey Popov <lol@chistydom.ru>
References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org>
	<47146FB4.6040306@chistydom.ru>
In-Reply-To: <47146FB4.6040306@chistydom.ru>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org
Subject: Re: amrd disk performance drop after running under high load
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Oct 2007 09:03:07 -0000

Alexey Popov wrote:
> Hi.
> 
> Kris Kennaway wrote:
> 
>>> After some time of running under high load disk performance become 
>>> expremely poor. At that periods 'systat -vm 1' shows something like  
>>> this:
>> What does "high load" mean?  You need to explain the system workload 
>> more.
> This web service is similiar to YouTube. This server is video store. I
> have around 200G of *.flv (flash video) files on the server.
> 
> I run lighttpd as a web server. Disk load is usually around 50%, network
> output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle.
> 
> As you can see it is a trivial service - sending files to network via HTTP.

A couple of comments.

Does lighttpd actually use HTTP accept filters?

Are you using ipfilter and ipfw?  You are paying a performance penalty 
for having them.

You might try increasing BUCKET_MAX in sys/vm/uma_core.c.  I don't 
really understand the code here, but you seem to be hitting a threshold 
behaviour where you are constantly running out of space in the per CPU 
caches.

This can happen if your workload is unbalanced between the CPUs and you 
are always allocating on one but freeing on another, but I wouldn't 
expect it should happen on your workload.  Maybe it can also happen if 
your turnover is high enough.  What does vmstat -z show during the good 
and bad times?

Kris