From owner-freebsd-questions@FreeBSD.ORG Tue Dec 9 05:10:58 2003 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5332616A4CE for ; Tue, 9 Dec 2003 05:10:58 -0800 (PST) Received: from stats.net (legolas.counted.com [66.230.158.38]) by mx1.FreeBSD.org (Postfix) with ESMTP id 92FEF43D21 for ; Tue, 9 Dec 2003 05:10:56 -0800 (PST) (envelope-from fthylmann@stats.net) Received: from notefabi ([172.176.95.126]) by stats.net ; Tue, 09 Dec 2003 08:10:34 -0500 Message-ID: <01b001c3be55$e932bd20$2102a8c0@NoteFabi> From: "Fabian Thylmann" To: Date: Tue, 9 Dec 2003 14:11:07 +0100 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on legolas.counted.com X-Spam-Level: X-Spam-Status: No, hits=0.5 required=5.0 tests=HTML_20_30,HTML_MESSAGE autolearn=no version=2.60 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: inode state X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Dec 2003 13:10:58 -0000 Hi, I have a heavily used threaded server program running on one of my Dell = Poweredge 1750 servers. Its a statistical analysis package for websites. = Currently it analyses over 60 million requests a day, which (because of = many different reasons) causes it to handle around 120 million http = requests a day. At peaks around 1500 requests a second. The system stores most many statistics in memory which is flushed to = disk in circles by a worker thread. Another big part is stored in an on-disk database which is mmap()'d into = memory. Because we do not have enough memory to keep everything in = memory at one time the mmap() system of course pages data in and out. When I look in systat -v I see that dirtybuf climbs to about 1700 and = then they get flushed to disk, causing high disk usage of around 300-400 = tps whcih renders the disks useless for anything else. When those flushes occure, my apps state as displayed by top(1) gets = into inode state, PRI is set to -14 and cpu usage rapidly drops. The = program and ALL of its threads are stalled at that time. Those inode = states take around 2 oe 3 seconds and happen every 30 seconds or so. In those 3 seconds we lose around 1500 hits at peak times for processing = because the app can not handle them fast enough. This results in around = 2 million or so hits lost over the day for processing. I am now wondering if anyone can explain to me why ALL threads and not = just the threads that actually do I/O work get blocked when dirty = buffers are flushed and what to do to fix this problem. I would be very happy if someone could reply and point me into the right = direction! Fabian Thylmann