From owner-freebsd-stable@FreeBSD.ORG Wed Nov 21 20:18:51 2007 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD1DE16A421; Wed, 21 Nov 2007 20:18:51 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (pointyhat.freebsd.org [IPv6:2001:4f8:fff6::2b]) by mx1.freebsd.org (Postfix) with ESMTP id 316E913C455; Wed, 21 Nov 2007 20:18:50 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <474492B0.1010108@FreeBSD.org> Date: Wed, 21 Nov 2007 21:18:56 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <47137D36.1020305@chistydom.ru> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> <4715C297.1020905@chistydom.ru> <4715C5D7.7060806@FreeBSD.org> <471EE4D9.5080307@chistydom.ru> <4723BF87.20302@FreeBSD.org> <47344E47.9050908@chistydom.ru> <47349A17.3080806@FreeBSD.org> <47373B43.9060406@chistydom.ru> <4739557A.6090209@chistydom.ru> <4741EE9E.9050406@FreeBSD.org> In-Reply-To: <4741EE9E.9050406@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Panagiotis Christias , freebsd-stable@freebsd.org, Alexey Popov Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2007 20:18:51 -0000 Kris Kennaway wrote: > Alexey Popov wrote: >> Hi. >> >> Panagiotis Christias wrote: >>>>>>> In the "good" case you are getting a much higher interrupt rate but >>>>>>> with the data you provided I can't tell where from. You need to run >>>>>>> vmstat -i at regular intervals (e.g. every 10 seconds for a minute) >>>>>>> during the "good" and "bad" times, since it only provides counters >>>>>>> and an average rate over the uptime of the system. >>>>>> Now I'm running 10-process lighttpd and the problem became no so big. >>>>>> >>>>>> I collected interrupt stats and it shows no relation beetween >>>>>> ionterrupts and slowdowns. Here is it: >>>>>> http://83.167.98.162/gprof/intr-graph/ >>>>>> >>>>>> Also I have similiar statistics on mutex profiling and it shows >>>>>> there's no problem in mutexes. >>>>>> http://83.167.98.162/gprof/mtx-graph/mtxgifnew/ >>>>>> >>>>>> I have no idea what else to check. >>>>> I don't know what this graph is showing me :) When precisely is the >>>>> system behaving poorly? >>> what is your RAID controller configuration (read ahead/cache/write >>> policy)? I have seen weird/bogus numbers (~100% busy) reported by >>> systat -v when read ahead was enabled on LSI/amr controllers. >> >> >> ********************************************************************** >> Existing Logical Drive Information >> By LSI Logic Corp.,USA >> >> ********************************************************************** >> [Note: For SATA-2, 4 and 6 channel controllers, please specify >> Ch=0 Id=0..15 for specifying physical drive(Ch=channel, >> Id=Target)] >> >> >> Logical Drive : 0( Adapter: 0 ): Status: OPTIMAL >> --------------------------------------------------- >> SpanDepth :01 RaidLevel: 5 RdAhead : Adaptive Cache: >> DirectIo >> StripSz :064KB Stripes : 6 WrPolicy: WriteBack >> >> Logical Drive 0 : SpanLevel_0 Disks >> Chnl Target StartBlock Blocks Physical Target Status >> ---- ------ ---------- ------ ---------------------- >> 0 00 0x00000000 0x22ec0000 ONLINE >> 0 01 0x00000000 0x22ec0000 ONLINE >> 0 02 0x00000000 0x22ec0000 ONLINE >> 0 03 0x00000000 0x22ec0000 ONLINE >> 0 04 0x00000000 0x22ec0000 ONLINE >> 0 05 0x00000000 0x22ec0000 ONLINE >> >> I tried to run with disabled Read-ahead, but it didn't help. > > I just ran into this myself, and apparently it can be caused by "Patrol > Reads" where the adapter periodically scans the disks to look for media > errors. You can turn this off using -stopPR with the megarc port. > > Kris > Oops, -disPR is the correct command to disable, -stopPR just halts a PR event in progress. Kris