Date: Mon, 13 Feb 2012 15:46:07 +0000 From: Tom Evans <tevans.uk@googlemail.com> To: Stephen Sanders <ssanders@softhammer.net> Cc: freebsd-hackers@freebsd.org Subject: Re: Odd RAID Performance Issue Message-ID: <CAFHbX1%2BzoQQ-91v7pkbPQs-Lxtfv3cbCSv%2BDaTy2TBA5DuVR-Q@mail.gmail.com> In-Reply-To: <4F3922A8.2090808@softhammer.net> References: <4F3922A8.2090808@softhammer.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 13, 2012 at 2:48 PM, Stephen Sanders <ssanders@softhammer.net> wrote: > We've an application that logs data on one very large raid6 array > and updates/accesses a database on another smaller raid5 array. > > Both arrays are connected to the same PCIe 3ware RAID controller. =C2=A0 = The > system has 2 six core 3Ghz processors and 24 GB of RAM. =C2=A0The system = is > running FreeBSD 8.1. > > The averaged read/write rate to the database is 2MB/s while the averaged > write raid to the data =C2=A0logging array is 300MB/s. =C2=A0Writes to th= e logging > array are somewhat bursty. > > The problem we're encountering is that the disk subsystem appears to > 'pause' periodically. =C2=A0 It looks as if this is a result of disk read= /write > operations from the database array taking a very long time to complete > (up to 8 sec). > > When the disk read operation takes such a long time, it appears that the > system starts to run out of memory due to bio block buffering. =C2=A0Most > processes end up in either getblk() or waithighrunning(). > > We've instrumented g_vfs_strategie() and bufdone_finish() using dtrace. > The indication from this effort is that a number of reads and writes are > taking 4-8 seconds. > > So far, it looks as if the disk driver and hardware are OK as read/write > operations appear to be in the milli-second region. =C2=A0We believe that= our > instrumentation is pointing to something between the VFS layer and the > CAM as the culprit. > > We've gotten the same result from FreeBSD 8.2 but have not tried FreeBSD > 9 as yet. > > This scenario is not limited to a single system and is occurring on a > couple of systems. > > Does this sound familiar to anyone out there? > > Thanks Do you have a BBU on the 3ware device? It sounds very similar to effects we used to suffer when we used servers with twa without a BBU. Our newer servers use LSI/Dell PERC (with BBUs!), and don't have this sort of issue anymore. Cheers Tom
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFHbX1%2BzoQQ-91v7pkbPQs-Lxtfv3cbCSv%2BDaTy2TBA5DuVR-Q>