From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 13 15:46:08 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C32DA1065672 for ; Mon, 13 Feb 2012 15:46:08 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8191B8FC17 for ; Mon, 13 Feb 2012 15:46:08 +0000 (UTC) Received: by vcmm1 with SMTP id m1so4681489vcm.13 for ; Mon, 13 Feb 2012 07:46:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=/nuL/wI9Ei6Qp1HShRZstzAWahDp0X+gM9sA40yLfoU=; b=aetPujmEJ6NTl6LprGq221gnvTCByziDTBcZ/l+Yd/2Co+UcZgVhbKZH5X0khVTRN/ zXK+W6OcZeuX+tb7329/0sBRAZ5xXHuHM6frmtEY+r6zqyyCFUZC87Fu6t4HhNbMV8tY sp/Kr2P6eJ0IoxQR/rytJiOSMjsRY8O2QuhYM= MIME-Version: 1.0 Received: by 10.220.151.67 with SMTP id b3mr8431813vcw.51.1329147967854; Mon, 13 Feb 2012 07:46:07 -0800 (PST) Received: by 10.52.91.210 with HTTP; Mon, 13 Feb 2012 07:46:07 -0800 (PST) In-Reply-To: <4F3922A8.2090808@softhammer.net> References: <4F3922A8.2090808@softhammer.net> Date: Mon, 13 Feb 2012 15:46:07 +0000 Message-ID: From: Tom Evans To: Stephen Sanders Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Odd RAID Performance Issue X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2012 15:46:08 -0000 On Mon, Feb 13, 2012 at 2:48 PM, Stephen Sanders wrote: > We've an application that logs data on one very large raid6 array > and updates/accesses a database on another smaller raid5 array. > > Both arrays are connected to the same PCIe 3ware RAID controller. =C2=A0 = The > system has 2 six core 3Ghz processors and 24 GB of RAM. =C2=A0The system = is > running FreeBSD 8.1. > > The averaged read/write rate to the database is 2MB/s while the averaged > write raid to the data =C2=A0logging array is 300MB/s. =C2=A0Writes to th= e logging > array are somewhat bursty. > > The problem we're encountering is that the disk subsystem appears to > 'pause' periodically. =C2=A0 It looks as if this is a result of disk read= /write > operations from the database array taking a very long time to complete > (up to 8 sec). > > When the disk read operation takes such a long time, it appears that the > system starts to run out of memory due to bio block buffering. =C2=A0Most > processes end up in either getblk() or waithighrunning(). > > We've instrumented g_vfs_strategie() and bufdone_finish() using dtrace. > The indication from this effort is that a number of reads and writes are > taking 4-8 seconds. > > So far, it looks as if the disk driver and hardware are OK as read/write > operations appear to be in the milli-second region. =C2=A0We believe that= our > instrumentation is pointing to something between the VFS layer and the > CAM as the culprit. > > We've gotten the same result from FreeBSD 8.2 but have not tried FreeBSD > 9 as yet. > > This scenario is not limited to a single system and is occurring on a > couple of systems. > > Does this sound familiar to anyone out there? > > Thanks Do you have a BBU on the 3ware device? It sounds very similar to effects we used to suffer when we used servers with twa without a BBU. Our newer servers use LSI/Dell PERC (with BBUs!), and don't have this sort of issue anymore. Cheers Tom