From owner-freebsd-current@FreeBSD.ORG Fri Jan 9 15:55:17 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D472C16A4CE for ; Fri, 9 Jan 2004 15:55:17 -0800 (PST) Received: from smtp.mho.com (smtp.mho.net [64.58.4.6]) by mx1.FreeBSD.org (Postfix) with SMTP id 29C5443D41 for ; Fri, 9 Jan 2004 15:55:12 -0800 (PST) (envelope-from scottl@freebsd.org) Received: (qmail 64278 invoked by uid 1002); 9 Jan 2004 23:55:11 -0000 Received: from unknown (HELO freebsd.org) (64.58.1.252) by smtp.mho.net with SMTP; 9 Jan 2004 23:55:11 -0000 Message-ID: <3FFF3EFA.5020208@freebsd.org> Date: Fri, 09 Jan 2004 16:53:30 -0700 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.5) Gecko/20031103 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Doug White References: <200401091231.i09CVdBG049283@robert2.mpe-garching.mpg.de> <20040109152046.F24587@carver.gumbysoft.com> In-Reply-To: <20040109152046.F24587@carver.gumbysoft.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-current@freebsd.org Subject: Re: writing to 3ware escalade uses up 100% cpu time in system. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2004 23:55:17 -0000 Doug White wrote: > On Fri, 9 Jan 2004, Klaus Robert Suetterlin wrote: > > >>Hello, >> >>I'm using a 3ware 8506-4LP controller with 4 250GB harddrives in >>Raid 0. All space is in a single slice, separated into 1GB swap and >>the rest into twed0s1d which is mounted under /space... >> >>When I do >> ``dd if=/dev/zero of=/space/test bs=1000000 count=10000'' >>the twed0 maxes out at 70MB/sec and (according to systat) 70% usage. >>Unfortunately my system goes to 100% cpu usage at the same time. > > > What speed of bus is the 3ware attached to? > > You might run 'systat -vmstat 2' and check the amount of CPU going to > interrupts. I suspect you're maxing out on the interrupt handler. Also > look at the # of interrupts to the twe. > The interrupt handler for the driver is actually pretty lean; most real work is offloaded to a taskqueue. However, the driver as a whole is not locked, so it's going to have contention on the Giant mutex. He might well get some better performance just my making the interrupt handler be INTR_MPSAFE. Unfortunately, this requires a bit of work and I don't have any hardware to test on. Another cheap hack to help performance would be to remove the INTR_ENTROPY flag from the call to bus_setup_intr() in twe_freebsd.c. This will affect the quality of your entropy a bit, but it would be an interesting test. One issue that might be at play here is that the 3ware hardware has some limits on DMA alignment for data buffers. If the buffers don't meet those limits, then the driver has to manually allocate and copy temporary buffers. The driver doesn't seem to keep any stats on this, but it would be quite useful if it did. Other tests to try would include doing I/O directly to the device, and turning off softupdates, especially if you're doing lots of file operations. Scott