From owner-freebsd-current@FreeBSD.ORG Thu Jan 15 04:44:20 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5A76316A4CE; Thu, 15 Jan 2004 04:44:20 -0800 (PST) Received: from mpehp1.mpe-garching.mpg.de (mpehp1.mpe-garching.mpg.de [130.183.70.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9D25F43D60; Thu, 15 Jan 2004 04:44:17 -0800 (PST) (envelope-from krs@robert2.mpe-garching.mpg.de) Received: from robert2.mpe-garching.mpg.de (robert2.mpe-garching.mpg.de [130.183.136.59])with ESMTP id NAA19778; Thu, 15 Jan 2004 13:44:13 +0100 (MET) Received: from robert2.mpe-garching.mpg.de (localhost.mpe-garching.mpg.de [127.0.0.1])i0FCiYHd000889; Thu, 15 Jan 2004 13:44:34 +0100 (CET) (envelope-from krs@robert2.mpe-garching.mpg.de) Message-Id: <200401151244.i0FCiYHd000889@robert2.mpe-garching.mpg.de> To: Scott Long , "Subhro" , Doug White In-Reply-To: Your message of "Fri, 09 Jan 2004 16:53:30 MST." <3FFF3EFA.5020208@freebsd.org> Date: Thu, 15 Jan 2004 13:44:34 +0100 From: Klaus Robert Suetterlin cc: freebsd-current@freebsd.org Subject: (long) Re: writing to 3ware escalade uses up 100% cpu time in system. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: robert@mpe.mpg.de List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 12:44:20 -0000 Hi, I'll try to provide more information on the system I use and on the problem I see along the line of three emails I recieved so far. My system is a P3-933 Compaq Proliant ML350, with 512MB of Ram. Using software raid (i.e. ccd) on 4 scsi-2 or ultra ata-100 disks I can get 80MByte/sec easily, without any significant cpu usage. Unfortunately ultra ata cable length and scsi prices are too limiting for my storage needs (1GByte in 4 Disks), thus I opted for a serial ata raid controller. "Subhro" asked: > Can you post this? > > sysctl -a |grep dma vfs.nfs.iodmaxidle: 120 vfs.nfs.iodmax: 20 hw.ata.ata_dma: 1 hw.ata.atapi_dma: 0 But I do not see the point here. twe(4) does not list any sysctl variables for dma, so I'd expect to find none. ``sysctl -a|grep twe'' gives kern.disks: twed0 ad0 twe commands 255 16K 16K 255 64 hw.twe0.driver_version: 1.50.00.000 Doug White wrote: > What speed of bus is the 3ware attached to? Unfortunately my system only has a 33MHz 64Bit PCI Slot. The Adapter I use supports 66MHz. > You might run 'systat -vmstat 2' and check the amount of CPU going to > interrupts. I suspect you're maxing out on the interrupt handler. Also > look at the # of interrupts to the twe. According to ``systat -vmstat 2'' my cpu is used 70-90% by System (the exact number depends on the amount of free memory), 2% by Interrupt. The number of interrupts in twe is approx. 270. The twe is serving 570 tps with 128KB/t for a grand total of 70MB/s and is 75% busy. I get a total of 500 Interrupts and the cpu is at 8%-30% Idle. Cpu usage depends on the ammount of free memory. In the beginning (free memory available) cpu is at approx. 30% Idle, after the first few GByte of transferred data cpu is at 10% Idle. > What kind of workload are you expecting to put on? Its very, very rare > that you would be getting large sequential writes like those you are > getting with dd. I expect linear writes of 200 GByte at 10 - 80 MByte / sec. Scott Long comments: > The interrupt handler for the driver is actually pretty lean; most real > work is offloaded to a taskqueue. However, the driver as a whole is not > locked, so it's going to have contention on the Giant mutex. He might > well get some better performance just my making the interrupt handler > be INTR_MPSAFE. Unfortunately, this requires a bit of work and I don't > have any hardware to test on. Sorry I had no time and no expertise to investigate this. > Another cheap hack to help performance would be to remove the > INTR_ENTROPY flag from the call to bus_setup_intr() in twe_freebsd.c. > This will affect the quality of your entropy a bit, but it would be an > interesting test. I will do this today, if I find out what it means :). > One issue that might be at play here is that the 3ware hardware has some > limits on DMA alignment for data buffers. If the buffers don't meet > those limits, then the driver has to manually allocate and copy > temporary buffers. The driver doesn't seem to keep any stats on this, > but it would be quite useful if it did. According to twe(4) this is only a problem when userland tries to access the raw media... I guess this is because kernel buffercache uses aligned buffers only? > Other tests to try would include doing I/O directly to the device, and > turning off softupdates, especially if you're doing lots of file > operations. Huh? I don't understand how this would help with throughput. Regards, Robert S.