From owner-freebsd-stable@FreeBSD.ORG Thu Aug 5 05:35:06 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03FE51065672 for ; Thu, 5 Aug 2010 05:35:06 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta12.emeryville.ca.mail.comcast.net (qmta12.emeryville.ca.mail.comcast.net [76.96.27.227]) by mx1.freebsd.org (Postfix) with ESMTP id D8FEA8FC14 for ; Thu, 5 Aug 2010 05:35:05 +0000 (UTC) Received: from omta05.emeryville.ca.mail.comcast.net ([76.96.30.43]) by qmta12.emeryville.ca.mail.comcast.net with comcast id qUdf1e0040vp7WLACVb59q; Thu, 05 Aug 2010 05:35:05 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta05.emeryville.ca.mail.comcast.net with comcast id qVb41e0033LrwQ28RVb46Y; Thu, 05 Aug 2010 05:35:05 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4FC469B423; Wed, 4 Aug 2010 22:35:04 -0700 (PDT) Date: Wed, 4 Aug 2010 22:35:04 -0700 From: Jeremy Chadwick To: "Alex V. Petrov" Message-ID: <20100805053504.GA91196@icarus.home.lan> References: <201008042008.25679.alexvpetrov@gmail.com> <20100805041752.GA89285@icarus.home.lan> <201008051247.43260.alexvpetrov@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201008051247.43260.alexvpetrov@gmail.com> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-stable@freebsd.org Subject: Re: zpool - low speed write X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Aug 2010 05:35:06 -0000 On Thu, Aug 05, 2010 at 12:47:42PM +0800, Alex V. Petrov wrote: > > Your ada3 disk is different from the other two. Can you please provide > > the output from the following 3 commands? > > > > camcontrol identify ada2 > > camcontrol identify ada3 > > camcontrol identify ada4 > > > > > vfs.zfs.cache_flush_disable=1 > > > vfs.zfs.zil_disable=1 > > > > I question both of these settings, especially the latter. Please remove > > them both and re-test your write performance. > > I removed all settings of zfs. > Now it default. > > camcontrol identify ada2 > pass2: ATA-8 SATA 2.x device > pass2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > > protocol ATA/ATAPI-8 SATA 2.x > device model WDC WD10EADS-00M2B0 > firmware revision 01.00A01 > serial number WD-WCAV51709425 > WWN 50014ee2adf88aae > cylinders 16383 > heads 16 > sectors/track 63 > sector size logical 512, physical 512, offset 0 > LBA supported 268435455 sectors > LBA48 supported 1953525168 sectors > PIO supported PIO4 > DMA supported WDMA2 UDMA6 > > Feature Support Enable Value Vendor > read ahead yes yes > write cache yes yes > flush cache yes yes > overlap no > Tagged Command Queuing (TCQ) no no > Native Command Queuing (NCQ) yes 32 tags > SMART yes yes > microcode download yes yes > security yes no > power management yes yes > advanced power management no no 0/0x00 > automatic acoustic management yes no 254/0xFE 128/0x80 > media status notification no no > power-up in Standby yes no > write-read-verify no no 0/0x0 > unload no no > free-fall no no > data set management (TRIM) no > > ************* > > camcontrol identify ada3 > pass3: ATA-8 SATA 2.x device > pass3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > > protocol ATA/ATAPI-8 SATA 2.x > device model WDC WD10EADS-00L5B1 > firmware revision 01.01A01 > serial number WD-WCAU4D726772 > WWN 50014ee238ab988 > cylinders 16383 > heads 16 > sectors/track 63 > sector size logical 512, physical 512, offset 0 > LBA supported 268435455 sectors > LBA48 supported 1953525168 sectors > PIO supported PIO4 > DMA supported WDMA2 UDMA6 > > Feature Support Enable Value Vendor > read ahead yes yes > write cache yes yes > flush cache yes yes > overlap no > Tagged Command Queuing (TCQ) no no > Native Command Queuing (NCQ) yes 32 tags > SMART yes yes > microcode download yes yes > security yes no > power management yes yes > advanced power management no no 0/0x00 > automatic acoustic management yes no 254/0xFE 128/0x80 > media status notification no no > power-up in Standby yes no > write-read-verify no no 0/0x0 > unload no no > free-fall no no > data set management (TRIM) no > > ************* > > camcontrol identify ada4 > pass4: ATA-8 SATA 2.x device > pass4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > > protocol ATA/ATAPI-8 SATA 2.x > device model WDC WD10EADS-00M2B0 > firmware revision 01.00A01 > serial number WD-WMAV50095864 > WWN 50014ee014f3265 > cylinders 16383 > heads 16 > sectors/track 63 > sector size logical 512, physical 512, offset 0 > LBA supported 268435455 sectors > LBA48 supported 1953525168 sectors > PIO supported PIO4 > DMA supported WDMA2 UDMA6 > > Feature Support Enable Value Vendor > read ahead yes yes > write cache yes yes > flush cache yes yes > overlap no > Tagged Command Queuing (TCQ) no no > Native Command Queuing (NCQ) yes 32 tags > SMART yes yes > microcode download yes yes > security yes no > power management yes yes > advanced power management no no 0/0x00 > automatic acoustic management yes no 254/0xFE 128/0x80 > media status notification no no > power-up in Standby yes no > write-read-verify no no 0/0x0 > unload no no > free-fall no no > data set management (TRIM) no > > ************* > > dd if=/dev/zero of=/tank/test.zero bs=3M count=1000 > 1000+0 records in > 1000+0 records out > 3145728000 bytes transferred in 527.727573 secs (5960894 bytes/sec) > > During execution dd: > > zpool iostat -v 10 10 > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 48 25,4K 4,35M > ada4 598G 333G 0 16 9,94K 1,56M > ada2 633G 295G 0 16 7,26K 1,54M > ada3 727G 201G 0 14 8,21K 1,25M > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 29 0 2,28M > ada4 598G 333G 0 10 0 870K > ada2 633G 295G 0 8 0 774K > ada3 727G 201G 0 9 0 689K > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 14 0 91,8K > ada4 598G 333G 0 4 0 25,2K > ada2 633G 295G 0 5 0 28,6K > ada3 727G 201G 0 4 0 38,0K > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 19 0 227K > ada4 598G 333G 0 6 0 75,3K > ada2 633G 295G 0 6 0 76,6K > ada3 727G 201G 0 6 0 75,3K > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 25 0 2,27M > ada4 598G 333G 0 9 0 877K > ada2 633G 295G 0 8 0 787K > ada3 727G 201G 0 7 0 659K > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 29 12,8K 2,51M > ada4 598G 333G 0 11 0 1,07M > ada2 633G 295G 0 9 0 750K > ada3 727G 201G 0 8 12,8K 724K > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 37 0 2,53M > ada4 598G 333G 0 12 0 847K > ada2 633G 295G 0 13 0 986K > ada3 727G 201G 0 11 0 759K > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 25 0 2,39M > ada4 598G 333G 0 9 0 915K > ada2 633G 295G 0 8 0 900K > ada3 727G 201G 0 7 0 632K > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 28 0 1,76M > ada4 598G 333G 0 11 0 919K > ada2 633G 295G 0 7 0 207K > ada3 727G 201G 0 9 0 676K > ---------- ----- ----- ----- ----- ----- ----- > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > tank 1,91T 829G 0 26 0 2,16M > ada4 598G 333G 0 10 0 856K > ada2 633G 295G 0 7 0 646K > ada3 727G 201G 0 8 0 711K > ---------- ----- ----- ----- ----- ----- ----- Write performance here is abysmal, agreed. This is very odd. I hate to say this, but can you remove ahci.ko (ahci_load="yes") from your loader.conf and reboot? You may need to change filesystem names around in /etc/fstab for your OS disk (assuming it's on ada0), but for ZFS it should just magically find the disks on adXX. If you could also provide pciconf -lvc output that would be helpful. Thanks. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |