From owner-freebsd-performance@FreeBSD.ORG Tue Aug 17 20:13:17 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3D1016A4CE; Tue, 17 Aug 2004 20:13:17 +0000 (GMT) Received: from gnosis.realityengine.ca (gnosis.realityengine.ca [69.55.224.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8789743D2F; Tue, 17 Aug 2004 20:13:17 +0000 (GMT) (envelope-from erik.rothwell@realityengine.ca) Received: from [192.168.5.3] (unknown [216.58.89.253]) by gnosis.realityengine.ca (Postfix) with ESMTP id 7D8EB9EF9; Tue, 17 Aug 2004 20:13:15 +0000 (UTC) User-Agent: Microsoft-Entourage/11.0.0.040405 Date: Tue, 17 Aug 2004 16:13:10 -0400 From: Erik Rothwell To: Robert Watson Message-ID: In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: Network performance issues when writing to disk (5.2.1-RELEASE) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Aug 2004 20:13:17 -0000 On 8/12/04 11:17 AM, "Robert Watson" wrote: > Did you replace it with another if_dc card, or with a card using a > different interface driver? Well, I took the afflicted box out of production this weekend for some further testing. After swapping out one of the if_dc NICs with an Intel card using the if_fxp driver, the problem no longer occurs on that interface. I can only reproduce the problem using an interface with the if_dc driver. The difference in performance is night and day. > - What sort of kernel configuration are you using -- GENERIC, or a custom > kernel, and if so, is it an SMP box and is SMP enabled in the kernel? This is a single processor box. The kernel is compiled without SMP or APIC and without a number of unused drivers (SCSI, ISA NICs, PCMCIA, umass, &c). IPFIREWALL, IPDIVERT, QUOTA and DEVICE_POLLING are compiled in. (Device polling is not currently enabled via the sysctl however). > - When the system performance suddenly degrades, what is the apparent load > and condition of the system? In particular, at that point if you > measure the load average and CPU utilization, perhaps with "systat > -vmstat" or "top", are the CPU(s) maxed out? How much time is in user > vs. system vs. interrupt (vs. idle). if_dc: nc > /dev/null Load 0.79 0.25 0.13 CPU 41.5% sys, 28.5% interrupt, 5.5% user, 24.5% idle Interrupts 2501 clock, 128 rtc, 11684 dc0, 83 ata Disk ad0: 12.33 KB/t 4 tps 0.04 MB/s Disk ad1: 7.80 KB/t 80 tps 0.61 MB/s 8% busy if_dc: nc > /data/junk Load 0.11 0.26 0.17 CPU 9.0% sys, 2.0% interrupt, 0.5% user, 88.5% idle Interrupts 2504 clock, 128 rtc, 434 dc0, 54 ata Disk ad0: - Disk ad1: 11.75 KB/t 51 tps 0.58 MB/s 100% busy if_fxp: nc > /data/junk Load 0.42 0.38 0.23 CPU 61.0% sys, 31.2% interrupt, 2.5% user, 4.5% idle Interrupts 2501 clock, 128 rtc, 6036 fxp0, 145 ata Disk ad0: - Disk ad1: 128 KB/t 73 tps 9.07 MB/s 37% busy Plenty of free memory, little if any swapping at any time. Both disks are running in udma66. devinfo/dmesg doesn't indicate any conflict between disk controller and the network interfaces. There certainly appears to be some contention between the if_dc cards and the disk, however. Erik. From owner-freebsd-performance@FreeBSD.ORG Fri Aug 20 19:35:52 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EDAEB16A4CE; Fri, 20 Aug 2004 19:35:51 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6C92043D2D; Fri, 20 Aug 2004 19:35:50 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 50401ACAFE; Fri, 20 Aug 2004 21:35:47 +0200 (CEST) Date: Fri, 20 Aug 2004 21:35:47 +0200 From: Pawel Jakub Dawidek To: Greg 'groggy' Lehey Message-ID: <20040820193547.GZ30151@darkness.comp.waw.pl> References: <20040817132740.GA32139@freebie.xs4all.nl> <41449.1092750244@critter.freebsd.dk> <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2bJ57vwr75KGnr5s" Content-Disposition: inline In-Reply-To: <20040819024359.GA85432@wantadilla.lemis.com> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: src-committers@FreeBSD.ORG cc: cvs-src@FreeBSD.ORG cc: cvs-all@FreeBSD.ORG cc: Poul-Henning Kamp cc: Wilko Bulte cc: freebsd-performance@freebsd.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2004 19:35:52 -0000 --2bJ57vwr75KGnr5s Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 19, 2004 at 12:13:59PM +0930, Greg 'groggy' Lehey wrote: +> On Tuesday, 17 August 2004 at 15:16:12 +0200, Pawel Jakub Dawidek wrote: +> > On Tue, Aug 17, 2004 at 10:10:20PM +0930, Greg 'groggy' Lehey wrote: +> > +> On the contrary. RAID-3 requires byte-level striping, which is +> > +> ridiculous on the hardware that FreeBSD supports. [...] +> > Want to compare performance with vinum's RAID5?:) +>=20 +> Feel free. But do it with more than a single process accessing the +> disks. Tests were done using this HW: CPU: Pentium II/Pentium II Xeon/Celeron (534.55-MHz 686-class CPU) Origin =3D "GenuineIntel" Id =3D 0x665 Stepping =3D 5 Features=3D0x183fbff real memory =3D 67108864 (64 MB) avail memory =3D 60256256 (57 MB) MPTable: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs [...] iir0: mem 0xd8100000-0xd8103fff irq 17 at devic= e 13.0 on pci0 [...] da0 at iir0 bus 2 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device=20 da0: Tagged Queueing Enabled da0: 8675MB (17767890 512 byte sectors: 255H 63S/T 1106C) da1 at iir0 bus 2 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device=20 da1: Tagged Queueing Enabled da1: 8675MB (17767890 512 byte sectors: 255H 63S/T 1106C) da2 at iir0 bus 2 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device=20 da2: Tagged Queueing Enabled da2: 8675MB (17767890 512 byte sectors: 255H 63S/T 1106C) The software you can find under /usr/src/tools/tools/raidtest/ in HEAD branch. Test has been done for 10000 random requests (offset, size and operation type was random). RAID3: Number of Bytes per Requests per Operations processes second second ---------------------------------------------------------------------- READ 3 6329500 95 READ 15 8981047 135 READ 100 10719314 161 WRITE 3 5073263 76 WRITE 15 7467387 112 WRITE 100 8631136 129 READ/WRITE 3 6041795 90 READ/WRITE 15 8104847 121 READ/WRITE 100 9494250 142 RAID5: Number of Bytes per Requests per Operations processes second second ---------------------------------------------------------------------- READ 3 6041795 90 READ 15 14768833 222 READ 100 19546985 294 WRITE 3 3479568 52 WRITE 15 5403231 81 WRITE 100 6211191 93 READ/WRITE 3 4521071 68 READ/WRITE 15 7911875 119 READ/WRITE 100 9360528 140 As you can see RAID5 is only faster in READ tests for 15 and 100 processes working in parallel. As I can see, RAID3 is faster in all the rest tests. Of course you are welcome to try by yourself. Anyway, if I can ask for something. Think twice before calling something ridiculous without understanding. PS. I wonder about read optimization, so parity component can be also used for reading in round-robin fashion... --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --2bJ57vwr75KGnr5s Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBJlKTForvXbEpPzQRAsw9AJ4iJ0Hhuuxnjz31dXV+ofw53x1SdACg8jfa f510URPthDkHrRA/rpIUwX0= =QqXC -----END PGP SIGNATURE----- --2bJ57vwr75KGnr5s-- From owner-freebsd-performance@FreeBSD.ORG Fri Aug 20 19:53:20 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C176916A4CE; Fri, 20 Aug 2004 19:53:20 +0000 (GMT) Received: from rms06.rommon.net (rms06.rommon.net [212.54.5.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id DE66443D46; Fri, 20 Aug 2004 19:53:19 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [193.64.42.134] (h86.vuokselantie10.fi [193.64.42.134]) by rms06.rommon.net (Postfix) with ESMTP id F079933C1B; Fri, 20 Aug 2004 22:53:17 +0300 (EEST) Message-ID: <412656AE.5020605@he.iki.fi> Date: Fri, 20 Aug 2004 22:53:18 +0300 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040803 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20040817132740.GA32139@freebie.xs4all.nl> <41449.1092750244@critter.freebsd.dk> <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> In-Reply-To: <20040820193547.GZ30151@darkness.comp.waw.pl> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: src-committers@freebsd.org cc: cvs-src@freebsd.org cc: cvs-all@freebsd.org cc: Poul-Henning Kamp cc: Greg 'groggy' Lehey cc: Wilko Bulte cc: freebsd-performance@freebsd.org Subject: Re: RAID-3? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2004 19:53:21 -0000 Pawel Jakub Dawidek wrote: >The software you can find under /usr/src/tools/tools/raidtest/ in HEAD >branch. >Test has been done for 10000 random requests (offset, size and operation >type was random). > > Would it make sense to have known initialization vector for the random function so a specific test pattern could be reproduced at will? This would also reduce the need for multiple repeats for a test to get good repeatable results. Pete From owner-freebsd-performance@FreeBSD.ORG Fri Aug 20 19:58:55 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5A07116A4DB; Fri, 20 Aug 2004 19:58:52 +0000 (GMT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id B16A443D2F; Fri, 20 Aug 2004 19:58:51 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.1/8.13.1) with ESMTP id i7KJwm1O024880; Fri, 20 Aug 2004 21:58:48 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Pawel Jakub Dawidek From: "Poul-Henning Kamp" In-Reply-To: Your message of "Fri, 20 Aug 2004 21:35:47 +0200." <20040820193547.GZ30151@darkness.comp.waw.pl> Date: Fri, 20 Aug 2004 21:58:48 +0200 Message-ID: <24879.1093031928@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: src-committers@FreeBSD.org cc: cvs-src@FreeBSD.org cc: cvs-all@FreeBSD.org cc: Greg 'groggy' Lehey cc: Wilko Bulte cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2004 19:58:55 -0000 In message <20040820193547.GZ30151@darkness.comp.waw.pl>, Pawel Jakub Dawidek w rites: >As you can see RAID5 is only faster in READ tests for 15 and 100 processes >working in parallel. As I can see, RAID3 is faster in all the rest tests. > >PS. I wonder about read optimization, so parity component can be also > used for reading in round-robin fashion... I would far rather have an option to give me data-integrity checking ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Fri Aug 20 20:45:02 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 601A816A4CE; Fri, 20 Aug 2004 20:45:02 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id D9E3443D49; Fri, 20 Aug 2004 20:45:01 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 12761ACAF1; Fri, 20 Aug 2004 22:44:58 +0200 (CEST) Date: Fri, 20 Aug 2004 22:44:58 +0200 From: Pawel Jakub Dawidek To: Petri Helenius Message-ID: <20040820204458.GA30151@darkness.comp.waw.pl> References: <41449.1092750244@critter.freebsd.dk> <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> <412656AE.5020605@he.iki.fi> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="FMJTF8LVhUQkvsEb" Content-Disposition: inline In-Reply-To: <412656AE.5020605@he.iki.fi> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: src-committers@freebsd.org cc: cvs-src@freebsd.org cc: cvs-all@freebsd.org cc: Poul-Henning Kamp cc: Greg 'groggy' Lehey cc: Wilko Bulte cc: freebsd-performance@freebsd.org Subject: Re: RAID-3? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2004 20:45:02 -0000 --FMJTF8LVhUQkvsEb Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 20, 2004 at 10:53:18PM +0300, Petri Helenius wrote: +> Pawel Jakub Dawidek wrote: +>=20 +> >The software you can find under /usr/src/tools/tools/raidtest/ in HEAD +> >branch. +> >Test has been done for 10000 random requests (offset, size and operation +> >type was random). +> >=20 +> > +> Would it make sense to have known initialization vector for the random= =20 +> function so a specific test pattern could be reproduced at will? This=20 +> would also reduce the need for multiple repeats for a test to get good= =20 +> repeatable results. Raidtest tool works like this, i.e. first you have to generate file with random data and then use this file for every test. --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --FMJTF8LVhUQkvsEb Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBJmLKForvXbEpPzQRAojWAJ90e1CUZoUUIedv2InK+wxvomCgeQCfZUB5 +fCMXAHxVy/jX7GSuwFH4kI= =sMxE -----END PGP SIGNATURE----- --FMJTF8LVhUQkvsEb-- From owner-freebsd-performance@FreeBSD.ORG Fri Aug 20 20:45:43 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6ABD416A4CE; Fri, 20 Aug 2004 20:45:43 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id 571E043D45; Fri, 20 Aug 2004 20:45:42 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 6076FACAFE; Fri, 20 Aug 2004 22:45:39 +0200 (CEST) Date: Fri, 20 Aug 2004 22:45:39 +0200 From: Pawel Jakub Dawidek To: Poul-Henning Kamp Message-ID: <20040820204539.GB30151@darkness.comp.waw.pl> References: <20040820193547.GZ30151@darkness.comp.waw.pl> <24879.1093031928@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="G2kvLHdEX2DcGdqq" Content-Disposition: inline In-Reply-To: <24879.1093031928@critter.freebsd.dk> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: src-committers@FreeBSD.org cc: cvs-src@FreeBSD.org cc: cvs-all@FreeBSD.org cc: Greg 'groggy' Lehey cc: Wilko Bulte cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2004 20:45:43 -0000 --G2kvLHdEX2DcGdqq Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 20, 2004 at 09:58:48PM +0200, Poul-Henning Kamp wrote: +> In message <20040820193547.GZ30151@darkness.comp.waw.pl>, Pawel Jakub Da= widek w +> rites: +>=20 +> >As you can see RAID5 is only faster in READ tests for 15 and 100 proces= ses +> >working in parallel. As I can see, RAID3 is faster in all the rest test= s. +> > +> >PS. I wonder about read optimization, so parity component can be also +> > used for reading in round-robin fashion... +>=20 +> I would far rather have an option to give me data-integrity checking ? Yes, I'm wondering about this as well... --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --G2kvLHdEX2DcGdqq Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBJmLzForvXbEpPzQRAsQHAKCEne0ONkZoQEefxiI3uxp0tcvrrACgyhKp wEtVo9W/KNwLO4stK6MgP0w= =HGXH -----END PGP SIGNATURE----- --G2kvLHdEX2DcGdqq-- From owner-freebsd-performance@FreeBSD.ORG Fri Aug 20 22:35:12 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 81FC416A4CE; Fri, 20 Aug 2004 22:35:12 +0000 (GMT) Received: from panzer.kdm.org (panzer.kdm.org [216.160.178.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id F0D4043D39; Fri, 20 Aug 2004 22:35:11 +0000 (GMT) (envelope-from ken@panzer.kdm.org) Received: from panzer.kdm.org (localhost [127.0.0.1]) by panzer.kdm.org (8.12.9/8.12.5) with ESMTP id i7KMYlui019034; Fri, 20 Aug 2004 16:34:47 -0600 (MDT) (envelope-from ken@panzer.kdm.org) Received: (from ken@localhost) by panzer.kdm.org (8.12.9/8.12.5/Submit) id i7KMYlPQ019033; Fri, 20 Aug 2004 16:34:47 -0600 (MDT) (envelope-from ken) Date: Fri, 20 Aug 2004 16:34:47 -0600 From: "Kenneth D. Merry" To: Pawel Jakub Dawidek Message-ID: <20040820223446.GA18838@panzer.kdm.org> References: <20040817132740.GA32139@freebie.xs4all.nl> <41449.1092750244@critter.freebsd.dk> <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040820193547.GZ30151@darkness.comp.waw.pl> User-Agent: Mutt/1.4.1i X-Mailman-Approved-At: Sat, 21 Aug 2004 02:58:46 +0000 cc: src-committers@FreeBSD.org cc: cvs-src@FreeBSD.org cc: cvs-all@FreeBSD.org cc: Poul-Henning Kamp cc: Greg 'groggy' Lehey cc: Wilko Bulte cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2004 22:35:12 -0000 On Fri, Aug 20, 2004 at 21:35:47 +0200, Pawel Jakub Dawidek wrote: > As you can see RAID5 is only faster in READ tests for 15 and 100 processes > working in parallel. As I can see, RAID3 is faster in all the rest tests. > > Of course you are welcome to try by yourself. > > Anyway, if I can ask for something. Think twice before calling something > ridiculous without understanding. > > PS. I wonder about read optimization, so parity component can be also > used for reading in round-robin fashion... That likely wouldn't speed things up too much. The hard drives are doing read ahead anyway, so they'll usually have the data ready when you go down to read the next block if you're doing sequential reads. You would also spend more CPU power to reconstruct the piece of data you didn't read from the disk. That'll probably increase your latency somewhat, and you would also be touching all of the data with the CPU. As PHK said, it might be more interesting to do data integrity checking on reads. The problem, of course, is that you wouldn't be able to correct problems, you would only be able to detect them. Ken -- Kenneth Merry ken@kdm.org From owner-freebsd-performance@FreeBSD.ORG Fri Aug 20 22:47:17 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 711C016A4CF; Fri, 20 Aug 2004 22:47:17 +0000 (GMT) Received: from harmony.village.org (rover.village.org [168.103.84.182]) by mx1.FreeBSD.org (Postfix) with ESMTP id DA7AC43D46; Fri, 20 Aug 2004 22:47:16 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.11/8.12.11) with ESMTP id i7KMk85l028152; Fri, 20 Aug 2004 16:46:08 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Fri, 20 Aug 2004 16:46:17 -0600 (MDT) Message-Id: <20040820.164617.17747402.imp@bsdimp.com> To: pjd@FreeBSD.org From: "M. Warner Losh" In-Reply-To: <20040820193547.GZ30151@darkness.comp.waw.pl> References: <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 21 Aug 2004 02:58:46 +0000 cc: src-committers@FreeBSD.org cc: cvs-src@FreeBSD.org cc: cvs-all@FreeBSD.org cc: phk@phk.freebsd.dk cc: grog@FreeBSD.org cc: wb@freebie.xs4all.nl cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2004 22:47:17 -0000 I'll reiterate what I said earlier: This is not the list to talk about RAID-3, its definition, its suitibility or performance issues with RAID-3. Please take these discussions elsewhere. Thanks Warner From owner-freebsd-performance@FreeBSD.ORG Sat Aug 21 08:06:16 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 438EC16A4CE; Sat, 21 Aug 2004 08:06:16 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id D050243D39; Sat, 21 Aug 2004 08:06:15 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 43A9BACAFE; Sat, 21 Aug 2004 10:06:14 +0200 (CEST) Date: Sat, 21 Aug 2004 10:06:14 +0200 From: Pawel Jakub Dawidek To: "Kenneth D. Merry" Message-ID: <20040821080614.GC30151@darkness.comp.waw.pl> References: <41449.1092750244@critter.freebsd.dk> <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> <20040820223446.GA18838@panzer.kdm.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="f9lFb+Z4UT82L8vr" Content-Disposition: inline In-Reply-To: <20040820223446.GA18838@panzer.kdm.org> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: Greg 'groggy' Lehey cc: Wilko Bulte cc: Poul-Henning Kamp cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Aug 2004 08:06:16 -0000 --f9lFb+Z4UT82L8vr Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 20, 2004 at 04:34:47PM -0600, Kenneth D. Merry wrote: +> > PS. I wonder about read optimization, so parity component can be also +> > used for reading in round-robin fashion... +>=20 +> That likely wouldn't speed things up too much. The hard drives are doing +> read ahead anyway, so they'll usually have the data ready when you go do= wn +> to read the next block if you're doing sequential reads. +>=20 +> You would also spend more CPU power to reconstruct the piece of data you +> didn't read from the disk. That'll probably increase your latency +> somewhat, and you would also be touching all of the data with the CPU. +>=20 +> As PHK said, it might be more interesting to do data integrity checking = on +> reads. The problem, of course, is that you wouldn't be able to correct +> problems, you would only be able to detect them. Those are results from RAID3 test without one data component, so parity has to be calculated. RAID3 in degraded mode: Number of Bytes per Requests per Operations processes second second ---------------------------------------------------------------------- READ 3 6329500 95 READ 15 9104075 136 READ 100 10895041 163 WRITE 3 5112288 76 WRITE 15 7911875 119 WRITE 100 9104075 136 READ/WRITE 3 6097224 91 READ/WRITE 15 8307468 125 READ/WRITE 100 9773492 147 Here are already presented results of full RAID3: RAID3: Number of Bytes per Requests per Operations processes second second ---------------------------------------------------------------------- READ 3 6329500 95 READ 15 8981047 135 READ 100 10719314 161 WRITE 3 5073263 76 WRITE 15 7467387 112 WRITE 100 8631136 129 READ/WRITE 3 6041795 90 READ/WRITE 15 8104847 121 READ/WRITE 100 9494250 142 So as you can see there are no difference in reading (actually reading in degraded mode is even a bit faster, hard to say why). Thread, which was responsible for XOR calculations was consuming 6-7% of CPU on 534MHz Celeron. PS. I removed cvs-*@ from CC. Let's continue on performance@. --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --f9lFb+Z4UT82L8vr Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBJwJ2ForvXbEpPzQRAnhcAJ9ZpCcFwICzp5WELvGhRK/63uMuBQCdEkRy NXosbN6Lcr6Wh7Iy4MvPkho= =nh6l -----END PGP SIGNATURE----- --f9lFb+Z4UT82L8vr-- From owner-freebsd-performance@FreeBSD.ORG Sat Aug 21 08:28:46 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5DAB716A4CE; Sat, 21 Aug 2004 08:28:46 +0000 (GMT) Received: from ozlabs.org (ozlabs.org [203.10.76.45]) by mx1.FreeBSD.org (Postfix) with ESMTP id D287243D31; Sat, 21 Aug 2004 08:28:44 +0000 (GMT) (envelope-from grog@lemis.com) Received: from blackwater.lemis.com (blackwater.lemis.com [192.109.197.80]) by ozlabs.org (Postfix) with ESMTP id 195C52BDEC; Sat, 21 Aug 2004 18:28:41 +1000 (EST) Received: by blackwater.lemis.com (Postfix, from userid 1004) id BEB4151201; Sat, 21 Aug 2004 17:58:38 +0930 (CST) Date: Sat, 21 Aug 2004 17:58:38 +0930 From: Greg 'groggy' Lehey To: Pawel Jakub Dawidek Message-ID: <20040821082838.GF92256@wantadilla.lemis.com> References: <20040817132740.GA32139@freebie.xs4all.nl> <41449.1092750244@critter.freebsd.dk> <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="J5MfuwkIyy7RmF4Q" Content-Disposition: inline In-Reply-To: <20040820193547.GZ30151@darkness.comp.waw.pl> User-Agent: Mutt/1.4.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 cc: Wilko Bulte cc: Poul-Henning Kamp cc: src-committers@FreeBSD.ORG cc: freebsd-performance@freebsd.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Aug 2004 08:28:46 -0000 --J5MfuwkIyy7RmF4Q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline [Removing cvs-src and cvs-all] On Friday, 20 August 2004 at 21:35:47 +0200, Pawel Jakub Dawidek wrote: > On Thu, Aug 19, 2004 at 12:13:59PM +0930, Greg 'groggy' Lehey wrote: > +> On Tuesday, 17 August 2004 at 15:16:12 +0200, Pawel Jakub Dawidek wrote: > +> > On Tue, Aug 17, 2004 at 10:10:20PM +0930, Greg 'groggy' Lehey wrote: > +> > +> On the contrary. RAID-3 requires byte-level striping, which is > +> > +> ridiculous on the hardware that FreeBSD supports. > [...] > +> > Want to compare performance with vinum's RAID5?:) > +> > +> Feel free. But do it with more than a single process accessing the > +> disks. > > Tests were done using this HW: > > da0 at iir0 bus 2 target 0 lun 0 > da0: Fixed Direct Access SCSI-2 device > da0: Tagged Queueing Enabled > da0: 8675MB (17767890 512 byte sectors: 255H 63S/T 1106C) > da1 at iir0 bus 2 target 1 lun 0 > da1: Fixed Direct Access SCSI-2 device > da1: Tagged Queueing Enabled > da1: 8675MB (17767890 512 byte sectors: 255H 63S/T 1106C) > da2 at iir0 bus 2 target 2 lun 0 > da2: Fixed Direct Access SCSI-2 device > da2: Tagged Queueing Enabled > da2: 8675MB (17767890 512 byte sectors: 255H 63S/T 1106C) So these are two disks + parity, right? That's not exactly a typical setup. > Test has been done for 10000 random requests (offset, size and operation > type was random). > > RAID3: > Number of Bytes per Requests per > Operations processes second second > ---------------------------------------------------------------------- > READ 3 6329500 95 > READ 15 8981047 135 > READ 100 10719314 161 > WRITE 3 5073263 76 > WRITE 15 7467387 112 > WRITE 100 8631136 129 > READ/WRITE 3 6041795 90 > READ/WRITE 15 8104847 121 > READ/WRITE 100 9494250 142 > > RAID5: What was the RAID-5 stripe size? > Number of Bytes per Requests per > Operations processes second second > ---------------------------------------------------------------------- > READ 3 6041795 90 > READ 15 14768833 222 > READ 100 19546985 294 > WRITE 3 3479568 52 > WRITE 15 5403231 81 > WRITE 100 6211191 93 > READ/WRITE 3 4521071 68 > READ/WRITE 15 7911875 119 > READ/WRITE 100 9360528 140 > > As you can see RAID5 is only faster in READ tests for 15 and 100 processes > working in parallel. As I can see, RAID3 is faster in all the rest tests. I don't really see enough to convince me either way. If you use small RAID-5 stripes, then yes, it's possible to get better performance from RAID-3. I'd also suggest that your figures would look very different with five or nine disks. It would also be interested to see the results of rawio on these configurations, and also the relative performance of a single disk. > Of course you are welcome to try by yourself. Yes, of course, but I don't have time > Anyway, if I can ask for something. Think twice before calling > something ridiculous without understanding. I'm sorry if I upset you, but I'm still by no means convinced of the usefulness of RAID-3. Greg -- Note: I discard all HTML mail unseen. Finger grog@FreeBSD.org for PGP public key. See complete headers for address and phone numbers. --J5MfuwkIyy7RmF4Q Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (FreeBSD) iD8DBQFBJwe2IubykFB6QiMRAuVfAJ45iUxaMzzyIM+JlEMA4maRLAfXfwCeKIw8 i3qeLGWGOOkP3ttbz5pH6gk= =tJAf -----END PGP SIGNATURE----- --J5MfuwkIyy7RmF4Q-- From owner-freebsd-performance@FreeBSD.ORG Sat Aug 21 08:32:11 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A096116A4CE; Sat, 21 Aug 2004 08:32:11 +0000 (GMT) Received: from ozlabs.org (ozlabs.org [203.10.76.45]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60FE443D45; Sat, 21 Aug 2004 08:32:11 +0000 (GMT) (envelope-from grog@lemis.com) Received: from blackwater.lemis.com (blackwater.lemis.com [192.109.197.80]) by ozlabs.org (Postfix) with ESMTP id A19002BD68; Sat, 21 Aug 2004 18:32:09 +1000 (EST) Received: by blackwater.lemis.com (Postfix, from userid 1004) id 02B7A51202; Sat, 21 Aug 2004 18:02:07 +0930 (CST) Date: Sat, 21 Aug 2004 18:02:07 +0930 From: Greg 'groggy' Lehey To: Pawel Jakub Dawidek Message-ID: <20040821083207.GG92256@wantadilla.lemis.com> References: <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> <20040820223446.GA18838@panzer.kdm.org> <20040821080614.GC30151@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4Epv4kl9IRBfg3rk" Content-Disposition: inline In-Reply-To: <20040821080614.GC30151@darkness.comp.waw.pl> User-Agent: Mutt/1.4.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 cc: Wilko Bulte cc: Poul-Henning Kamp cc: "Kenneth D. Merry" cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Aug 2004 08:32:11 -0000 --4Epv4kl9IRBfg3rk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Saturday, 21 August 2004 at 10:06:14 +0200, Pawel Jakub Dawidek wrote: > So as you can see there are no difference in reading (actually > reading in degraded mode is even a bit faster, hard to say why). I would have half expected this. If you're reading parity as well, then without spindle sync it will take longer for three I/Os to complete than 2. In a fractionally different context, I describe a similar result for RAID-5 at http://www.vinumvm.org/vinum/implementation.html. Greg -- Note: I discard all HTML mail unseen. Finger grog@FreeBSD.org for PGP public key. See complete headers for address and phone numbers. --4Epv4kl9IRBfg3rk Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (FreeBSD) iD8DBQFBJwiHIubykFB6QiMRAkwxAJ90n7MuEA+YyNHlddScx5gMk+fHZwCff9Fv ncYnhdxFXB/A8na2ocfGeVY= =H0xY -----END PGP SIGNATURE----- --4Epv4kl9IRBfg3rk-- From owner-freebsd-performance@FreeBSD.ORG Sat Aug 21 08:33:56 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EF4C916A4CE; Sat, 21 Aug 2004 08:33:56 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9577343D3F; Sat, 21 Aug 2004 08:33:56 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 491CBAC977; Sat, 21 Aug 2004 10:33:55 +0200 (CEST) Date: Sat, 21 Aug 2004 10:33:55 +0200 From: Pawel Jakub Dawidek To: Greg 'groggy' Lehey Message-ID: <20040821083355.GD30151@darkness.comp.waw.pl> References: <41449.1092750244@critter.freebsd.dk> <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> <20040821082838.GF92256@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Wl2tksQRuA3a9k2N" Content-Disposition: inline In-Reply-To: <20040821082838.GF92256@wantadilla.lemis.com> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: Wilko Bulte cc: Poul-Henning Kamp cc: src-committers@FreeBSD.ORG cc: freebsd-performance@freebsd.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Aug 2004 08:33:57 -0000 --Wl2tksQRuA3a9k2N Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Aug 21, 2004 at 05:58:38PM +0930, Greg 'groggy' Lehey wrote: +> So these are two disks + parity, right? That's not exactly a typical +> setup. Yes. +> > RAID5: +>=20 +> What was the RAID-5 stripe size? 256kB. +> > As you can see RAID5 is only faster in READ tests for 15 and 100 proce= sses +> > working in parallel. As I can see, RAID3 is faster in all the rest tes= ts. +>=20 +> I don't really see enough to convince me either way. If you use small +> RAID-5 stripes, then yes, it's possible to get better performance from +> RAID-3. I'd also suggest that your figures would look very different +> with five or nine disks. It would also be interested to see the +> results of rawio on these configurations, and also the relative +> performance of a single disk. Ok, wait for my reading optimization results:) --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --Wl2tksQRuA3a9k2N Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBJwjzForvXbEpPzQRAk2uAKDzWlM4i7djtP90OnbQjz9d5AYj0gCgrVrH z7czl5tbnyf76yaA8a9cP/8= =k0SP -----END PGP SIGNATURE----- --Wl2tksQRuA3a9k2N-- From owner-freebsd-performance@FreeBSD.ORG Sat Aug 21 08:37:58 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ADDB216A4CE; Sat, 21 Aug 2004 08:37:58 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id 449FF43D39; Sat, 21 Aug 2004 08:37:58 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 4B880AC977; Sat, 21 Aug 2004 10:37:57 +0200 (CEST) Date: Sat, 21 Aug 2004 10:37:57 +0200 From: Pawel Jakub Dawidek To: Greg 'groggy' Lehey Message-ID: <20040821083757.GE30151@darkness.comp.waw.pl> References: <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> <20040820223446.GA18838@panzer.kdm.org> <20040821080614.GC30151@darkness.comp.waw.pl> <20040821083207.GG92256@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="STig1+2QdxPS4TAh" Content-Disposition: inline In-Reply-To: <20040821083207.GG92256@wantadilla.lemis.com> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: Wilko Bulte cc: Poul-Henning Kamp cc: "Kenneth D. Merry" cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Aug 2004 08:37:58 -0000 --STig1+2QdxPS4TAh Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Aug 21, 2004 at 06:02:07PM +0930, Greg 'groggy' Lehey wrote: +> On Saturday, 21 August 2004 at 10:06:14 +0200, Pawel Jakub Dawidek wrote: +> > So as you can see there are no difference in reading (actually +> > reading in degraded mode is even a bit faster, hard to say why). +>=20 +> I would have half expected this. If you're reading parity as well, +> then without spindle sync it will take longer for three I/Os to +> complete than 2. I'm not sure if I understand you well... In RAID3, when array is complete you don't use parity component for reading at all. So in complete mode, as well as in degraded mode RAID3 is using only two components for reading. --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --STig1+2QdxPS4TAh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBJwnlForvXbEpPzQRAvOVAKD38qVZqO7fHm1lYaMLJzJ94UummACeNPA8 /AhARspUSvn++Kt6ZHC/za4= =u0qn -----END PGP SIGNATURE----- --STig1+2QdxPS4TAh-- From owner-freebsd-performance@FreeBSD.ORG Sat Aug 21 08:45:28 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 46D7416A4CE; Sat, 21 Aug 2004 08:45:28 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id E314D43D39; Sat, 21 Aug 2004 08:45:27 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id ACB2DAC977; Sat, 21 Aug 2004 10:45:26 +0200 (CEST) Date: Sat, 21 Aug 2004 10:45:26 +0200 From: Pawel Jakub Dawidek To: "Kenneth D. Merry" Message-ID: <20040821084526.GF30151@darkness.comp.waw.pl> References: <200408161043.i7GAhfXs079045@repoman.freebsd.org> <20040817004407.GA81257@wantadilla.lemis.com> <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> <20040820223446.GA18838@panzer.kdm.org> <20040821080614.GC30151@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZMT28BdW279F9lxY" Content-Disposition: inline In-Reply-To: <20040821080614.GC30151@darkness.comp.waw.pl> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: Greg 'groggy' Lehey cc: Wilko Bulte cc: Poul-Henning Kamp cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Aug 2004 08:45:28 -0000 --ZMT28BdW279F9lxY Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Aug 21, 2004 at 10:06:14AM +0200, Pawel Jakub Dawidek wrote: +> Here are already presented results of full RAID3: +>=20 +> RAID3: +> Number of Bytes per Requests per +> Operations processes second second +> ---------------------------------------------------------------------- +> READ 3 6329500 95 +> READ 15 8981047 135 +> READ 100 10719314 161 [...] Whoa! I hacked RAID3 to do something like this while reading (for 3 components): disk0+disk1 disk0+(disk0^disk2) disk1+(disk1^disk2) disk0+disk1 disk0+(disk0^disk2) disk1+(disk1^disk2) [...] And results are really impressive: RAID3 with new reading algorithm: Number of Bytes per Requests per Operations processes second second ---------------------------------------------------------------------- READ 3 8412626 126 READ 15 12083590 181 READ 100 15104488 227 This is more than 40% of speed-up for 100 processes! --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --ZMT28BdW279F9lxY Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBJwumForvXbEpPzQRAmGGAJ43ddkWJL1X6p35MdOurGWVyMGTZgCeJ9Z5 FhTn9CbYdY+6pT4Aall21+c= =JUhx -----END PGP SIGNATURE----- --ZMT28BdW279F9lxY-- From owner-freebsd-performance@FreeBSD.ORG Sat Aug 21 08:46:19 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3E8B716A4CE; Sat, 21 Aug 2004 08:46:19 +0000 (GMT) Received: from ozlabs.org (ozlabs.org [203.10.76.45]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9726343D49; Sat, 21 Aug 2004 08:46:16 +0000 (GMT) (envelope-from grog@lemis.com) Received: from blackwater.lemis.com (blackwater.lemis.com [192.109.197.80]) by ozlabs.org (Postfix) with ESMTP id 821A42BD68; Sat, 21 Aug 2004 18:46:14 +1000 (EST) Received: by blackwater.lemis.com (Postfix, from userid 1004) id D954C51201; Sat, 21 Aug 2004 18:16:12 +0930 (CST) Date: Sat, 21 Aug 2004 18:16:12 +0930 From: Greg 'groggy' Lehey To: Pawel Jakub Dawidek Message-ID: <20040821084612.GH92256@wantadilla.lemis.com> References: <20040817074633.GO30151@darkness.comp.waw.pl> <20040817112900.GA31635@freebie.xs4all.nl> <20040817124020.GK88156@wantadilla.lemis.com> <20040817131612.GT30151@darkness.comp.waw.pl> <20040819024359.GA85432@wantadilla.lemis.com> <20040820193547.GZ30151@darkness.comp.waw.pl> <20040820223446.GA18838@panzer.kdm.org> <20040821080614.GC30151@darkness.comp.waw.pl> <20040821083207.GG92256@wantadilla.lemis.com> <20040821083757.GE30151@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7cm2iqirTL37Ot+N" Content-Disposition: inline In-Reply-To: <20040821083757.GE30151@darkness.comp.waw.pl> User-Agent: Mutt/1.4.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 cc: Wilko Bulte cc: Poul-Henning Kamp cc: "Kenneth D. Merry" cc: freebsd-performance@FreeBSD.org Subject: Re: RAID-3? (was: cvs commit: src MAINTAINERS) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Aug 2004 08:46:19 -0000 --7cm2iqirTL37Ot+N Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Saturday, 21 August 2004 at 10:37:57 +0200, Pawel Jakub Dawidek wrote: > On Sat, Aug 21, 2004 at 06:02:07PM +0930, Greg 'groggy' Lehey wrote: > +> On Saturday, 21 August 2004 at 10:06:14 +0200, Pawel Jakub Dawidek wrote: > +> > So as you can see there are no difference in reading (actually > +> > reading in degraded mode is even a bit faster, hard to say why). > +> > +> I would have half expected this. If you're reading parity as well, > +> then without spindle sync it will take longer for three I/Os to > +> complete than 2. > > I'm not sure if I understand you well... I think you understood what I said, but it looks like it was based on a misassumption. > In RAID3, when array is complete you don't use parity component for > reading at all. So in complete mode, as well as in degraded mode > RAID3 is using only two components for reading. OK, that seems reasonable. phk suggested that you would check parity on reads. Yes, in that case you'd expect things to be identical. Greg -- Note: I discard all HTML mail unseen. Finger grog@FreeBSD.org for PGP public key. See complete headers for address and phone numbers. --7cm2iqirTL37Ot+N Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (FreeBSD) iD8DBQFBJwvUIubykFB6QiMRAg8xAKCKid9G6RFJEPAYsu10eU6W9PfciQCgopuP Qvwfa0eo4N+QOd6XItfv7JU= =dCEj -----END PGP SIGNATURE----- --7cm2iqirTL37Ot+N--