From owner-freebsd-stable@FreeBSD.ORG Fri Apr 8 21:34:03 2005 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 859F716A4CE; Fri, 8 Apr 2005 21:34:03 +0000 (GMT) Received: from prosporo.hedron.org (hedron.org [66.11.182.60]) by mx1.FreeBSD.org (Postfix) with ESMTP id 87ED643D1D; Fri, 8 Apr 2005 21:34:02 +0000 (GMT) (envelope-from ean@hedron.org) Received: from localhost.hedron.org (localhost.hedron.org [127.0.0.1]) by prosporo.hedron.org (Postfix) with ESMTP id EC9DEC118; Fri, 8 Apr 2005 17:34:03 -0400 (EDT) From: Ean Kingston To: freebsd-questions@freebsd.org Date: Fri, 8 Apr 2005 17:34:02 -0400 User-Agent: KMail/1.7.2 References: <4256EC7A.5060107@toldme.com> In-Reply-To: <4256EC7A.5060107@toldme.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200504081734.03403.ean@hedron.org> cc: Danny Howard cc: freebsd-stable@freebsd.org Subject: Re: (LONG) ATA Benchmark: 5.x Reads Slower than Writes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Apr 2005 21:34:03 -0000 On April 8, 2005 04:41 pm, Danny Howard wrote: > Hello, > > BACKGROUND > > I need to purchase a new system for our developers, for use as a > Postgres database test server. Having a RAID, probably RAID1, is > desirable for performance and reliability. I have recently set up a > system with a gmirror-based software RAID1 on a pair of 250GB ATA > drives. I would like to stick with gmirror, because: > > - We save money on extra hardware. > - Since gmirror is part of FreeBSD, maintenance is a lot easier than > with a hardware solution. > > SUMMARY > > But before I go balls out, I should see how well it compares to hardware > RAID. So, I do some benchmarks with bonnie++. Since this simulates > create, write and read on thousands of random files, this sounds like a > good approximation of what Postgres does. :) > > I don't have the time and hardware to do very scientific tests, but I > have been able to run a series of benchmarks using bonnie++ on some > systems I have available to me. The ATA-based gmirror performs > extremely well, compared to a few Adaptec RAIDs that we have, EXCEPT > that the sequential and random reads are MUCH SLOWER than the hardware > solution, and even *slower than the preceding write operations*. This > is counter-intuitive, especially since RAID1 implies slowed writes and > faster reads. I tried the benchmark on my workstation (single 2.5" IDE > in a laptop) and got comparable write-faster-than-read results. May I suggest that you turn softupdates off and sync on for the filesystems you are testing. If you don't you are not really testing the hardware. You are testing the FreeBSD disk caching system in the kernel. > DATA > > I was able to make use of the following test systems. I ran tests in > multi-user, but tried to favor times when there wasn't much background > activity: > > mito: (lone 2.5" ATA) > 5.4-PRERELEASE > CPU: Intel(R) Pentium(R) M processor 1.50GHz (1495.16-MHz 686-class CPU) > atapci0: > ata0: channel #0 on atapci0 > ata1: channel #1 on atapci0 > ad0: 28615MB [58140/16/63] at ata0-master > UDMA100 > > amun: (gmirror RAID1 2 x WD250GB High Intensity) > FreeBSD 5.3-RELEASE > CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.22-MHz 686-class CPU) > atapci0: > ata2: channel #0 on atapci0 > ata3: channel #1 on atapci0 > ad4: 238475MB [484521/16/63] at > ata2-master SATA150 > ad6: 238475MB [484521/16/63] at > ata3-master SATA150 > atapci1: > acd0: CDROM at ata1-master UDMA33 > > janus: (Adaptec RAID1 2 x 72G 10,000 RPM SCSI) > FreeBSD 4.10-STABLE > CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) [DUAL] > aac0: mem 0xf8000000-0xfbffffff irq 16 at > device 1.0 on pci2 > aac0: i960RX 100MHz, 48MB cache memory, optional battery present > aac0: Kernel 4.0-0, Build 6011, S/N baec64 > aac0: Supported > Options=1f7eOND,SGMAP64,ALARM,NONDASD> aacd0: on aac0 > aacd0: 69998MB (143357184 sectors) > > db2: (Adaptec RAID10 4 x 36G 15,000 RPM SCSI) > FreeBSD 4.8-STABLE > CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3065.81-MHz 686-class CPU) [DUAL] > aac0: mem 0xf8000000-0xfbffffff irq 18 at > device 2.0 on pci5 > aac0: i960RX 100MHz, 48MB cache memory, optional battery present > aac0: Kernel 4.0-0, Build 6008, S/N b97ce8 > aac0: Supported > Options=1f7eOND,SGMAP64,ALARM,NONDASD> aacp0: on aac0 > > The raw data can be viewed at > http://dannyman.toldme.com/scratch/benchmarks/ > > ANALYSIS > > Unfortunately, my hardware RAIDs are on FreeBSD 4, and gmirror is on 5. > My hardware RAIDs are on dual CPU systems, with 2G RAM, and my gmirror > is on a single hyperthreaded CPU with 512M. Yes, sorry, not especially > scientific. Maybe the changes in FreeBSD make a big difference? Maybe > RAM makes a big difference? > > The first results show a serious advantage for the gmirror setup: > > Sequential output (char) > gmirror ATA RAID1: avg 320K/s > Adaptec SCSI RAID1: avg 222K/s > Adaptec SCSI RAID10: avg 202K/s > > Sequential input (char) > gmirror ATA RAID1: avg 617K/s > Adaptec SCSI RAID1: avg 345K/s > Adaptec SCSI RAID10: avg 336K/s > > Sequential Output (block) > gmirror ATA RAID1: avg 37893K/s > Adaptec SCSI RAID1: avg 13829K/s > Adaptec SCSI RAID10: avg 40440K/s > > The gmirror sees slightly poorer performance in random seeks: > > Rndom Seeks > gmirror ATA RAID1: avg 4144/s > Adaptec SCSI RAID1: avg 5428/s > Adaptec SCSI RAID10: avg 13302/s > > That all sounds great if I was streaming video, but I want to run a > database, opening and closing, reading, writing, and rewriting several > small files. This is where things seem to go rotten. > > We see the ATA performance go to heck on the File Create tests: > > Sequential Create > laptop 2.5" ATA: avg 101/s > gmirror ATA RAID1: avg 365/s > Adaptec SCSI RAID1: avg 160/s > Adaptec SCSI RAID10: avg 412/s > > Sequential Read > laptop 2.5" ATA: avg 76/s # SLOWER than write! > gmirror ATA RAID1: avg 251/s # SLOWER than write! > Adaptec SCSI RAID1: avg 7862/s > Adaptec SCSI RAID10: avg 7618/s > > Random Create > laptop 2.5" ATA: avg 124/s > gmirror ATA RAID1: avg 354/s > Adaptec SCSI RAID1: avg 155/s > Adaptec SCSI RAID10: avg 504/s > > Random Read > laptop 2.5" ATA: avg 57/s # SLOWER than write! > gmirror ATA RAID1: avg 144/s # SLOWER than write! > Adaptec SCSI RAID1: avg 7655/s > Adaptec SCSI RAID10: avg 7413/s > > CONFUSION > > Now, I could explain poor read performance by: > - Less RAM == Less buffer > - Bigger Disks == Slower Seeks > - Less CPU == ??? > > I DO have a 4.8-STABLE with a single IDE disk, no Soft Updates, and > faster read than write: > > Version 1.93c ------Sequential Create------ --------Random > Create-------- > anubis.xxxxxxxxxxxx -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 10:104884:0/5 183 32 1739 97 502 16 176 32 1624 94 > 368 13 > Latency 707ms 11487us 32824us 488ms 207ms > 117ms > > However, seeing read SLOWER than write ... I have to wonder if something > fishy is going on. Suggestions? Ideas? I'm fresh out, at the moment. > My suspicion is that something in 5.x is out-of-tune!? > > Thanks a lot. > > Sincerely, > -danny -- Ean Kingston E-Mail: ean AT hedron DOT org URL: http://www.hedron.org/