From owner-freebsd-performance@FreeBSD.ORG Sun Oct 29 17:20:26 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1077216A412 for ; Sun, 29 Oct 2006 17:20:26 +0000 (UTC) (envelope-from stevep-hv@zpfe.com) Received: from mail.zpfe.com (mail.zpfe.com [208.42.168.33]) by mx1.FreeBSD.org (Postfix) with SMTP id A20B343D46 for ; Sun, 29 Oct 2006 17:20:25 +0000 (GMT) (envelope-from stevep-hv@zpfe.com) Received: (qmail 45767 invoked from network); 29 Oct 2006 17:20:23 -0000 Received: from unknown (HELO revere.zpfe.com) (208.42.168.33) by mail.zpfe.com with SMTP; 29 Oct 2006 17:20:23 -0000 Message-Id: <6.2.3.4.0.20061029095927.05833580@localhost> X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.4 Date: Sun, 29 Oct 2006 11:20:33 -0600 To: Petri Helenius ,performance@freebsd.org From: Steve Peterson In-Reply-To: <4543AD35.30205@he.iki.fi> References: <6.2.3.4.0.20061027180329.020bed68@localhost> <4542D941.2070204@centtech.com> <6.2.3.4.0.20061028124559.02105930@localhost> <4543AD35.30205@he.iki.fi> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII"; format=flowed Cc: Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Oct 2006 17:20:26 -0000 Petri -- thanks for the idea. I ran 2 dds in parallel; they took roughly twice as long in clock time, and had about 1/2 the throughput of the single dd. On my system it doesn't look like how the work is offered to the disk subsystem matters. # time dd if=/dev/zero of=blort1 bs=1m count=1000 & ; time dd if=/dev/zero of=blort2 bs=1m count=1000& [1] 69356 [2] 69357 # 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 382.511089 secs (2741296 bytes/sec) 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 385.677897 secs (2718787 bytes/sec) [2] Done dd if=/dev/zero of=blort2 bs=1m count=1000 0.023u 16.913s 6:25.75 4.3% 20+2813k 3+8772io 2pf+0w [1] + Done dd if=/dev/zero of=blort1 bs=1m count=1000 0.023u 16.880s 6:22.70 4.4% 20+2810k 1+8765io 0pf+0w vmstat output during this run: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad4 ad6 ad8 ad10 in sy cs us sy id 1 1 0 101692 13132 0 0 0 0 5872 1857 0 54 62 69 59 2293 321 3914 2 62 35 3 1 0 101692 15576 0 4 0 0 5840 1947 0 52 54 60 57 2023 308 3379 2 64 35 2 1 0 101692 10072 0 0 0 0 5344 0 0 61 54 55 56 2133 368 3496 0 58 42 2 2 0 101692 11496 0 0 0 0 5984 1860 0 61 65 57 61 2377 429 4118 1 69 30 3 0 0 101692 13124 0 0 0 0 5568 1799 0 57 62 62 65 2440 466 4252 1 64 36 1 2 0 101692 12404 0 0 0 0 6176 1868 0 61 70 67 59 1923 482 3337 1 64 36 0 3 0 101692 15080 0 0 0 0 6576 1805 0 67 71 65 68 2423 468 4300 1 68 31 0 2 0 101692 17448 0 0 0 0 5760 1792 0 56 57 57 60 2463 399 4155 4 65 31 That got me thinking about whether there was a bottleneck to the disk somewhere else. I ran a dd reading 1GB simultaneous from each of the 4 disks. Each disk could be read at about 25MB/sec for a combined read transfer rate of 100MB/sec. A snipped of iostat output: tty ad0 ad4 ad6 ad8 ad10 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 334 0.00 0 0.00 127.36 201 24.97 126.73 201 24.85 127.35 198 24.60 126.71 199 24.60 2 0 11 12 75 0 115 0.00 0 0.00 128.00 191 23.86 126.98 188 23.30 128.00 196 24.48 127.36 199 24.73 2 0 15 9 73 0 115 0.00 0 0.00 127.37 203 25.23 127.08 208 25.79 126.75 205 25.35 127.27 175 21.73 2 0 9 9 80 So I'm definitely not getting the full 60MB/sec performance out of the disk with my current controller and mobo. I guess the fundamental question is this -- if I have a 4 disk subsystem that supports an aggregate ~100MB/sec transfer raw to the underlying disks, is it reasonable to expect a ~5MB/sec transfer rate for a RAID5 hosted on that subsystem -- a 95% overhead. Steve At 01:19 PM 10/28/2006, Petri Helenius wrote: >According to my understanding vinum does not overlap requests to >multiple disks when running in raid5 configuration so you're not >going to achieve good numbers with just "single stream" tests. > >Pete > > >Steve Peterson wrote: >>Eric -- thanks for looking at my issue. Here's a dd reading from >>one of the disks underlying the array (the others have basically >>the same performance): >> >>$ time dd if=/dev/ad10 of=/dev/null bs=1m count=1000 >>1000+0 records in >>1000+0 records out >>1048576000 bytes transferred in 15.322421 secs (68434095 bytes/sec) >>0.008u 0.506s 0:15.33 3.2% 20+2715k 0+0io 0pf+0w >> >>and here's a dd reading from the raw gvinum device /dev/gvinum/vol1: >> >>$ time dd if=/dev/gvinum/vol1 of=/dev/null bs=1m count=1000 >>1000+0 records in >>1000+0 records out >>1048576000 bytes transferred in 25.870684 secs (40531437 bytes/sec) >>0.006u 0.552s 0:25.88 2.1% 23+3145k 0+0io 0pf+0w >> >>Is there a way to nondestructively write to the raw disk or gvinum device? >> >>For comparison, here's a read against the raw PATA device on the machine: >> >>$ time dd if=/dev/ad0 of=/dev/null bs=1m count=1000 >>1000+0 records in >>1000+0 records out >>1048576000 bytes transferred in 26.096070 secs (40181376 bytes/sec) >>0.013u 0.538s 0:26.10 2.0% 24+3322k 0+0io 0pf+0w >> >>Steve >> >> >>At 11:14 PM 10/27/2006, Eric Anderson wrote: >>>On 10/27/06 18:03, Steve Peterson wrote: >>>>I recently set up a media server for home use and decided to try >>>>the gvinum raid5 support to learn about it and see how it >>>>performs. It seems slower than I'd expect -- a little under >>>>6MB/second, with about 50 IOs/drive/second -- and I'm trying to >>>>understand why. Any assistance/pointers would be appreciated. >>>>The disk system consists of 4 Seagate NL35 SATA ST3250623NS >>>>drives connected to a Promise TX4300 (PDC40719) controller, >>>>organized as a RAID5 volume via gvinum using this configuration: >>>>drive drive01 device /dev/ad10 >>>>drive drive02 device /dev/ad6 >>>>drive drive03 device /dev/ad4 >>>>drive drive04 device /dev/ad8 >>>>volume vol1 >>>> plex org raid5 256k >>>> sd length 200001m drive drive01 >>>> sd length 200001m drive drive02 >>>> sd length 200001m drive drive03 >>>> sd length 200001m drive drive04 >>>>dd reports the following performance on a 1G file write to the >>>>RAID5 hosted volume: >>>>$ time dd if=/dev/zero of=big.file bs=1m count=1000 >>>>1000+0 records in >>>>1000+0 records out >>>>1048576000 bytes transferred in 179.717742 secs (5834571 bytes/sec) >>>> 179.76 real 0.02 user 16.60 sys >>>>By comparison, creating the same file on the system disk (an old >>>>ATA ST380021A connected via a SIS 730 on the motherboard): >>>>time dd if=/dev/zero of=big.file bs=1m count=1000 >>>>1000+0 records in >>>>1000+0 records out >>>>1048576000 bytes transferred in 28.264056 secs (37099275 bytes/sec) >>>> 28.32 real 0.01 user 19.13 sys >>>>and vmstat reports about 280-300 IOs/second to that drive. >>>>The CPU is pretty weak -- an Athlon 750. Is that the source of >>>>my problem? If you look at the vmstat output below the machine >>>>is busy but not pegged. >>> >>> >>>Try the dd to the raw gvinum device instead of through a >>>filesystem, and also to the individual disks. That will at least >>>tell us where to look. >>> >>> >>>Eric >>> >>> >>> >>>-- >>>------------------------------------------------------------------------ >>>Eric Anderson Sr. Systems Administrator Centaur Technology >>>Anything that works is better than anything that doesn't. >>>------------------------------------------------------------------------ >>>_______________________________________________ >>>freebsd-performance@freebsd.org mailing list >>>http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>>To unsubscribe, send any mail to >>>"freebsd-performance-unsubscribe@freebsd.org" >> >> >>_______________________________________________ >>freebsd-performance@freebsd.org mailing list >>http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>To unsubscribe, send any mail to >>"freebsd-performance-unsubscribe@freebsd.org" >