From owner-freebsd-performance@FreeBSD.ORG Fri Oct 27 23:04:00 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 20F5316A407 for ; Fri, 27 Oct 2006 23:04:00 +0000 (UTC) (envelope-from stevep-hv@zpfe.com) Received: from mail.zpfe.com (mail.zpfe.com [208.42.168.33]) by mx1.FreeBSD.org (Postfix) with SMTP id 7D7F843D58 for ; Fri, 27 Oct 2006 23:03:59 +0000 (GMT) (envelope-from stevep-hv@zpfe.com) Received: (qmail 46722 invoked from network); 27 Oct 2006 23:03:58 -0000 Received: from unknown (HELO revere.zpfe.com) (208.42.168.33) by mail.zpfe.com with SMTP; 27 Oct 2006 23:03:58 -0000 Message-Id: <6.2.3.4.0.20061027180329.020bed68@localhost> X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.4 Date: Fri, 27 Oct 2006 18:03:33 -0500 To: performance@freebsd.org From: Steve Peterson Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII"; format=flowed Cc: Subject: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Oct 2006 23:04:00 -0000 I recently set up a media server for home use and decided to try the gvinum raid5 support to learn about it and see how it performs. It seems slower than I'd expect -- a little under 6MB/second, with about 50 IOs/drive/second -- and I'm trying to understand why. Any assistance/pointers would be appreciated. The disk system consists of 4 Seagate NL35 SATA ST3250623NS drives connected to a Promise TX4300 (PDC40719) controller, organized as a RAID5 volume via gvinum using this configuration: drive drive01 device /dev/ad10 drive drive02 device /dev/ad6 drive drive03 device /dev/ad4 drive drive04 device /dev/ad8 volume vol1 plex org raid5 256k sd length 200001m drive drive01 sd length 200001m drive drive02 sd length 200001m drive drive03 sd length 200001m drive drive04 dd reports the following performance on a 1G file write to the RAID5 hosted volume: $ time dd if=/dev/zero of=big.file bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 179.717742 secs (5834571 bytes/sec) 179.76 real 0.02 user 16.60 sys By comparison, creating the same file on the system disk (an old ATA ST380021A connected via a SIS 730 on the motherboard): time dd if=/dev/zero of=big.file bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 28.264056 secs (37099275 bytes/sec) 28.32 real 0.01 user 19.13 sys and vmstat reports about 280-300 IOs/second to that drive. The CPU is pretty weak -- an Athlon 750. Is that the source of my problem? If you look at the vmstat output below the machine is busy but not pegged. $ vmstat -n 5 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad4 ad6 ad8 ad10 in sy cs us sy id 1 0 0 96700 134124 19 3 0 0 65 12 0 0 0 0 0 1375 239 621 1 1 98 0 0 0 96700 134124 2 0 0 0 0 0 0 0 0 0 0 1305 180 445 2 0 98 0 0 0 96700 134124 0 0 0 0 0 0 0 0 0 0 0 1302 201 449 1 1 98 0 0 0 96700 134124 0 0 0 0 0 0 0 0 0 0 0 1353 165 545 1 1 98 1 0 0 99088 129828 311 0 0 0 1328 0 0 14 24 23 13 1448 310 1286 0 18 82 0 1 0 99088 120228 0 0 0 0 5088 0 0 56 48 48 56 1686 187 2794 0 64 36 0 1 0 99088 114212 0 0 0 0 4992 0 0 54 50 52 48 1659 184 2694 2 54 44 0 1 0 99088 107044 0 0 0 0 5088 0 0 50 51 57 50 1670 188 2777 2 56 42 0 1 0 99088 101668 0 0 0 0 5184 0 0 51 58 52 51 1678 190 2801 1 56 43 0 1 0 99088 95140 0 0 0 0 4896 0 1 48 52 48 52 1662 186 2665 1 57 42 0 1 0 99088 88356 0 0 0 0 4896 0 0 50 47 50 53 1648 184 2641 1 54 45 0 1 0 99088 81316 0 0 0 0 4800 0 0 52 48 48 53 1620 184 2602 0 57 43 0 1 0 99088 73892 0 0 0 0 4992 0 0 54 50 49 46 1685 191 2698 1 62 37 0 1 0 99088 68260 0 0 0 0 4992 0 0 48 50 54 51 1661 184 2663 1 56 43 0 1 0 99088 61032 0 0 0 0 5088 0 0 51 53 53 51 1656 186 2735 1 57 42 0 1 0 99088 54436 0 0 0 0 4768 0 0 48 50 50 47 1637 184 2580 0 56 44 0 1 0 99088 48292 0 0 0 0 5120 0 2 50 55 49 53 1647 186 2710 1 62 38 0 1 0 99088 42020 0 0 0 0 4928 0 0 53 47 49 55 1657 192 2671 1 58 41 0 1 0 99088 35492 0 0 0 0 4768 0 0 50 48 47 49 1639 181 2544 0 59 41 0 1 0 99088 28836 0 0 0 0 4996 0 2 54 50 52 48 1654 186 2689 0 57 43 0 1 0 99088 21540 0 0 0 0 4704 0 0 47 47 49 49 1648 184 2580 2 58 40 0 1 0 99088 14756 0 0 0 0 4928 0 5 49 51 54 48 1638 184 2645 1 58 41 0 1 0 99088 10276 0 0 0 0 4864 0 0 48 49 48 49 1679 196 2662 1 55 45 0 1 0 99088 10152 0 0 0 0 4800 1937 0 47 51 50 48 1651 181 2637 1 58 42 1 1 0 99088 14560 0 6746 0 0 4948 9308 0 46 37 40 49 1616 176 2396 1 57 42 0 1 0 99088 15336 0 9815 0 0 5328 11178 0 61 64 59 67 1612 188 2741 0 62 38 0 1 0 99088 15532 0 6188 0 0 5536 7452 0 49 49 57 38 1618 190 2526 1 61 38 1 0 0 99088 12012 0 3054 0 0 5728 3726 0 69 63 67 61 1637 189 2859 2 58 41 0 1 0 99088 15532 0 12692 0 0 5700 14904 1 55 49 53 57 1640 183 2677 0 58 42 0 1 0 99088 9388 0 0 0 0 5696 0 0 65 62 61 66 1600 184 2699 1 61 38 0 1 0 99088 14828 0 2925 0 0 5328 5565 0 64 68 57 65 1602 194 2758 1 53 46 0 1 0 99088 13100 0 2718 0 0 5904 3726 0 50 52 53 55 1594 184 2556 1 59 40 0 1 0 99088 14508 0 13368 0 0 5984 14904 0 64 66 61 70 1642 184 2901 2 63 36 0 1 0 99088 13608 0 0 0 0 5888 1855 0 65 71 71 65 1630 177 2849 1 63 36 0 1 0 99088 13932 0 10265 0 0 5728 11178 0 62 64 63 51 1633 176 2742 1 60 39 configuration from last boot: Sep 24 12:11:35 archive kernel: Copyright (c) 1992-2006 The FreeBSD Project. Sep 24 12:11:35 archive kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Sep 24 12:11:35 archive kernel: The Regents of the University of California. All rights reserved. Sep 24 12:11:35 archive kernel: FreeBSD 6.1-RELEASE-p7 #5: Wed Sep 20 14:03:56 CDT 2006 Sep 24 12:11:35 archive kernel: root@archive.zpfe.com:/usr/obj/usr/src/sys/ARCHIVE Sep 24 12:11:35 archive kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 Sep 24 12:11:35 archive kernel: CPU: AMD Athlon(tm) Processor (746.67-MHz 686-class CPU) Sep 24 12:11:35 archive kernel: Origin = "AuthenticAMD" Id = 0x644 Stepping = 4 Sep 24 12:11:35 archive kernel: Features=0x183f9ff Sep 24 12:11:35 archive kernel: AMD Features=0xc0440800,MMX+,3DNow+,3DNow> Sep 24 12:11:35 archive kernel: real memory = 251592704 (239 MB) Sep 24 12:11:35 archive kernel: avail memory = 236695552 (225 MB) Sep 24 12:11:35 archive kernel: kbd1 at kbdmux0 Sep 24 12:11:35 archive kernel: acpi0: on motherboard Sep 24 12:11:35 archive kernel: acpi0: Power Button (fixed) Sep 24 12:11:35 archive kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 Sep 24 12:11:35 archive kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x5008-0x500b on acpi0 Sep 24 12:11:35 archive kernel: cpu0: on acpi0 Sep 24 12:11:35 archive kernel: acpi_throttle0: on cpu0 Sep 24 12:11:35 archive kernel: acpi_button0: on acpi0 Sep 24 12:11:35 archive kernel: pcib0: port 0xcf8-0xcff on acpi0 Sep 24 12:11:35 archive kernel: pci0: on pcib0 Sep 24 12:11:35 archive kernel: agp0: mem 0xd0000000-0xd3ffffff at device 0.0 on pci0 Sep 24 12:11:35 archive kernel: atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 0.1 on pci0 Sep 24 12:11:35 archive kernel: ata0: on atapci0 Sep 24 12:11:35 archive kernel: ata1: on atapci0 Sep 24 12:11:35 archive kernel: isab0: at device 1.0 on pci0 Sep 24 12:11:35 archive kernel: isa0: on isab0 Sep 24 12:11:35 archive kernel: sis0: port 0xd400-0xd4ff mem 0xcfff8000-0xcfff8fff irq 11 at device 1.1 on pci0 Sep 24 12:11:35 archive kernel: miibus0: on sis0 Sep 24 12:11:35 archive kernel: ukphy0: on miibus0 Sep 24 12:11:35 archive kernel: ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto Sep 24 12:11:35 archive kernel: sis0: Ethernet address: 00:0a:e6:14:e3:53 Sep 24 12:11:35 archive kernel: ohci0: mem 0xcfff9000-0xcfff9fff irq 10 at device 1.2 on pci0 Sep 24 12:11:35 archive kernel: ohci0: [GIANT-LOCKED] Sep 24 12:11:35 archive kernel: usb0: OHCI version 1.0, legacy support Sep 24 12:11:35 archive kernel: usb0: on ohci0 Sep 24 12:11:35 archive kernel: usb0: USB revision 1.0 Sep 24 12:11:35 archive kernel: uhub0: SiS OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 Sep 24 12:11:35 archive kernel: uhub0: 3 ports with 3 removable, self powered Sep 24 12:11:35 archive kernel: ohci1: mem 0xcfffa000-0xcfffafff irq 10 at device 1.3 on pci0 Sep 24 12:11:35 archive kernel: ohci1: [GIANT-LOCKED] Sep 24 12:11:35 archive kernel: usb1: OHCI version 1.0, legacy support Sep 24 12:11:35 archive kernel: usb1: on ohci1 Sep 24 12:11:35 archive kernel: usb1: USB revision 1.0 Sep 24 12:11:35 archive kernel: uhub1: SiS OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 Sep 24 12:11:35 archive kernel: uhub1: 3 ports with 3 removable, self powered Sep 24 12:11:35 archive kernel: pcib1: at device 2.0 on pci0 Sep 24 12:11:35 archive kernel: pci1: on pcib1 Sep 24 12:11:35 archive kernel: pci1: at device 0.0 (no driver attached) Sep 24 12:11:35 archive kernel: atapci1: port 0xdc00-0xdc7f,0xd800-0xd8ff mem 0xcfffb000-0xcfffbfff,0xcffc0000-0xcffdffff irq 11 at device 11.0 on pci0 Sep 24 12:11:35 archive kernel: ata2: on atapci1 Sep 24 12:11:35 archive kernel: ata3: on atapci1 Sep 24 12:11:35 archive kernel: ata4: on atapci1 Sep 24 12:11:35 archive kernel: ata5: on atapci1 Sep 24 12:11:35 archive kernel: acpi_button1: on acpi0 Sep 24 12:11:35 archive kernel: atkbdc0: port 0x60,0x64 irq 1 on acpi0 Sep 24 12:11:35 archive kernel: atkbd0: irq 1 on atkbdc0 Sep 24 12:11:35 archive kernel: atkbd0: irq 1 on atkbdc0 Sep 24 12:11:35 archive kernel: kbd0 at atkbd0 Sep 24 12:11:35 archive kernel: atkbd0: [GIANT-LOCKED] Sep 24 12:11:35 archive kernel: pmtimer0 on isa0 Sep 24 12:11:35 archive kernel: orm0: at iomem 0xc0000-0xcbfff,0xcc000-0xd57ff,0xd5800-0xdffff on isa0 Sep 24 12:11:35 archive kernel: ppc0: parallel port not found. Sep 24 12:11:35 archive kernel: sc0: at flags 0x100 on isa0 Sep 24 12:11:35 archive kernel: sc0: VGA <16 virtual consoles, flags=0x300> Sep 24 12:11:35 archive kernel: sio0: configured irq 4 not in bitmap of probed irqs 0 Sep 24 12:11:35 archive kernel: sio0: port may not be enabled Sep 24 12:11:35 archive kernel: sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 Sep 24 12:11:35 archive kernel: sio0: type 8250 or not responding Sep 24 12:11:35 archive kernel: sio1: configured irq 3 not in bitmap of probed irqs 0 Sep 24 12:11:35 archive kernel: sio1: port may not be enabled Sep 24 12:11:35 archive kernel: vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Sep 24 12:11:35 archive kernel: Timecounter "TSC" frequency 746674414 Hz quality 800 Sep 24 12:11:35 archive kernel: Timecounters tick every 1.000 msec Sep 24 12:11:35 archive kernel: ad0: 76319MB at ata0-master UDMA100 Sep 24 12:11:35 archive kernel: acd0: CDROM at ata1-master PIO3 Sep 24 12:11:35 archive kernel: ad4: 238475MB at ata2-master SATA150 Sep 24 12:11:35 archive kernel: ad6: 238475MB at ata3-master SATA150 Sep 24 12:11:35 archive kernel: ad8: 238475MB at ata4-master SATA150 Sep 24 12:11:35 archive kernel: ad10: 238475MB at ata5-master SATA150