From owner-freebsd-fs@FreeBSD.ORG Fri Oct 28 14:04:58 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E540106566B for ; Fri, 28 Oct 2011 14:04:58 +0000 (UTC) (envelope-from freebsd@penx.com) Received: from Elmer.dco.penx.com (elmer.dco.penx.com [174.46.214.165]) by mx1.freebsd.org (Postfix) with ESMTP id C3A428FC13 for ; Fri, 28 Oct 2011 14:04:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by Elmer.dco.penx.com (8.14.5/8.14.4) with ESMTP id p9SE4tLu021045 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 28 Oct 2011 08:04:56 -0600 (MDT) (envelope-from freebsd@penx.com) Date: Fri, 28 Oct 2011 08:04:55 -0600 (MDT) From: Dennis Glatting X-X-Sender: dennisg@Elmer.dco.penx.com To: freebsd-fs@freebsd.org Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: ZFS across controllers X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 14:04:58 -0000 This seems like a stupid question but is there any significant performance problems with a ZFS RAIDz volume served across multiple controllers? iirc# zpool status disk-1 pool: disk-1 state: ONLINE scan: scrub in progress since Thu Oct 27 23:42:44 2011 268G scanned out of 2.15T at 5.06M/s, 108h34m to go 0 repaired, 12.17% done config: NAME STATE READ WRITE CKSUM disk-1 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada6 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 cache ad4 ONLINE 0 0 0 ad6 ONLINE 0 0 0 errors: No known data errors iirc# zpool iostat -v disk-1 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- disk-1 2.14T 16.0T 462 20 10.3M 69.4K raidz1 2.14T 16.0T 462 20 10.3M 69.4K ada2 - - 372 5 1.22M 9.49K ada3 - - 370 5 1.20M 8.89K ada4 - - 372 5 1.22M 9.49K ada5 - - 369 5 1.20M 8.89K ada7 - - 375 5 1.21M 9.49K ada6 - - 371 5 1.20M 8.88K da3 - - 294 7 1.21M 9.49K da4 - - 103 6 1.24M 8.67K da5 - - 294 7 1.21M 9.49K da6 - - 100 6 1.24M 8.68K cache - - - - - - ad4 36.9M 55.9G 0 0 6 1.87K ad6 111M 55.8G 0 1 6 5.65K ---------- ----- ----- ----- ----- ----- ----- I have seen two cases that confuse me. First, performance degrades over time. I have read there was previously a prefetch bug but it has been fixed. I also recently turned off prefetching to test (no opinion yet). Second, my RAID controller isn't supported by FreeBSD but works well except for one little annoyance: on two occasions it lost its mind (i.e., its firmware went poof). I noticed when I moved disks away from the controller (da) to the MB SATA ports, my performance decreased, which was not what what I was expecting if one assumes an unsupported controller is the crucible of evil. I have three of those controllers, BTW. One is in a Gigabyte GA-X58A-UD9 and gives me no trouble. The ones that lost their minds were in a Gigabyte GA-X58A-UD7 (rev 2.0). I included dmesg and some config data below. Also, I am running RELENG_8 as of a few days ago. Comments and sarcasm is welcome, and expected. Finally, could someone recommend a HBA suitable for 16 (minimum) to 24 (potentially) SATA II-III disks or how to acheive that feat? I recently purchased two Supermicro AOC-USAS2-L8e but they haven't yet arrived that I will reflash to IT. If they make me happy, since I am building two more servers of 16 disks (plus SSDs) each, I'll buy more; however, lets not assume I have a clue. TIA iirc# cat /boot/loader.conf /boot/loader.conf.local coretemp_load="YES" ahci_load="YES" ipmi_load="YES" xhci_load="YES" vfs.zfs.arc_max="16G" iirc# cat /etc/sysctl.conf # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith Exp $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Uncomment this to prevent users from seeing information about processes that # are being run under another UID. #security.bsd.see_other_uids=0 hw.intr_storm_threshold=2500 vfs.zfs.dedup.prefetch=0 vfs.zfs.prefetch_disable=1 iirc# dmesg Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #33: Wed Oct 26 17:22:14 PDT 2011 root@iirc:/disk-1/src/sys/amd64/compile/IIRC amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM) i7 CPU 940 @ 2.93GHz (2934.20-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x106a4 Family = 6 Model = 1a Stepping = 4 Features=0xbfebfbff Features2=0x98e3bd AMD Features=0x28100800 AMD Features2=0x1 TSC: P-state invariant real memory = 25769803776 (24576 MB) avail memory = 24825696256 (23675 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dfdd0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 ahci0: port 0x8f00-0x8f07,0x8e00-0x8e03,0x8d00-0x8d07,0x8c00-0x8c03,0x8b00-0x8b0f mem 0xfbeff000-0xfbeff7ff irq 16 at device 0.0 on pci1 ahci0: [ITHREAD] ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS ahcich0: at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: at channel 1 on ahci0 ahcich1: [ITHREAD] pcib2: irq 16 at device 2.0 on pci0 pci2: on pcib2 xhci0: mem 0xfbdfe000-0xfbdfffff irq 16 at device 0.0 on pci2 xhci0: [ITHREAD] xhci0: 32 byte context size. usbus0 on xhci0 pcib3: irq 16 at device 3.0 on pci0 pci3: on pcib3 vgapci0: port 0xee00-0xeeff mem 0xe0000000-0xefffffff,0xfb9e0000-0xfb9effff irq 16 at device 0.0 on pci3 pci3: at device 0.1 (no driver attached) pcib4: irq 16 at device 7.0 on pci0 pci4: on pcib4 arcmsr0: port 0xde00-0xdeff mem 0xfb8f0000-0xfb8fffff,0xfb880000-0xfb8bffff irq 16 at device 0.0 on pci4 ARECA RAID ADAPTER0: Driver Version 1.20.00.22 2011-07-04 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.49 2011-08-02 arcmsr0: [ITHREAD] pcib5: irq 16 at device 9.0 on pci0 pci5: on pcib5 pcib6: irq 16 at device 0.0 on pci5 pci6: on pcib6 atapci0: port 0x9f00-0x9f0f mem 0xfb7ff000-0xfb7ff07f,0xfb7f0000-0xfb7f7fff irq 16 at device 0.0 on pci6 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] ata4: on atapci0 ata4: [ITHREAD] ata5: on atapci0 ata5: [ITHREAD] pci0: at device 16.0 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 17.0 (no driver attached) pci0: at device 17.1 (no driver attached) pci0: at device 20.0 (no driver attached) pci0: at device 20.1 (no driver attached) pci0: at device 20.2 (no driver attached) uhci0: port 0xff00-0xff1f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup = 0x2f00 usbus1: on uhci0 uhci1: port 0xfe00-0xfe1f irq 21 at device 26.1 on pci0 uhci1: [ITHREAD] uhci1: LegSup = 0x2f00 usbus2: on uhci1 uhci2: port 0xfd00-0xfd1f irq 18 at device 26.2 on pci0 uhci2: [ITHREAD] uhci2: LegSup = 0x2f00 usbus3: on uhci2 ehci0: mem 0xfbffe000-0xfbffe3ff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus4: EHCI version 1.0 usbus4: on ehci0 pcib7: irq 16 at device 28.0 on pci0 pci7: on pcib7 em0: port 0xcf00-0xcf1f mem 0xfbcc0000-0xfbcdffff,0xfbc00000-0xfbc7ffff,0xfbcfc000-0xfbcfffff irq 16 at device 0.0 on pci7 em0: Using MSIX interrupts with 3 vectors em0: [ITHREAD] em0: [ITHREAD] em0: [ITHREAD] em0: Ethernet address: 00:1b:21:c1:aa:97 pcib8: irq 16 at device 28.4 on pci0 pci8: on pcib8 re0: port 0xbe00-0xbeff mem 0xfbbff000-0xfbbfffff,0xfbbf8000-0xfbbfbfff irq 16 at device 0.0 on pci8 re0: Using 1 MSI-X message re0: Chip rev. 0x2c000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 1c:6f:65:3e:ee:50 re0: [ITHREAD] pcib9: irq 17 at device 28.5 on pci0 pci9: on pcib9 re1: port 0xae00-0xaeff mem 0xfbaff000-0xfbafffff,0xfbaf8000-0xfbafbfff irq 17 at device 0.0 on pci9 re1: Using 1 MSI-X message re1: Chip rev. 0x2c000000 re1: MAC rev. 0x00000000 miibus1: on re1 rgephy1: PHY 1 on miibus1 rgephy1: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re1: Ethernet address: 1c:6f:65:3e:ee:60 re1: [ITHREAD] uhci3: port 0xfc00-0xfc1f irq 23 at device 29.0 on pci0 uhci3: [ITHREAD] usbus5: on uhci3 uhci4: port 0xfb00-0xfb1f irq 19 at device 29.1 on pci0 uhci4: [ITHREAD] usbus6: on uhci4 uhci5: port 0xfa00-0xfa1f irq 18 at device 29.2 on pci0 uhci5: [ITHREAD] usbus7: on uhci5 ehci1: mem 0xfbffd000-0xfbffd3ff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus8: EHCI version 1.0 usbus8: on ehci1 pcib10: at device 30.0 on pci0 pci10: on pcib10 isab0: at device 31.0 on pci0 isa0: on isab0 ahci1: port 0xf900-0xf907,0xf800-0xf803,0xf700-0xf707,0xf600-0xf603,0xf500-0xf51f mem 0xfbffc000-0xfbffc7ff irq 19 at device 31.2 on pci0 ahci1: [ITHREAD] ahci1: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported ahcich2: at channel 0 on ahci1 ahcich2: [ITHREAD] ahcich3: at channel 1 on ahci1 ahcich3: [ITHREAD] ahcich4: at channel 2 on ahci1 ahcich4: [ITHREAD] ahcich5: at channel 3 on ahci1 ahcich5: [ITHREAD] ahcich6: at channel 4 on ahci1 ahcich6: [ITHREAD] ahcich7: at channel 5 on ahci1 ahcich7: [ITHREAD] pci0: at device 31.3 (no driver attached) acpi_hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 atrtc0: port 0x70-0x73 on acpi0 qpi0: on motherboard orm0: at iomem 0xd5000-0xd5fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: cannot reserve I/O port range coretemp0: on cpu0 est0: on cpu0 p4tcc0: on cpu0 coretemp1: on cpu1 est1: on cpu1 p4tcc1: on cpu1 coretemp2: on cpu2 est2: on cpu2 p4tcc2: on cpu2 coretemp3: on cpu3 est3: on cpu3 p4tcc3: on cpu3 coretemp4: on cpu4 est4: on cpu4 p4tcc4: on cpu4 coretemp5: on cpu5 est5: on cpu5 p4tcc5: on cpu5 coretemp6: on cpu6 est6: on cpu6 p4tcc6: on cpu6 coretemp7: on cpu7 est7: on cpu7 p4tcc7: on cpu7 Timecounters tick every 1.000 msec usbus0: 5.0Gbps Super Speed USB v3.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 480Mbps High Speed USB v2.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 12Mbps Full Speed USB v1.0 usbus8: 480Mbps High Speed USB v2.0 ugen0.1: <0x1033> at usbus0 uhub0: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 ugen8.1: at usbus8 uhub8: on usbus8 ad4: 57241MB at ata2-master UDMA100 SATA 3Gb/s ad6: 57241MB at ata3-master UDMA100 SATA 3Gb/s uhub0: 4 ports with 4 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered uhub7: 2 ports with 2 removable, self powered uhub4: 6 ports with 6 removable, self powered uhub8: 6 ports with 6 removable, self powered ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 57241MB (117231408 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 57241MB (117231408 512 byte sectors: 16H 63S/T 16383C) ada2 at ahcich2 bus 0 scbus3 target 0 lun 0 ada2: ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3 at ahcich3 bus 0 scbus4 target 0 lun 0 ada3: ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada4 at ahcich4 bus 0 scbus5 target 0 lun 0 ada4: ATA-8 SATA 2.x device ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada5 at ahcich5 bus 0 scbus6 target 0 lun 0 ada5: ATA-8 SATA 2.x device ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada6 at ahcich6 bus 0 scbus7 target 0 lun 0 ada6: ATA-8 SATA 2.x device ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada7 at ahcich7 bus 0 scbus8 target 0 lun 0 ada7: ATA-8 SATA 3.x device ada7: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada7: Command Queueing enabled ada7: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) da0 at arcmsr0 bus 0 scbus2 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da0: Command Queueing enabled da0: 953869MB (1953523712 512 byte sectors: 255H 63S/T 121601C) da1 at arcmsr0 bus 0 scbus2 target 0 lun 1 da1: Fixed Direct Access SCSI-5 device da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da1: Command Queueing enabled da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da2 at arcmsr0 bus 0 scbus2 target 0 lun 2 da2: Fixed Direct Access SCSI-5 device da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da2: Command Queueing enabled da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da3 at arcmsr0 bus 0 scbus2 target 0 lun 3 da3: Fixed Direct Access SCSI-5 device da3: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da3: Command Queueing enabled da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da4 at arcmsr0 bus 0 scbus2 target 0 lun 4 da4: Fixed Direct Access SCSI-5 device da4: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da4: Command Queueing enabled da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da5 at arcmsr0 bus 0 scbus2 target 0 lun 5 da5: Fixed Direct Access SCSI-5 device da5: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da5: Command Queueing enabled da5: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da6 at arcmsr0 bus 0 scbus2 target 0 lun 6 da6: Fixed Direct Access SCSI-5 device da6: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da6: Command Queueing enabled da6: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) pass9 at arcmsr0 bus 0 scbus2 target 16 lun 0 pass9: Fixed Processor SCSI-0 device SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #2 Launched! ugen1.2: at usbus1 ums0: on usbus1 ums0: 3 buttons and [XYZ] coordinates ID=0 ugen2.2: at usbus2 ukbd0: on usbus2 kbd2 at ukbd0 uhid0: on usbus2 Trying to mount root from ufs:/dev/da0p3 ugen3.2: at usbus3 ZFS filesystem version 5 ZFS storage pool version 28 drm0: on vgapci0 info: [drm] MSI enabled 1 message(s) info: [drm] Initialized radeon 1.31.0 20080613