From owner-freebsd-stable@FreeBSD.ORG Tue Dec 2 05:45:31 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B26361065673 for ; Tue, 2 Dec 2008 05:45:31 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from mail.transactionware.com (mail.transactionware.com [203.14.245.7]) by mx1.freebsd.org (Postfix) with SMTP id A31398FC22 for ; Tue, 2 Dec 2008 05:45:30 +0000 (UTC) (envelope-from janm@transactionware.com) Received: (qmail 80933 invoked from network); 2 Dec 2008 05:45:50 -0000 Received: from midgard.transactionware.com (192.168.1.55) by dm.transactionware.com with SMTP; 2 Dec 2008 05:45:50 -0000 Received: (qmail 9538 invoked by uid 907); 2 Dec 2008 05:45:27 -0000 Received: from [192.168.1.32] (HELO [192.168.1.32]) (192.168.1.32) by midgard.transactionware.com (qpsmtpd/0.32) with ESMTP; Tue, 02 Dec 2008 16:45:27 +1100 Message-ID: <4934CB77.30906@transactionware.com> Date: Tue, 02 Dec 2008 16:45:27 +1100 From: Jan Mikkelsen User-Agent: Thunderbird 2.0.0.18 (Windows/20081105) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> In-Reply-To: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2008 05:45:31 -0000 Replying to my own post ... I have done a test on the same machine comparing 6.3-p1 to 7.1-PRE. The performance is the expected ~6MB/s (because of the lack of cache) on 6.3-p1, so the BIOS change doesn't seem to be at fault. This seems to be a regression somewhere between 6.3 to 7.1. The Areca driver is the same in 6.3 and 7.1, so the problem seems to be elsewhere. I think this is more than just a "performance" problem. The observations with gstat showing extremely high ms/w values (I have seen them as high as 22000) makes it look like IO completion interrupts are being lost. Any suggestions on where to look next? Are there obvious candidates? Jan Mikkelsen wrote: > Hi, > > I am seeing extremely poor performance (~100kB/s) when untaring large > tar files into fresh ufs filesystems. I see the problem with > softupdates and without softupdates but with an async mount. This is a > Supermicro X7DB8 board, 4GB, 2 x Xeon 5140. > > Sample gstat output: > > dT: 1.033s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 585 61 0 0 0.0 61 170 13812.0 100.1| da2 > > I see ms/w start at about 200ms with a ~3MB/s throughput, and then I see > ms/w rise and kBps drop. ms/w goes as high as 16-20s, and then suddenly > drops back down to about 200ms. Using iostat, while the performance is > high(er), kb/t is 64kB, as the problem starts it drops towards 2kB. > > Copying a single large file doesn't exhibit this problem, although > throughput isn't great (~3-5MB/s). However, that's better that 100kB/s. > > arcmsr0: > mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0 >> on pci10 > ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 > ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06 > arcmsr0: [ITHREAD] > > There are eight disks connected in a RAID-6 configuration. The > controller's cache is write-through and the disks' write caches are > disabled. NCQ is enabled on the drives. > > The same hardware when it ran 6.3-p1 didn't have this problem. However, > the system BIOS was updated at the same time as the operating system (in > an attempt to solve a recent em problem), so it is possible that it is a > BIOS related problem. The same build on an entirely different machine > with an aac controller and SAS disks also doesn't show this problem. > > Running 'devinfo -r' doesn't list arcmsr as having an interrupt at all. > (see below). That strikes me as odd; checking another machine that is > still running 6.2 with an arcmsr controller, I can see the interrupt > just fine. > > So: > > - Does anyone have any suggestions? > > - Is it normal for arcmsr to not show an interrupt in the output from > devinfo in 7.1? > > Full dmesg, devinfo below. > > Thanks, > > Jan Mikkelsen > > > Copyright (c) 1992-2008 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.1-PRERELEASE #0: Mon Dec 1 14:53:12 EST 2008 > > root@valhalla.transactionware.com:/home/janm/p4/freebsd-image-std-2008.2/work/base-freebsd/home/janm/p4/freebsd-image-std-2008.2/FreeBSD/src/sys/TW-SMP > > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2333.35-MHz > K8-class CPU) > Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 > Features=0xbfebfbff > > Features2=0x4e3bd > > AMD Features=0x20100800 > AMD Features2=0x1 > Cores per package: 2 > usable memory = 4280651776 (4082 MB) > avail memory = 4117843968 (3927 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 6 > cpu3 (AP): APIC ID: 7 > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-47 on motherboard > kbd1 at kbdmux0 > ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) > acpi0: on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: at device 2.0 on pci0 > pci1: on pcib1 > pcib2: irq 16 at device 0.0 on pci1 > pci2: on pcib2 > pcib3: irq 16 at device 0.0 on pci2 > pci3: on pcib3 > pcib4: at device 0.0 on pci3 > pci4: on pcib4 > ahd0: port > 0x2400-0x24ff,0x2000-0x20ff mem 0xd8500000-0xd8501fff irq 16 at device > 2.0 on pci4 > ahd0: [ITHREAD] > aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs > ahd1: port > 0x2c00-0x2cff,0x2800-0x28ff mem 0xd8502000-0xd8503fff irq 17 at device > 2.1 on pci4 > ahd1: [ITHREAD] > aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs > pcib5: at device 0.2 on pci3 > pci5: on pcib5 > bge0: mem > 0xd8600000-0xd860ffff irq 16 at device 1.0 on pci5 > miibus0: on bge0 > brgphy0: PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-FDX, auto > bge0: Ethernet address: 00:40:f4:66:b1:56 > bge0: [ITHREAD] > pcib6: irq 18 at device 2.0 on pci2 > pci6: on pcib6 > em0: port 0x3000-0x301f mem > 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6 > em0: Using MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:30:48:31:67:86 > em1: port 0x3020-0x303f mem > 0xd8420000-0xd843ffff irq 19 at device 0.1 on pci6 > em1: Using MSI interrupt > em1: [FILTER] > em1: Ethernet address: 00:30:48:31:67:87 > pcib7: at device 0.3 on pci1 > pci7: on pcib7 > pcib8: at device 4.0 on pci0 > pci8: on pcib8 > pcib9: at device 6.0 on pci0 > pci9: on pcib9 > pcib10: at device 0.0 on pci9 > pci10: on pcib10 > arcmsr0: > mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0 >> on pci10 > ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 > ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06 > arcmsr0: [ITHREAD] > pcib11: at device 0.2 on pci9 > pci11: on pcib11 > pci0: at device 8.0 (no driver attached) > pcib12: irq 17 at device 28.0 on pci0 > pci12: on pcib12 > uhci0: port > 0x1800-0x181f irq 17 at device 29.0 on pci0 > uhci0: [GIANT-LOCKED] > uhci0: [ITHREAD] > usb0: on uhci0 > usb0: USB revision 1.0 > uhub0: on usb0 > uhub0: 2 ports with 2 removable, self powered > uhci1: port > 0x1820-0x183f irq 19 at device 29.1 on pci0 > uhci1: [GIANT-LOCKED] > uhci1: [ITHREAD] > usb1: on uhci1 > usb1: USB revision 1.0 > uhub1: on usb1 > uhub1: 2 ports with 2 removable, self powered > uhci2: port > 0x1840-0x185f irq 18 at device 29.2 on pci0 > uhci2: [GIANT-LOCKED] > uhci2: [ITHREAD] > usb2: on uhci2 > usb2: USB revision 1.0 > uhub2: on usb2 > uhub2: 2 ports with 2 removable, self powered > ehci0: mem 0xd8c00400-0xd8c007ff irq > 17 at device 29.7 on pci0 > ehci0: [GIANT-LOCKED] > ehci0: [ITHREAD] > usb3: EHCI version 1.0 > usb3: companion controllers, 2 ports each: usb0 usb1 usb2 > usb3: on ehci0 > usb3: USB revision 2.0 > uhub3: on usb3 > uhub3: 6 ports with 6 removable, self powered > pcib13: at device 30.0 on pci0 > pci13: on pcib13 > vgapci0: port 0x4000-0x40ff mem > 0xd0000000-0xd7ffffff,0xd8800000-0xd880ffff irq 18 at device 1.0 on pci13 > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci0: port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0 > ata0: on atapci0 > ata0: [ITHREAD] > pci0: at device 31.3 (no driver attached) > acpi_button0: on acpi0 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > psm0: irq 12 on atkbdc0 > psm0: [GIANT-LOCKED] > psm0: [ITHREAD] > psm0: model Generic PS/2 mouse, device ID 0 > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on > acpi0 > sio0: type 16550A > sio0: [FILTER] > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 > sio1: type 16550A > sio1: [FILTER] > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > fdc0: [FILTER] > ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppbus0: on ppc0 > ppbus0: [ITHREAD] > plip0: on ppbus0 > plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag > lpt0: on ppbus0 > lpt0: Interrupt-driven port > ppi0: on ppbus0 > ppc0: [GIANT-LOCKED] > ppc0: [ITHREAD] > cpu0: on acpi0 > est0: on cpu0 > p4tcc0: on cpu0 > cpu1: on acpi0 > est1: on cpu1 > p4tcc1: on cpu1 > cpu2: on acpi0 > est2: on cpu2 > p4tcc2: on cpu2 > cpu3: on acpi0 > est3: on cpu3 > p4tcc3: on cpu3 > orm0: at iomem 0xc0000-0xcafff on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Timecounters tick every 1.000 msec > acd0: DVDR at ata0-master UDMA66 > Waiting 5 seconds for SCSI devices to settle > (probe46:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step > da0 at arcmsr0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da0: 77247MB (158201856 512 byte sectors: 255H 63S/T 9847C) > da1 at arcmsr0 bus 0 target 1 lun 0 > da1: Fixed Direct Access SCSI-5 device > da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da1: 953673MB (1953122304 512 byte sectors: 255H 63S/T 121576C) > da2 at arcmsr0 bus 0 target 2 lun 0 > da2: Fixed Direct Access SCSI-5 device > da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da2: 800131MB (1638669312 512 byte sectors: 255H 63S/T 102002C) > sa0 at ahd1 bus 0 target 6 lun 0 > sa0: Removable Sequential Access SCSI-2 > device > sa0: 10.000MB/s transfers (10.000MHz, offset 15) > SMP: AP CPU #1 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #2 Launched! > Trying to mount root from ufs:/dev/da0s2a > This module (opensolaris) contains code covered by the > Common Development and Distribution License (CDDL) > see http://opensolaris.org/os/licensing/opensolaris_license/ > WARNING: ZFS is considered to be an experimental feature in FreeBSD. > ZFS filesystem version 6 > ZFS storage pool version 6 > bge0: link state changed to UP > em0: link state changed to UP > em1: link state changed to UP > > > nexus0 > acpi0 > Interrupt request lines: > 9 > I/O ports: > 0x10-0x1f > 0x24-0x25 > 0x28-0x29 > 0x2c-0x2d > 0x2e-0x2f > 0x30-0x31 > 0x34-0x35 > 0x38-0x39 > 0x3c-0x3d > 0x4e-0x4f > 0x50-0x53 > 0x63 > 0x65 > 0x67 > 0x72-0x77 > 0x80 > 0x90-0x9f > 0xa4-0xa5 > 0xa8-0xa9 > 0xac-0xad > 0xb0-0xb5 > 0xb8-0xb9 > 0xbc-0xbd > 0x295-0x296 > 0x4d0-0x4d1 > 0x800-0x80f > 0xca2-0xca3 > 0xca8-0xcaf > 0x1000-0x107f > 0x1180-0x11bf > 0xfe00 > I/O memory addresses: > 0xe0000000-0xefffffff > 0xfe000000-0xfe01ffff > 0xfe600000-0xfe6fffff > 0xfec80000-0xfec80fff > 0xfed1c000-0xfed1ffff > 0xfee00000-0xfee0ffff > cpu0 > acpi_perf0 > est0 > p4tcc0 > cpufreq0 > cpu1 > acpi_perf1 > est1 > p4tcc1 > cpufreq1 > cpu2 > acpi_perf2 > est2 > p4tcc2 > cpufreq2 > cpu3 > acpi_perf3 > est3 > p4tcc3 > cpufreq3 > pcib0 > pci0 > hostb0 > pcib1 > pci1 > pcib2 > pci2 > pcib3 > pci3 > pcib4 > pci4 > ahd0 > Interrupt request lines: > 16 > I/O ports: > 0x2000-0x20ff > 0x2400-0x24ff > I/O memory addresses: > 0xd8500000-0xd8501fff > ahd1 > Interrupt request lines: > 17 > I/O ports: > 0x2800-0x28ff > 0x2c00-0x2cff > I/O memory addresses: > 0xd8502000-0xd8503fff > pcib5 > pci5 > bge0 > I/O memory addresses: > 0xd8600000-0xd860ffff > miibus0 > brgphy0 > pcib6 > pci6 > em0 > Interrupt request lines: > 256 > I/O ports: > 0x3000-0x301f > I/O memory addresses: > 0xd8400000-0xd841ffff > em1 > Interrupt request lines: > 257 > I/O ports: > 0x3020-0x303f > I/O memory addresses: > 0xd8420000-0xd843ffff > pcib7 > pci7 > pcib8 > pci8 > pcib9 > pci9 > pcib10 > pci10 > arcmsr0 > I/O memory addresses: > 0xd8000000-0xd83fffff > 0xd8900000-0xd8900fff > pcib11 > pci11 > hostb1 > hostb2 > hostb3 > hostb4 > hostb5 > hostb6 > hostb7 > pcib12 > pci12 > uhci0 > I/O ports: > 0x1800-0x181f > usb0 > uhub0 > uhci1 > Interrupt request lines: > 19 > I/O ports: > 0x1820-0x183f > usb1 > uhub1 > uhci2 > Interrupt request lines: > 18 > I/O ports: > 0x1840-0x185f > usb2 > uhub2 > ehci0 > I/O memory addresses: > 0xd8c00400-0xd8c007ff > usb3 > uhub3 > pcib13 > pci13 > vgapci0 > I/O ports: > 0x4000-0x40ff > I/O memory addresses: > 0xd0000000-0xd7ffffff > 0xd8800000-0xd880ffff > isab0 > isa0 > sc0 > vga0 > I/O ports: > 0x3c0-0x3df > I/O memory addresses: > 0xa0000-0xbffff > orm0 > I/O memory addresses: > 0xc0000-0xcafff > atapci0 > I/O ports: > 0x170-0x177 > 0x1f0-0x1f7 > 0x376 > 0x3f6 > 0x1860-0x186f > ata0 > Interrupt request lines: > 14 > acd0 > acpi_sysresource0 > atdma0 > fpupnp0 > attimer0 > attimer1 > pci_link0 > pci_link1 > pci_link2 > pci_link3 > pci_link4 > pci_link5 > pci_link6 > pci_link7 > atkbdc0 > I/O ports: > 0x60 > 0x64 > atkbd0 > Interrupt request lines: > 1 > psm0 > Interrupt request lines: > 12 > psmcpnp0 > sio0 > Interrupt request lines: > 4 > I/O ports: > 0x3f8-0x3ff > sio1 > Interrupt request lines: > 3 > I/O ports: > 0x2f8-0x2ff > fdc0 > Interrupt request lines: > 6 > DMA request lines: > 2 > I/O ports: > 0x3f0-0x3f5 > 0x3f7 > ppc0 > Interrupt request lines: > 7 > DMA request lines: > 3 > I/O ports: > 0x378-0x37f > ppbus0 > plip0 > lpt0 > ppi0 > acpi_button0 > acpi_timer0 > ACPI I/O ports: > 0x1008-0x100b > apic0 > I/O memory addresses: > 0xfec00000-0xfec0001f > ram0 > I/O memory addresses: > 0x0-0x9dfff > 0x100000-0xcff4ffff > 0x100000000-0x12fffffff > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"