From owner-freebsd-stable@FreeBSD.ORG Mon Dec 1 14:17:44 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 841AA10656D8 for ; Mon, 1 Dec 2008 14:17:44 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from mail.transactionware.com (mail.transactionware.com [203.14.245.7]) by mx1.freebsd.org (Postfix) with SMTP id 7C6DB8FC27 for ; Mon, 1 Dec 2008 14:17:43 +0000 (UTC) (envelope-from janm@transactionware.com) Received: (qmail 39639 invoked from network); 1 Dec 2008 14:00:52 -0000 Received: from midgard.transactionware.com (192.168.1.55) by dm.transactionware.com with SMTP; 1 Dec 2008 14:00:52 -0000 Received: (qmail 2644 invoked by uid 907); 1 Dec 2008 14:00:29 -0000 Received: from midgard.transactionware.com (HELO jmlaptop) (192.168.1.55) by midgard.transactionware.com (qpsmtpd/0.40) with SMTP; Tue, 02 Dec 2008 01:00:29 +1100 Message-ID: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop> From: "Jan Mikkelsen" To: Date: Tue, 2 Dec 2008 01:00:23 +1100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5512 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 Subject: 7.1-PRERELEASE: arcmsr write performance problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Dec 2008 14:17:44 -0000 Hi, I am seeing extremely poor performance (~100kB/s) when untaring large tar files into fresh ufs filesystems. I see the problem with softupdates and without softupdates but with an async mount. This is a Supermicro X7DB8 board, 4GB, 2 x Xeon 5140. Sample gstat output: dT: 1.033s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 585 61 0 0 0.0 61 170 13812.0 100.1| da2 I see ms/w start at about 200ms with a ~3MB/s throughput, and then I see ms/w rise and kBps drop. ms/w goes as high as 16-20s, and then suddenly drops back down to about 200ms. Using iostat, while the performance is high(er), kb/t is 64kB, as the problem starts it drops towards 2kB. Copying a single large file doesn't exhibit this problem, although throughput isn't great (~3-5MB/s). However, that's better that 100kB/s. arcmsr0: mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0 > on pci10 ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06 arcmsr0: [ITHREAD] There are eight disks connected in a RAID-6 configuration. The controller's cache is write-through and the disks' write caches are disabled. NCQ is enabled on the drives. The same hardware when it ran 6.3-p1 didn't have this problem. However, the system BIOS was updated at the same time as the operating system (in an attempt to solve a recent em problem), so it is possible that it is a BIOS related problem. The same build on an entirely different machine with an aac controller and SAS disks also doesn't show this problem. Running 'devinfo -r' doesn't list arcmsr as having an interrupt at all. (see below). That strikes me as odd; checking another machine that is still running 6.2 with an arcmsr controller, I can see the interrupt just fine. So: - Does anyone have any suggestions? - Is it normal for arcmsr to not show an interrupt in the output from devinfo in 7.1? Full dmesg, devinfo below. Thanks, Jan Mikkelsen Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-PRERELEASE #0: Mon Dec 1 14:53:12 EST 2008 root@valhalla.transactionware.com:/home/janm/p4/freebsd-image-std-2008.2/work/base-freebsd/home/janm/p4/freebsd-image-std-2008.2/FreeBSD/src/sys/TW-SMP Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2333.35-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 Features=0xbfebfbff Features2=0x4e3bd AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 2 usable memory = 4280651776 (4082 MB) avail memory = 4117843968 (3927 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: at device 0.0 on pci3 pci4: on pcib4 ahd0: port 0x2400-0x24ff,0x2000-0x20ff mem 0xd8500000-0xd8501fff irq 16 at device 2.0 on pci4 ahd0: [ITHREAD] aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs ahd1: port 0x2c00-0x2cff,0x2800-0x28ff mem 0xd8502000-0xd8503fff irq 17 at device 2.1 on pci4 ahd1: [ITHREAD] aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs pcib5: at device 0.2 on pci3 pci5: on pcib5 bge0: mem 0xd8600000-0xd860ffff irq 16 at device 1.0 on pci5 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:40:f4:66:b1:56 bge0: [ITHREAD] pcib6: irq 18 at device 2.0 on pci2 pci6: on pcib6 em0: port 0x3000-0x301f mem 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:30:48:31:67:86 em1: port 0x3020-0x303f mem 0xd8420000-0xd843ffff irq 19 at device 0.1 on pci6 em1: Using MSI interrupt em1: [FILTER] em1: Ethernet address: 00:30:48:31:67:87 pcib7: at device 0.3 on pci1 pci7: on pcib7 pcib8: at device 4.0 on pci0 pci8: on pcib8 pcib9: at device 6.0 on pci0 pci9: on pcib9 pcib10: at device 0.0 on pci9 pci10: on pcib10 arcmsr0: mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0 > on pci10 ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06 arcmsr0: [ITHREAD] pcib11: at device 0.2 on pci9 pci11: on pcib11 pci0: at device 8.0 (no driver attached) pcib12: irq 17 at device 28.0 on pci0 pci12: on pcib12 uhci0: port 0x1800-0x181f irq 17 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x1820-0x183f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x1840-0x185f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered ehci0: mem 0xd8c00400-0xd8c007ff irq 17 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: on ehci0 usb3: USB revision 2.0 uhub3: on usb3 uhub3: 6 ports with 6 removable, self powered pcib13: at device 30.0 on pci0 pci13: on pcib13 vgapci0: port 0x4000-0x40ff mem 0xd0000000-0xd7ffffff,0xd8800000-0xd880ffff irq 18 at device 1.0 on pci13 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model Generic PS/2 mouse, device ID 0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: on ppc0 ppbus0: [ITHREAD] plip0: on ppbus0 plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] cpu0: on acpi0 est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 est1: on cpu1 p4tcc1: on cpu1 cpu2: on acpi0 est2: on cpu2 p4tcc2: on cpu2 cpu3: on acpi0 est3: on cpu3 p4tcc3: on cpu3 orm0: at iomem 0xc0000-0xcafff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec acd0: DVDR at ata0-master UDMA66 Waiting 5 seconds for SCSI devices to settle (probe46:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step da0 at arcmsr0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da0: 77247MB (158201856 512 byte sectors: 255H 63S/T 9847C) da1 at arcmsr0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da1: 953673MB (1953122304 512 byte sectors: 255H 63S/T 121576C) da2 at arcmsr0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da2: 800131MB (1638669312 512 byte sectors: 255H 63S/T 102002C) sa0 at ahd1 bus 0 target 6 lun 0 sa0: Removable Sequential Access SCSI-2 device sa0: 10.000MB/s transfers (10.000MHz, offset 15) SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Trying to mount root from ufs:/dev/da0s2a This module (opensolaris) contains code covered by the Common Development and Distribution License (CDDL) see http://opensolaris.org/os/licensing/opensolaris_license/ WARNING: ZFS is considered to be an experimental feature in FreeBSD. ZFS filesystem version 6 ZFS storage pool version 6 bge0: link state changed to UP em0: link state changed to UP em1: link state changed to UP nexus0 acpi0 Interrupt request lines: 9 I/O ports: 0x10-0x1f 0x24-0x25 0x28-0x29 0x2c-0x2d 0x2e-0x2f 0x30-0x31 0x34-0x35 0x38-0x39 0x3c-0x3d 0x4e-0x4f 0x50-0x53 0x63 0x65 0x67 0x72-0x77 0x80 0x90-0x9f 0xa4-0xa5 0xa8-0xa9 0xac-0xad 0xb0-0xb5 0xb8-0xb9 0xbc-0xbd 0x295-0x296 0x4d0-0x4d1 0x800-0x80f 0xca2-0xca3 0xca8-0xcaf 0x1000-0x107f 0x1180-0x11bf 0xfe00 I/O memory addresses: 0xe0000000-0xefffffff 0xfe000000-0xfe01ffff 0xfe600000-0xfe6fffff 0xfec80000-0xfec80fff 0xfed1c000-0xfed1ffff 0xfee00000-0xfee0ffff cpu0 acpi_perf0 est0 p4tcc0 cpufreq0 cpu1 acpi_perf1 est1 p4tcc1 cpufreq1 cpu2 acpi_perf2 est2 p4tcc2 cpufreq2 cpu3 acpi_perf3 est3 p4tcc3 cpufreq3 pcib0 pci0 hostb0 pcib1 pci1 pcib2 pci2 pcib3 pci3 pcib4 pci4 ahd0 Interrupt request lines: 16 I/O ports: 0x2000-0x20ff 0x2400-0x24ff I/O memory addresses: 0xd8500000-0xd8501fff ahd1 Interrupt request lines: 17 I/O ports: 0x2800-0x28ff 0x2c00-0x2cff I/O memory addresses: 0xd8502000-0xd8503fff pcib5 pci5 bge0 I/O memory addresses: 0xd8600000-0xd860ffff miibus0 brgphy0 pcib6 pci6 em0 Interrupt request lines: 256 I/O ports: 0x3000-0x301f I/O memory addresses: 0xd8400000-0xd841ffff em1 Interrupt request lines: 257 I/O ports: 0x3020-0x303f I/O memory addresses: 0xd8420000-0xd843ffff pcib7 pci7 pcib8 pci8 pcib9 pci9 pcib10 pci10 arcmsr0 I/O memory addresses: 0xd8000000-0xd83fffff 0xd8900000-0xd8900fff pcib11 pci11 hostb1 hostb2 hostb3 hostb4 hostb5 hostb6 hostb7 pcib12 pci12 uhci0 I/O ports: 0x1800-0x181f usb0 uhub0 uhci1 Interrupt request lines: 19 I/O ports: 0x1820-0x183f usb1 uhub1 uhci2 Interrupt request lines: 18 I/O ports: 0x1840-0x185f usb2 uhub2 ehci0 I/O memory addresses: 0xd8c00400-0xd8c007ff usb3 uhub3 pcib13 pci13 vgapci0 I/O ports: 0x4000-0x40ff I/O memory addresses: 0xd0000000-0xd7ffffff 0xd8800000-0xd880ffff isab0 isa0 sc0 vga0 I/O ports: 0x3c0-0x3df I/O memory addresses: 0xa0000-0xbffff orm0 I/O memory addresses: 0xc0000-0xcafff atapci0 I/O ports: 0x170-0x177 0x1f0-0x1f7 0x376 0x3f6 0x1860-0x186f ata0 Interrupt request lines: 14 acd0 acpi_sysresource0 atdma0 fpupnp0 attimer0 attimer1 pci_link0 pci_link1 pci_link2 pci_link3 pci_link4 pci_link5 pci_link6 pci_link7 atkbdc0 I/O ports: 0x60 0x64 atkbd0 Interrupt request lines: 1 psm0 Interrupt request lines: 12 psmcpnp0 sio0 Interrupt request lines: 4 I/O ports: 0x3f8-0x3ff sio1 Interrupt request lines: 3 I/O ports: 0x2f8-0x2ff fdc0 Interrupt request lines: 6 DMA request lines: 2 I/O ports: 0x3f0-0x3f5 0x3f7 ppc0 Interrupt request lines: 7 DMA request lines: 3 I/O ports: 0x378-0x37f ppbus0 plip0 lpt0 ppi0 acpi_button0 acpi_timer0 ACPI I/O ports: 0x1008-0x100b apic0 I/O memory addresses: 0xfec00000-0xfec0001f ram0 I/O memory addresses: 0x0-0x9dfff 0x100000-0xcff4ffff 0x100000000-0x12fffffff