Date: Tue, 2 Dec 2008 01:00:23 +1100 From: "Jan Mikkelsen" <janm@transactionware.com> To: <freebsd-stable@freebsd.org> Subject: 7.1-PRERELEASE: arcmsr write performance problem Message-ID: <2515BCEE3A2F4CBA8FFB9F1C052924AA@jmlaptop>
next in thread | raw e-mail | index | archive | help
Hi,
I am seeing extremely poor performance (~100kB/s) when untaring large tar
files into fresh ufs filesystems. I see the problem with softupdates and
without softupdates but with an async mount. This is a Supermicro X7DB8
board, 4GB, 2 x Xeon 5140.
Sample gstat output:
dT: 1.033s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
585 61 0 0 0.0 61 170 13812.0 100.1| da2
I see ms/w start at about 200ms with a ~3MB/s throughput, and then I see
ms/w rise and kBps drop. ms/w goes as high as 16-20s, and then suddenly
drops back down to about 200ms. Using iostat, while the performance is
high(er), kb/t is 64kB, as the problem starts it drops towards 2kB.
Copying a single large file doesn't exhibit this problem, although
throughput isn't great (~3-5MB/s). However, that's better that 100kB/s.
arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
> mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0
> on pci10
ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06
arcmsr0: [ITHREAD]
There are eight disks connected in a RAID-6 configuration. The
controller's cache is write-through and the disks' write caches are
disabled. NCQ is enabled on the drives.
The same hardware when it ran 6.3-p1 didn't have this problem. However,
the system BIOS was updated at the same time as the operating system (in
an attempt to solve a recent em problem), so it is possible that it is a
BIOS related problem. The same build on an entirely different machine
with an aac controller and SAS disks also doesn't show this problem.
Running 'devinfo -r' doesn't list arcmsr as having an interrupt at all.
(see below). That strikes me as odd; checking another machine that is
still running 6.2 with an arcmsr controller, I can see the interrupt just
fine.
So:
- Does anyone have any suggestions?
- Is it normal for arcmsr to not show an interrupt in the output from
devinfo in 7.1?
Full dmesg, devinfo below.
Thanks,
Jan Mikkelsen
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-PRERELEASE #0: Mon Dec 1 14:53:12 EST 2008
root@valhalla.transactionware.com:/home/janm/p4/freebsd-image-std-2008.2/work/base-freebsd/home/janm/p4/freebsd-image-std-2008.2/FreeBSD/src/sys/TW-SMP
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2333.35-MHz
K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
usable memory = 4280651776 (4082 MB)
avail memory = 4117843968 (3927 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413,
RF5413)
acpi0: <SMCI SMCISLP2> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port
0x2400-0x24ff,0x2000-0x20ff mem 0xd8500000-0xd8501fff irq 16 at device
2.0 on pci4
ahd0: [ITHREAD]
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port
0x2c00-0x2cff,0x2800-0x28ff mem 0xd8502000-0xd8503fff irq 17 at device
2.1 on pci4
ahd1: [ITHREAD]
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
bge0: <Altima Gigabit Ethernet Controller, ASIC rev. 0x105> mem
0xd8600000-0xd860ffff irq 16 at device 1.0 on pci5
miibus0: <MII bus> on bge0
brgphy0: <BCM5701 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge0: Ethernet address: 00:40:f4:66:b1:56
bge0: [ITHREAD]
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3000-0x301f mem
0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:30:48:31:67:86
em1: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3020-0x303f mem
0xd8420000-0xd843ffff irq 19 at device 0.1 on pci6
em1: Using MSI interrupt
em1: [FILTER]
em1: Ethernet address: 00:30:48:31:67:87
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <PCI-PCI bridge> at device 0.0 on pci9
pci10: <PCI bus> on pcib10
arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
> mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0
> on pci10
ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06
arcmsr0: [ITHREAD]
pcib11: <PCI-PCI bridge> at device 0.2 on pci9
pci11: <PCI bus> on pcib11
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib12: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci12: <ACPI PCI bus> on pcib12
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port
0x1800-0x181f irq 17 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port
0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd8c00400-0xd8c007ff irq
17 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 63XXESB USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 6 ports with 6 removable, self powered
pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci13: <ACPI PCI bus> on pcib13
vgapci0: <VGA-compatible display> port 0x4000-0x40ff mem
0xd0000000-0xd7ffffff,0xd8800000-0xd880ffff irq 18 at device 1.0 on pci13
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model Generic PS/2 mouse, device ID 0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
plip0: <PLIP network interface> on ppbus0
plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: DVDR <PIONEER DVD-RW DVR-111D/1.23> at ata0-master UDMA66
Waiting 5 seconds for SCSI devices to settle
(probe46:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step
da0 at arcmsr0 bus 0 target 0 lun 0
da0: <Areca ARC-1220-VOL#00 R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da0: 77247MB (158201856 512 byte sectors: 255H 63S/T 9847C)
da1 at arcmsr0 bus 0 target 1 lun 0
da1: <Areca ARC-1220-VOL#01 R001> Fixed Direct Access SCSI-5 device
da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da1: 953673MB (1953122304 512 byte sectors: 255H 63S/T 121576C)
da2 at arcmsr0 bus 0 target 2 lun 0
da2: <Areca ARC-1220-VOL#02 R001> Fixed Direct Access SCSI-5 device
da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da2: 800131MB (1638669312 512 byte sectors: 255H 63S/T 102002C)
sa0 at ahd1 bus 0 target 6 lun 0
sa0: <SEAGATE DAT 04106-XXX 7600> Removable Sequential Access SCSI-2
device
sa0: 10.000MB/s transfers (10.000MHz, offset 15)
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/da0s2a
This module (opensolaris) contains code covered by the
Common Development and Distribution License (CDDL)
see http://opensolaris.org/os/licensing/opensolaris_license/
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6
bge0: link state changed to UP
em0: link state changed to UP
em1: link state changed to UP
nexus0
acpi0
Interrupt request lines:
9
I/O ports:
0x10-0x1f
0x24-0x25
0x28-0x29
0x2c-0x2d
0x2e-0x2f
0x30-0x31
0x34-0x35
0x38-0x39
0x3c-0x3d
0x4e-0x4f
0x50-0x53
0x63
0x65
0x67
0x72-0x77
0x80
0x90-0x9f
0xa4-0xa5
0xa8-0xa9
0xac-0xad
0xb0-0xb5
0xb8-0xb9
0xbc-0xbd
0x295-0x296
0x4d0-0x4d1
0x800-0x80f
0xca2-0xca3
0xca8-0xcaf
0x1000-0x107f
0x1180-0x11bf
0xfe00
I/O memory addresses:
0xe0000000-0xefffffff
0xfe000000-0xfe01ffff
0xfe600000-0xfe6fffff
0xfec80000-0xfec80fff
0xfed1c000-0xfed1ffff
0xfee00000-0xfee0ffff
cpu0
acpi_perf0
est0
p4tcc0
cpufreq0
cpu1
acpi_perf1
est1
p4tcc1
cpufreq1
cpu2
acpi_perf2
est2
p4tcc2
cpufreq2
cpu3
acpi_perf3
est3
p4tcc3
cpufreq3
pcib0
pci0
hostb0
pcib1
pci1
pcib2
pci2
pcib3
pci3
pcib4
pci4
ahd0
Interrupt request lines:
16
I/O ports:
0x2000-0x20ff
0x2400-0x24ff
I/O memory addresses:
0xd8500000-0xd8501fff
ahd1
Interrupt request lines:
17
I/O ports:
0x2800-0x28ff
0x2c00-0x2cff
I/O memory addresses:
0xd8502000-0xd8503fff
pcib5
pci5
bge0
I/O memory addresses:
0xd8600000-0xd860ffff
miibus0
brgphy0
pcib6
pci6
em0
Interrupt request lines:
256
I/O ports:
0x3000-0x301f
I/O memory addresses:
0xd8400000-0xd841ffff
em1
Interrupt request lines:
257
I/O ports:
0x3020-0x303f
I/O memory addresses:
0xd8420000-0xd843ffff
pcib7
pci7
pcib8
pci8
pcib9
pci9
pcib10
pci10
arcmsr0
I/O memory addresses:
0xd8000000-0xd83fffff
0xd8900000-0xd8900fff
pcib11
pci11
hostb1
hostb2
hostb3
hostb4
hostb5
hostb6
hostb7
pcib12
pci12
uhci0
I/O ports:
0x1800-0x181f
usb0
uhub0
uhci1
Interrupt request lines:
19
I/O ports:
0x1820-0x183f
usb1
uhub1
uhci2
Interrupt request lines:
18
I/O ports:
0x1840-0x185f
usb2
uhub2
ehci0
I/O memory addresses:
0xd8c00400-0xd8c007ff
usb3
uhub3
pcib13
pci13
vgapci0
I/O ports:
0x4000-0x40ff
I/O memory addresses:
0xd0000000-0xd7ffffff
0xd8800000-0xd880ffff
isab0
isa0
sc0
vga0
I/O ports:
0x3c0-0x3df
I/O memory addresses:
0xa0000-0xbffff
orm0
I/O memory addresses:
0xc0000-0xcafff
atapci0
I/O ports:
0x170-0x177
0x1f0-0x1f7
0x376
0x3f6
0x1860-0x186f
ata0
Interrupt request lines:
14
acd0
acpi_sysresource0
atdma0
fpupnp0
attimer0
attimer1
pci_link0
pci_link1
pci_link2
pci_link3
pci_link4
pci_link5
pci_link6
pci_link7
atkbdc0
I/O ports:
0x60
0x64
atkbd0
Interrupt request lines:
1
psm0
Interrupt request lines:
12
psmcpnp0
sio0
Interrupt request lines:
4
I/O ports:
0x3f8-0x3ff
sio1
Interrupt request lines:
3
I/O ports:
0x2f8-0x2ff
fdc0
Interrupt request lines:
6
DMA request lines:
2
I/O ports:
0x3f0-0x3f5
0x3f7
ppc0
Interrupt request lines:
7
DMA request lines:
3
I/O ports:
0x378-0x37f
ppbus0
plip0
lpt0
ppi0
acpi_button0
acpi_timer0
ACPI I/O ports:
0x1008-0x100b
apic0
I/O memory addresses:
0xfec00000-0xfec0001f
ram0
I/O memory addresses:
0x0-0x9dfff
0x100000-0xcff4ffff
0x100000000-0x12fffffff
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2515BCEE3A2F4CBA8FFB9F1C052924AA>