Date: Tue, 12 May 2009 14:16:21 -0400 (EDT) From: Andy Dills <andy@xecu.net> To: freebsd-questions@freebsd.org Subject: Extreme filesystem performance decrese going from FreeBSD4 to FreeBSD7 Message-ID: <20090512135226.N64283@shell.xecu.net>
next in thread | raw e-mail | index | archive | help
A customer of a customer needed to expand the size of the filesystem on a
webserver they run, and since it was still running 4.11, they decided to
go ahead and upgrade to 7.1.
Unfortunately, they have seen the performance decrease significantly. Now,
under 7.1, numerous perl processes will queue up (from their web
applications), the load average will be in the 40s to 90s, and according
to top they perl scripts are commonly in the "ufs" state, which I assume
means stuck waiting for read or write responses.
I have also experienced pretty abysmal read performance from the array,
using dd...we're talking sub 1MBps. So, my assumption is that something is
wrong on the filesystem layer.
When I look at the CPU utilization in top, I see that 80-90% is constantly
used by "system".
However, when I look at iostat, I see very low numbers for the raid, in
fact across the board.
It's a Dell server with a Perc4/Di RAID controller, which uses the amr
driver. They upgraded to larger, 15k rpm ultra 320 disks (from smaller 15k
rpm ultra 320 disks). They made a backup of their web root, which is a
seperate partition, using dump to a temporary drive, then swapped in the
new disks, installed FreeBSD 7.1, and restored their webroot to a
partition on the new array.
Does anybody have any insight into what could be going on here?
dmesg is below.
Thanks!
Andy
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-RELEASE #0: Thu Jan 1 14:37:25 UTC 2009
root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3056.81-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4400<CNXT-ID,xTPR>
Logical CPUs per core: 2
real memory = 4160552960 (3967 MB)
avail memory = 4069429248 (3880 MB)
ACPI APIC Table: <DELL PE2600 >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic2: Changing APIC ID to 10
ioapic3: Changing APIC ID to 11
ioapic4: Changing APIC ID to 12
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 72-95 on motherboard
ioapic3 <Version 2.0> irqs 120-143 on motherboard
ioapic4 <Version 2.0> irqs 144-167 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <DELL PE2600> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci1
pci3: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xece0-0xecff mem
0xfe8e0000-0xfe8fffff,0xfe8c0000-0xfe8dffff irq 28 at device 1.0 on pci3
em0: [FILTER]
em0: Ethernet address: 00:0f:1f:69:e3:7d
pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci4
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci4
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 29.0 on pci7
pci8: <ACPI PCI bus> on pcib8
amr0: <LSILogic MegaRAID 1.53> mem 0xfebf0000-0xfebfffff irq 120 at device
8.0 on pci8
amr0: [ITHREAD]
amr0: delete logical drives supported by controller
amr0: <LSILogic PERC 4/Di> Firmware 2.48, BIOS 1.06, 128MB RAM
pcib9: <ACPI PCI-PCI bridge> at device 31.0 on pci7
pci10: <ACPI PCI bus> on pcib9
uhci0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> port 0xcce0-0xccff
irq 16 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci11: <ACPI PCI bus> on pcib10
vgapci0: <VGA-compatible display> port 0xdc00-0xdcff mem
0xfc000000-0xfcffffff,0xfdeff000-0xfdefffff at device 4.0 on pci11
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse Explorer, device ID 4
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xec000-0xeffff pnpid
ORM0000 on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
plip0: <PLIP network interface> on ppbus0
plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: CDRW <SAMSUNG CDRW/DVD SN-324F/U204> at ata0-master UDMA33
amr0: delete logical drives supported by controller
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 419700MB (859545600 sectors) RAID 5 (optimal)
amrd1: <LSILogic MegaRAID logical drive> on amr0
amrd1: 139900MB (286515200 sectors) RAID 0 (optimal)
ses0 at amr0 bus 0 target 6 lun 0
ses0: <PE/PV 1x6 SCSI BP 1.1> Fixed Processor SCSI-2 device
ses0: 166.664MB/s transfers (41.666MHz, offset 31, 32bit)
ses0: SAF-TE Compliant Device
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/amrd0s1a
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090512135226.N64283>