From owner-freebsd-questions@FreeBSD.ORG Tue May 12 18:44:54 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0EFA1065672 for ; Tue, 12 May 2009 18:44:54 +0000 (UTC) (envelope-from andy@xecu.net) Received: from shell.xecu.net (shell.xecu.net [216.127.136.216]) by mx1.freebsd.org (Postfix) with ESMTP id 824928FC1C for ; Tue, 12 May 2009 18:44:54 +0000 (UTC) (envelope-from andy@xecu.net) Received: from shell.xecu.net (shell.xecu.net [216.127.136.216]) by shell.xecu.net (Postfix) with ESMTP id E2F8E251B4F for ; Tue, 12 May 2009 14:16:21 -0400 (EDT) Date: Tue, 12 May 2009 14:16:21 -0400 (EDT) From: Andy Dills To: freebsd-questions@freebsd.org Message-ID: <20090512135226.N64283@shell.xecu.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Extreme filesystem performance decrese going from FreeBSD4 to FreeBSD7 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 May 2009 18:44:55 -0000 A customer of a customer needed to expand the size of the filesystem on a webserver they run, and since it was still running 4.11, they decided to go ahead and upgrade to 7.1. Unfortunately, they have seen the performance decrease significantly. Now, under 7.1, numerous perl processes will queue up (from their web applications), the load average will be in the 40s to 90s, and according to top they perl scripts are commonly in the "ufs" state, which I assume means stuck waiting for read or write responses. I have also experienced pretty abysmal read performance from the array, using dd...we're talking sub 1MBps. So, my assumption is that something is wrong on the filesystem layer. When I look at the CPU utilization in top, I see that 80-90% is constantly used by "system". However, when I look at iostat, I see very low numbers for the raid, in fact across the board. It's a Dell server with a Perc4/Di RAID controller, which uses the amr driver. They upgraded to larger, 15k rpm ultra 320 disks (from smaller 15k rpm ultra 320 disks). They made a backup of their web root, which is a seperate partition, using dump to a temporary drive, then swapped in the new disks, installed FreeBSD 7.1, and restored their webroot to a partition on the new array. Does anybody have any insight into what could be going on here? dmesg is below. Thanks! Andy Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-RELEASE #0: Thu Jan 1 14:37:25 UTC 2009 root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3056.81-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff Features2=0x4400 Logical CPUs per core: 2 real memory = 4160552960 (3967 MB) avail memory = 4069429248 (3880 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 ioapic0: Changing APIC ID to 8 ioapic1: Changing APIC ID to 9 ioapic2: Changing APIC ID to 10 ioapic3: Changing APIC ID to 11 ioapic4: Changing APIC ID to 12 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 72-95 on motherboard ioapic3 irqs 120-143 on motherboard ioapic4 irqs 144-167 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: at device 29.0 on pci1 pci2: on pcib2 pcib3: at device 31.0 on pci1 pci3: on pcib3 em0: port 0xece0-0xecff mem 0xfe8e0000-0xfe8fffff,0xfe8c0000-0xfe8dffff irq 28 at device 1.0 on pci3 em0: [FILTER] em0: Ethernet address: 00:0f:1f:69:e3:7d pcib4: at device 3.0 on pci0 pci4: on pcib4 pcib5: at device 29.0 on pci4 pci5: on pcib5 pcib6: at device 31.0 on pci4 pci6: on pcib6 pcib7: at device 4.0 on pci0 pci7: on pcib7 pcib8: at device 29.0 on pci7 pci8: on pcib8 amr0: mem 0xfebf0000-0xfebfffff irq 120 at device 8.0 on pci8 amr0: [ITHREAD] amr0: delete logical drives supported by controller amr0: Firmware 2.48, BIOS 1.06, 128MB RAM pcib9: at device 31.0 on pci7 pci10: on pcib9 uhci0: port 0xcce0-0xccff irq 16 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered pcib10: at device 30.0 on pci0 pci11: on pcib10 vgapci0: port 0xdc00-0xdcff mem 0xfc000000-0xfcffffff,0xfdeff000-0xfdefffff at device 4.0 on pci11 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse Explorer, device ID 4 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] cpu0: on acpi0 p4tcc0: on cpu0 cpu1: on acpi0 p4tcc1: on cpu1 cpu2: on acpi0 p4tcc2: on cpu2 cpu3: on acpi0 p4tcc3: on cpu3 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff,0xec000-0xeffff pnpid ORM0000 on isa0 ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/8 bytes threshold ppbus0: on ppc0 ppbus0: [ITHREAD] plip0: on ppbus0 plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec acd0: CDRW at ata0-master UDMA33 amr0: delete logical drives supported by controller amrd0: on amr0 amrd0: 419700MB (859545600 sectors) RAID 5 (optimal) amrd1: on amr0 amrd1: 139900MB (286515200 sectors) RAID 0 (optimal) ses0 at amr0 bus 0 target 6 lun 0 ses0: Fixed Processor SCSI-2 device ses0: 166.664MB/s transfers (41.666MHz, offset 31, 32bit) ses0: SAF-TE Compliant Device SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/amrd0s1a --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---