Date: Wed, 06 Dec 2006 10:14:40 -0800 From: Rob Connon <rob@packetsafe.net> To: freebsd-questions@freebsd.org Subject: Disk I/O DEAD slow after upgrade from 5.5 -> 6.1 Message-ID: <45770890.5030406@packetsafe.net>
next in thread | raw e-mail | index | archive | help
Hi All,
We recently upgraded a mail server from FreeBSD 5.5 to 6.1 and also
moved the /home partition from a shared RAID 1 Volume with the
system disk to a 3 x 300GB RAID 5 volume.
Upon users returning the complaints starting to roll in about IMAP
(squirrelmail / MUA's) access being incredibly slow and mail clients
timing out.. all the daemons have been rebuilt.
Another colleague of mine posted and received feedback saying that you
will take a performance hit moving from a RAID 1 to RAID 5 volume which
is understood. Thinking it was maybe an issue with 6.1 we moved to 6.2
RC1 to see if that would help and it has not.. so last night we moved
all the mail back to a single 146GB disk to see if it was the RAID 5
volume and the problems persisted, same behavior HIGH iops for barely
any server activity resulting is sluggish.
Could this be related to FreeBSD 6 and the RAID card? below is dmesg
plus some other info. The Server is a HP DL380G4
Dual Xeon, 2GB mem, RAID 1 Volume (system Disk) RAID 5 Volume (homedir
mail delivery)
Thanks in advance!
Rob.
vmstat output
mail# vmstat 4
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us
sy id
3 9 0 1394696 78324 1532 0 0 0 1142 99 0 0 583 4832 1636
9 2 89
0 17 0 1342264 85972 1382 0 0 0 2559 0 42 0 968 16504 3108
13 5 82
0 20 0 1291744 127668 4158 0 0 1 4468 3289 5 0 820 9305 2785
10 5 85
0 18 0 1292056 114240 1149 0 0 0 961 0 3 0 873 7531 2482
6 2 92
0 13 0 1437268 70104 3101 0 0 0 2212 0 2 0 952 10561 2624
16 3 80
1 15 0 1429296 111668 1119 0 0 0 2092 3384 11 0 1007 10564
3093 6 3 91
0 12 0 1351700 92228 881 0 0 0 1930 0 15 0 902 11177 2644
13 3 84
1 11 0 1261692 124004 1046 0 0 0 3890 0 3 0 752 6718 2107
10 2 88
0 10 0 1261176 116220 766 0 0 0 710 0 26 0 916 8603 2489
4 2 93
0 14 0 1264976 103120 449 0 0 0 291 0 9 0 735 6775 2248
3 2 94
1 12 0 1265988 90356 642 0 0 0 425 0 2 0 874 7657 2679
14 3 83
0 10 0 1400440 111168 5349 13 0 1 3549 3279 4 0 950 11753
2657 22 5 73
1 19 0 1395484 76816 2443 13 0 0 3101 0 3 0 877 9711 2718
17 4 80
Systat -vmstat output
2 users Load 1.84 2.47 1.84 Dec 6 10:11
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 796644 11224 1261664 23804 180164 count
All 1965316 87552873999692 237848 pages
997 zfod Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt 1155 cow 8351 total
14265 2403 2377 4948 825 27 2261 199080 wire 1:
atkb
840156 act 6:
fdc0
1.9%Sys 0.4%Intr 8.7%User 0.0%Nice 89.0%Idl 831232 inact
14: ata
| | | | | | | | | | 100760 cache 138
29: bge
=>>>> 79404 free 208
30: cis
daefr 1
31: bge
Namei Name-cache Dir-cache 342 prcfr 2001
cpu0: time
Calls hits % hits % react 2001
cpu3: time
25829 24848 96 pdwak 2001
cpu1: time
pdpgs 2001
cpu2: time
Disks da0 da1 da2 pass0 pass1 pass2 intrn
KB/t 16.91 0.00 9.86 0.00 0.00 0.00 114464 buf
tps 7 0 207 0 0 0 182 dirtybuf
MB/s 0.11 0.00 1.99 0.00 0.00 0.00 100000 desiredvnodes
% busy 4 0 100 0 0 0 53711 numvnodes
24867 freevnodes
These tests below were ran at 1AM while barely any users were connected..
Diskinfo test run on single 146GB disk while generating I/O
------------------------------------
mail# diskinfo -t /dev/da1
/dev/da1
512 # sectorsize
146807930880 # mediasize in bytes (137G)
286734240 # mediasize in sectors
35139 # Cylinders according to firmware.
255 # Heads according to firmware.
32 # Sectors according to firmware.
Seek times:
Full stroke: 250 iter in 12.477037 sec = 49.908 msec
Half stroke: 250 iter in 7.018588 sec = 28.074 msec
Quarter stroke: 500 iter in 9.455907 sec = 18.912 msec
Short forward: 400 iter in 1.788364 sec = 4.471 msec
Short backward: 400 iter in 1.821630 sec = 4.554 msec
Seq outer: 2048 iter in 0.361632 sec = 0.177 msec
Seq inner: 2048 iter in 0.353460 sec = 0.173 msec
Transfer rates:
outside: 102400 kbytes in 13.899195 sec = 7367 kbytes/sec
middle: 102400 kbytes in 8.228883 sec = 12444 kbytes/sec
inside: 102400 kbytes in 6.058732 sec = 16901 kbytes/sec
Same Tests Ran on RAID 5 --------------------------------
mail# diskinfo -t /dev/da2
/dev/da2
512 # sectorsize
599990482944 # mediasize in bytes (559G)
1171856412 # mediasize in sectors
72944 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
Seek times:
Full stroke: 250 iter in 4.577300 sec = 18.309 msec
Half stroke: 250 iter in 3.393974 sec = 13.576 msec
Quarter stroke: 500 iter in 6.655164 sec = 13.310 msec
Short forward: 400 iter in 4.336646 sec = 10.842 msec
Short backward: 400 iter in 3.596825 sec = 8.992 msec
Seq outer: 2048 iter in 1.303522 sec = 0.636 msec
Seq inner: 2048 iter in 2.096910 sec = 1.024 msec
Transfer rates:
outside: 102400 kbytes in 14.755706 sec = 6940 kbytes/sec
middle: 102400 kbytes in 11.122171 sec = 9207 kbytes/sec
inside: 102400 kbytes in 41.512284 sec = 2467 kbytes/sec
Dmesg -----------------------------------------------
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RC1 #0: Tue Dec 5 15:20:31 PST 2006
root@mail:/usr/src/sys/i386/compile/SMP
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2786.88-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf25 Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4400<CNTX-ID,<b14>>
Logical CPUs per core: 2
real memory = 2147459072 (2047 MB)
avail memory = 2096340992 (1999 MB)
ACPI APIC Table: <COMPAQ 00000083>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic2 <Version 1.1> irqs 32-47 on motherboard
ioapic3 <Version 1.1> irqs 48-63 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <COMPAQ P29> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <display, VGA> at device 3.0 (no driver attached)
pci0: <base peripheral> at device 4.0 (no driver attached)
pci0: <base peripheral> at device 4.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks CSB5 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2000-0x200f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
ohci0: <OHCI (generic) USB controller> mem 0xeeef0000-0xeeef0fff irq 7
at device 15.2 on pci0
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
pcib1: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib1
ciss0: <Compaq Smart Array 5i> port 0x3000-0x30ff mem
0xf05c0000-0xf05fffff,0xf04f0000-0xf04f3fff irq 30 at device 3.0 on pci1
ciss0: [GIANT-LOCKED]
pcib2: <ACPI Host-PCI bridge> on acpi0
pci2: <ACPI PCI bus> on pcib2
bge0: <Broadcom BCM5703 A2, ASIC rev. 0x1002> mem 0xf06f0000-0xf06fffff
irq 29 at device 1.0 on pci2
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 10/100/1000baseTX PHY> on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
bge0: Ethernet address: 00:11:0a:9e:2f:49
bge1: <Broadcom BCM5703 A2, ASIC rev. 0x1002> mem 0xf06e0000-0xf06effff
irq 31 at device 2.0 on pci2
miibus1: <MII bus> on bge1
brgphy1: <BCM5703 10/100/1000baseTX PHY> on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
bge1: Ethernet address: 00:11:0a:9e:2f:48
pcib3: <ACPI Host-PCI bridge> on acpi0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pci6: <ACPI PCI bus> on pcib4
pci6: <base peripheral, PCI hot-plug controller> at device 30.0 (no
driver attached)
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xcd7ff,0xee000-0xeffff on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: CDROM <COMPAQ CD-ROM SN-124/N104> at ata0-master PIO4
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 1 VOLUME OK> Fixed Direct Access SCSI-0 device
da0: 135.168MB/s transfers
da0: 140006MB (286734240 512 byte sectors: 255H 32S/T 35139C)
da1 at ciss0 bus 0 target 1 lun 0
da1: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-0 device
da1: 135.168MB/s transfers
da1: 140006MB (286734240 512 byte sectors: 255H 32S/T 35139C)
da2 at ciss0 bus 0 target 2 lun 0
da2: <COMPAQ RAID 5 VOLUME OK> Fixed Direct Access SCSI-0 device
da2: 135.168MB/s transfers
da2: 572195MB (1171856412 512 byte sectors: 255H 63S/T 65535C)
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/da0s1a
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?45770890.5030406>