From owner-freebsd-questions@FreeBSD.ORG Wed Dec 6 18:24:55 2006 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 383DC16A40F for ; Wed, 6 Dec 2006 18:24:55 +0000 (UTC) (envelope-from rob@packetsafe.net) Received: from mail.packetsafe.net (hercules.packetsafe.net [208.181.60.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4932F440BF for ; Wed, 6 Dec 2006 18:14:48 +0000 (GMT) (envelope-from rob@packetsafe.net) Received: from michigan.nerdcorps.ca ([66.119.167.1] helo=[10.10.10.35]) by mail.packetsafe.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.63 (FreeBSD)) (envelope-from ) id 1Gs1I1-0001pr-MH for freebsd-questions@freebsd.org; Wed, 06 Dec 2006 10:14:47 -0800 Message-ID: <45770890.5030406@packetsafe.net> Date: Wed, 06 Dec 2006 10:14:40 -0800 From: Rob Connon User-Agent: Thunderbird 1.5.0.8 (Windows/20061025) MIME-Version: 1.0 To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.4 (/) X-Spam-Report: Spam detection software, running on the system "hercules.packetsafe.net", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see The administrator of that system for details. Content preview: Hi All, We recently upgraded a mail server from FreeBSD 5.5 to 6.1 and also moved the /home partition from a shared RAID 1 Volume with the system disk to a 3 x 300GB RAID 5 volume. Upon users returning the complaints starting to roll in about IMAP (squirrelmail / MUA's) access being incredibly slow and mail clients timing out.. all the daemons have been rebuilt. Another colleague of mine posted and received feedback saying that you will take a performance hit moving from a RAID 1 to RAID 5 volume which is understood. Thinking it was maybe an issue with 6.1 we moved to 6.2 RC1 to see if that would help and it has not.. so last night we moved all the mail back to a single 146GB disk to see if it was the RAID 5 volume and the problems persisted, same behavior HIGH iops for barely any server activity resulting is sluggish. [...] Content analysis details: (-0.4 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.4 ALL_TRUSTED Passed through trusted hosts only via SMTP 0.1 TW_BD BODY: Odd Letter Triples with BD 0.1 TW_BF BODY: Odd Letter Triples with BF 0.1 TW_KB BODY: Odd Letter Triples with KB 0.1 TW_II BODY: Odd Letter Triples with II 0.1 TW_MV BODY: Odd Letter Triples with MV 0.1 TW_ZF BODY: Odd Letter Triples with ZF 0.1 TW_PG BODY: Odd Letter Triples with PG 0.1 TW_DV BODY: Odd Letter Triples with DV 0.1 TW_XB BODY: Odd Letter Triples with XB 0.1 TW_TK BODY: Odd Letter Triples with TK 0.1 TW_DP BODY: Odd Letter Triples with DP 0.1 TW_PD BODY: Odd Letter Triples with PD 0.1 TW_DR BODY: Odd Letter Triples with DR Subject: Disk I/O DEAD slow after upgrade from 5.5 -> 6.1 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2006 18:24:55 -0000 Hi All, We recently upgraded a mail server from FreeBSD 5.5 to 6.1 and also moved the /home partition from a shared RAID 1 Volume with the system disk to a 3 x 300GB RAID 5 volume. Upon users returning the complaints starting to roll in about IMAP (squirrelmail / MUA's) access being incredibly slow and mail clients timing out.. all the daemons have been rebuilt. Another colleague of mine posted and received feedback saying that you will take a performance hit moving from a RAID 1 to RAID 5 volume which is understood. Thinking it was maybe an issue with 6.1 we moved to 6.2 RC1 to see if that would help and it has not.. so last night we moved all the mail back to a single 146GB disk to see if it was the RAID 5 volume and the problems persisted, same behavior HIGH iops for barely any server activity resulting is sluggish. Could this be related to FreeBSD 6 and the RAID card? below is dmesg plus some other info. The Server is a HP DL380G4 Dual Xeon, 2GB mem, RAID 1 Volume (system Disk) RAID 5 Volume (homedir mail delivery) Thanks in advance! Rob. vmstat output mail# vmstat 4 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 3 9 0 1394696 78324 1532 0 0 0 1142 99 0 0 583 4832 1636 9 2 89 0 17 0 1342264 85972 1382 0 0 0 2559 0 42 0 968 16504 3108 13 5 82 0 20 0 1291744 127668 4158 0 0 1 4468 3289 5 0 820 9305 2785 10 5 85 0 18 0 1292056 114240 1149 0 0 0 961 0 3 0 873 7531 2482 6 2 92 0 13 0 1437268 70104 3101 0 0 0 2212 0 2 0 952 10561 2624 16 3 80 1 15 0 1429296 111668 1119 0 0 0 2092 3384 11 0 1007 10564 3093 6 3 91 0 12 0 1351700 92228 881 0 0 0 1930 0 15 0 902 11177 2644 13 3 84 1 11 0 1261692 124004 1046 0 0 0 3890 0 3 0 752 6718 2107 10 2 88 0 10 0 1261176 116220 766 0 0 0 710 0 26 0 916 8603 2489 4 2 93 0 14 0 1264976 103120 449 0 0 0 291 0 9 0 735 6775 2248 3 2 94 1 12 0 1265988 90356 642 0 0 0 425 0 2 0 874 7657 2679 14 3 83 0 10 0 1400440 111168 5349 13 0 1 3549 3279 4 0 950 11753 2657 22 5 73 1 19 0 1395484 76816 2443 13 0 0 3101 0 3 0 877 9711 2718 17 4 80 Systat -vmstat output 2 users Load 1.84 2.47 1.84 Dec 6 10:11 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 796644 11224 1261664 23804 180164 count All 1965316 87552873999692 237848 pages 997 zfod Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt 1155 cow 8351 total 14265 2403 2377 4948 825 27 2261 199080 wire 1: atkb 840156 act 6: fdc0 1.9%Sys 0.4%Intr 8.7%User 0.0%Nice 89.0%Idl 831232 inact 14: ata | | | | | | | | | | 100760 cache 138 29: bge =>>>> 79404 free 208 30: cis daefr 1 31: bge Namei Name-cache Dir-cache 342 prcfr 2001 cpu0: time Calls hits % hits % react 2001 cpu3: time 25829 24848 96 pdwak 2001 cpu1: time pdpgs 2001 cpu2: time Disks da0 da1 da2 pass0 pass1 pass2 intrn KB/t 16.91 0.00 9.86 0.00 0.00 0.00 114464 buf tps 7 0 207 0 0 0 182 dirtybuf MB/s 0.11 0.00 1.99 0.00 0.00 0.00 100000 desiredvnodes % busy 4 0 100 0 0 0 53711 numvnodes 24867 freevnodes These tests below were ran at 1AM while barely any users were connected.. Diskinfo test run on single 146GB disk while generating I/O ------------------------------------ mail# diskinfo -t /dev/da1 /dev/da1 512 # sectorsize 146807930880 # mediasize in bytes (137G) 286734240 # mediasize in sectors 35139 # Cylinders according to firmware. 255 # Heads according to firmware. 32 # Sectors according to firmware. Seek times: Full stroke: 250 iter in 12.477037 sec = 49.908 msec Half stroke: 250 iter in 7.018588 sec = 28.074 msec Quarter stroke: 500 iter in 9.455907 sec = 18.912 msec Short forward: 400 iter in 1.788364 sec = 4.471 msec Short backward: 400 iter in 1.821630 sec = 4.554 msec Seq outer: 2048 iter in 0.361632 sec = 0.177 msec Seq inner: 2048 iter in 0.353460 sec = 0.173 msec Transfer rates: outside: 102400 kbytes in 13.899195 sec = 7367 kbytes/sec middle: 102400 kbytes in 8.228883 sec = 12444 kbytes/sec inside: 102400 kbytes in 6.058732 sec = 16901 kbytes/sec Same Tests Ran on RAID 5 -------------------------------- mail# diskinfo -t /dev/da2 /dev/da2 512 # sectorsize 599990482944 # mediasize in bytes (559G) 1171856412 # mediasize in sectors 72944 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. Seek times: Full stroke: 250 iter in 4.577300 sec = 18.309 msec Half stroke: 250 iter in 3.393974 sec = 13.576 msec Quarter stroke: 500 iter in 6.655164 sec = 13.310 msec Short forward: 400 iter in 4.336646 sec = 10.842 msec Short backward: 400 iter in 3.596825 sec = 8.992 msec Seq outer: 2048 iter in 1.303522 sec = 0.636 msec Seq inner: 2048 iter in 2.096910 sec = 1.024 msec Transfer rates: outside: 102400 kbytes in 14.755706 sec = 6940 kbytes/sec middle: 102400 kbytes in 11.122171 sec = 9207 kbytes/sec inside: 102400 kbytes in 41.512284 sec = 2467 kbytes/sec Dmesg ----------------------------------------------- Copyright (c) 1992-2006 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RC1 #0: Tue Dec 5 15:20:31 PST 2006 root@mail:/usr/src/sys/i386/compile/SMP Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2786.88-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf25 Stepping = 5 Features=0xbfebfbff Features2=0x4400> Logical CPUs per core: 2 real memory = 2147459072 (2047 MB) avail memory = 2096340992 (1999 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-15 on motherboard ioapic1 irqs 16-31 on motherboard ioapic2 irqs 32-47 on motherboard ioapic3 irqs 48-63 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 3.0 (no driver attached) pci0: at device 4.0 (no driver attached) pci0: at device 4.2 (no driver attached) isab0: at device 15.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2000-0x200f at device 15.1 on pci0 ata0: on atapci0 ata1: on atapci0 ohci0: mem 0xeeef0000-0xeeef0fff irq 7 at device 15.2 on pci0 ohci0: [GIANT-LOCKED] usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: on ohci0 usb0: USB revision 1.0 uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 4 ports with 4 removable, self powered pcib1: on acpi0 pci1: on pcib1 ciss0: port 0x3000-0x30ff mem 0xf05c0000-0xf05fffff,0xf04f0000-0xf04f3fff irq 30 at device 3.0 on pci1 ciss0: [GIANT-LOCKED] pcib2: on acpi0 pci2: on pcib2 bge0: mem 0xf06f0000-0xf06fffff irq 29 at device 1.0 on pci2 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge0: Ethernet address: 00:11:0a:9e:2f:49 bge1: mem 0xf06e0000-0xf06effff irq 31 at device 2.0 on pci2 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge1: Ethernet address: 00:11:0a:9e:2f:48 pcib3: on acpi0 pci3: on pcib3 pcib4: on acpi0 pci6: on pcib4 pci6: at device 30.0 (no driver attached) acpi_tz0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 sio0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A fdc0: port 0x3f2-0x3f5 irq 6 drq 2 on acpi0 fdc0: [FAST] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xcd7ff,0xee000-0xeffff on isa0 ppc0: parallel port not found. sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec acd0: CDROM at ata0-master PIO4 da0 at ciss0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device da0: 135.168MB/s transfers da0: 140006MB (286734240 512 byte sectors: 255H 32S/T 35139C) da1 at ciss0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-0 device da1: 135.168MB/s transfers da1: 140006MB (286734240 512 byte sectors: 255H 32S/T 35139C) da2 at ciss0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-0 device da2: 135.168MB/s transfers da2: 572195MB (1171856412 512 byte sectors: 255H 63S/T 65535C) SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! Trying to mount root from ufs:/dev/da0s1a