From owner-freebsd-stable@FreeBSD.ORG Thu Apr 10 22:32:06 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0122C1065670 for ; Thu, 10 Apr 2008 22:32:06 +0000 (UTC) (envelope-from karl@FS.denninger.net) Received: from FS.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) by mx1.freebsd.org (Postfix) with ESMTP id 536028FC0A for ; Thu, 10 Apr 2008 22:32:04 +0000 (UTC) (envelope-from karl@FS.denninger.net) Received: from fs.denninger.net (localhost [127.0.0.1]) by FS.denninger.net (8.14.2/8.13.1) with SMTP id m3AMVruJ014868 for ; Thu, 10 Apr 2008 17:31:53 -0500 (CDT) (envelope-from karl@FS.denninger.net) Received: from fs.denninger.net [127.0.0.1] by Spamblock-sys (LOCAL); Thu Apr 10 17:31:53 2008 Received: (from karl@localhost) by FS.denninger.net (8.14.2/8.13.1/Submit) id m3AMVrBd014863 for freebsd-stable@freebsd.org; Thu, 10 Apr 2008 17:31:53 -0500 (CDT) (envelope-from karl) Date: Thu, 10 Apr 2008 17:31:53 -0500 From: Karl Denninger To: freebsd-stable@freebsd.org Message-ID: <20080410223153.GA336@FS.denninger.net> Mail-Followup-To: freebsd-stable@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Organization: Karl's Sushi and Packet Smashers X-Die-Spammers: Spammers cheerfully broiled for supper and served with ketchup! Subject: Disk I/O Question 7.0-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Apr 2008 22:32:06 -0000 FreeBSD 7.0-STABLE #9: Sat Mar 29 08:31:56 CST 2008 karl@FS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP Running generally well. I have disks on the internal ICH7 adapters (on the motherboard), SATA, and also a TWE controller with two disks. When hitting the TWE controller hard I can hose the I/O performance on the primary (onboard) adapter quite severely to the point that it seriously impairs the system's overall performance. "systat -vm" shows the primary channels going to 100% busy but the transaction and I/O rate count is not all that high. The twe disk I am hammering, of course, is saturated (and blowing data at a high rate of speed - 70MBps+) It would seem to me that shouldn't be possible in that I can't possibly be saturating the processor DMA capability with that sort of load - but it is. Am I using a controller with a driver that is doing "not so good things"? The machine in question is a quad-core - here's the boot info Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-STABLE #9: Sat Mar 29 08:31:56 CST 2008 karl@FS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Quad CPU @ 2.40GHz (2409.70-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100000 AMD Features2=0x1 Cores per package: 4 real memory = 2146082816 (2046 MB) avail memory = 2089783296 (1992 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 est1: on cpu1 p4tcc1: on cpu1 cpu2: on acpi0 est2: on cpu2 p4tcc2: on cpu2 cpu3: on acpi0 est3: on cpu3 p4tcc3: on cpu3 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: mem 0x91000000-0x91ffffff,0x80000000-0x8fffffff,0x90000000-0x90ffffff irq 16 at device 0.0 on pci1 pcib2: at device 3.0 on pci0 pci2: on pcib2 pcib3: at device 0.0 on pci2 pci3: on pcib3 puc0: port 0x4060-0x407f,0x4040-0x405f mem 0x92d03000-0x92d03fff,0x92d02000-0x92d02fff irq 16 at device 0.0 on pci3 puc0: [FILTER] uart0: <16550 or compatible> on puc0 uart0: [FILTER] uart1: <16550 or compatible> on puc0 uart1: [FILTER] uart2: <16550 or compatible> on puc0 uart2: [FILTER] uart3: <16550 or compatible> on puc0 uart3: [FILTER] pci3: at device 0.1 (no driver attached) pcm0: mem 0x92e00000-0x92e03fff irq 22 at device 27.0 on pci0 pcm0: [ITHREAD] pcib4: at device 28.0 on pci0 pci4: on pcib4 pcib5: at device 28.4 on pci0 pci5: on pcib5 atapci0: port 0x3018-0x301f,0x3024-0x3027,0x3010-0x3017,0x3020-0x3023,0x3000-0x300f mem 0x92c00000-0x92c003ff irq 16 at device 0.0 on pci5 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] pcib6: at device 28.5 on pci0 pci6: on pcib6 em0: port 0x2000-0x201f mem 0x92b00000-0x92b1ffff irq 17 at device 0.0 on pci6 em0: Using MSI interrupt em0: Ethernet address: 00:19:d1:e3:b4:b8 em0: [FILTER] uhci0: port 0x5080-0x509f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x5060-0x507f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x5040-0x505f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x5020-0x503f irq 16 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: on uhci3 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0x92e04400-0x92e047ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 8 ports with 8 removable, self powered pcib7: at device 30.0 on pci0 pci7: on pcib7 fxp0: port 0x1000-0x101f mem 0x92a00000-0x92a00fff,0x92800000-0x928fffff irq 21 at device 0.0 on pci7 miibus0: on fxp0 inphy0: PHY 1 on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:a0:c9:a4:78:c3 fxp0: [ITHREAD] twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x1020-0x102f mem 0x92904800-0x9290480f,0x92000000-0x927fffff irq 18 at device 2.0 on pci7 twe0: [GIANT-LOCKED] twe0: [ITHREAD] twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048 fwohci0: mem 0x92904000-0x929047ff,0x92900000-0x92903fff irq 18 at device 4.0 on pci7 fwohci0: [FILTER] fwohci0: OHCI version 1.10 (ROM=0) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:90:27:00:01:c8:f1:cc fwohci0: Phy 1394a available S400, 3 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0x7c82c000 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:90:27:c8:f1:cc fwe0: Ethernet address: 02:90:27:c8:f1:cc fwip0: on firewire0 fwip0: Firewire address: 00:90:27:00:01:c8:f1:cc @ 0xfffe00000000, S400, maxrec 2048 sbp0: on firewire0 fwohci0: Initiate bus reset fwohci0: BUS reset fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode isab0: at device 31.0 on pci0 isa0: on isab0 atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x50b0-0x50bf irq 18 at device 31.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] ata1: on atapci1 ata1: [ITHREAD] atapci2: port 0x50c8-0x50cf,0x50e4-0x50e7,0x50c0-0x50c7,0x50e0-0x50e3,0x50a0-0x50af mem 0x92e04000-0x92e043ff irq 19 at device 31.2 on pci0 atapci2: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] pci0: at device 31.3 (no driver attached) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] pmtimer0 on isa0 orm0: at iomem 0xcf000-0xcffff pnpid ORM0000 on isa0 ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/8 bytes threshold ppbus0: on ppc0 ppbus0: [ITHREAD] plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ugen0: on uhub0 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me) firewire0: bus manager 0 (me) ipfw2 (+ipv6) initialized, divert enabled, nat loadable, rule-based forwarding disabled, default to deny, logging disabled acd0: CDROM at ata0-master UDMA33 ad8: 476940MB at ata4-master SATA150 ad10: 476940MB at ata5-master SATA150 pcm0: pcm0: twed0: on twe0 twed0: 953869MB (1953525168 sectors) twed1: on twe0 twed1: 953869MB (1953525168 sectors) GEOM_MIRROR: Force device b500 start due to timeout. GEOM_MIRROR: Device mirror/b500 launched (2/3). SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/mirror/b500a em0: link state changed to UP Ideas? If the answer is "buy a different smart SATA controller of brand X" that's fine so long as the name and model number follows :) I can accomodate both PCI and PCI Express (if I remove the TWE card, which is a PCI card) Thanks in advance! -- -- Karl Denninger (karl@denninger.net) Internet Consultant & Kids Rights Activist http://www.denninger.net My home on the net - links to everything I do! http://scubaforum.org Your UNCENSORED place to talk about DIVING! http://genesis3.blogspot.com Musings Of A Sentient Mind