From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 15 17:17:59 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8D2016A417 for ; Mon, 15 Oct 2007 17:17:59 +0000 (UTC) (envelope-from dclements@gmail.com) Received: from el-out-1112.google.com (el-out-1112.google.com [209.85.162.182]) by mx1.freebsd.org (Postfix) with ESMTP id 66BF613C447 for ; Mon, 15 Oct 2007 17:17:59 +0000 (UTC) (envelope-from dclements@gmail.com) Received: by el-out-1112.google.com with SMTP id r27so327339ele for ; Mon, 15 Oct 2007 10:17:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=eWiiF6fBSWheOuOB8Oi+Dgn1RSTuzdzVwoRR0wkasiE=; b=gzi7GbDSyslf5COTKLWDkGs22aEZg6IFJLuzaloE5GEOOvQysEwshhlBZVHwPDFFwYNvjjEac3BCRYTYQX9V4LhD3DCVjv7jV4Oo1aH0dZTZA7/nXYiYi06x+l2yyQKz2NMgm5vkf4JHWJJTxazf4d32gwhNmuQDEEcR9rrEaZA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=Ts+SdWAGf7np9kttiKm5zou/paZXt0GK5QaQbdqkZmOtWZRjxoAK4tcMsNyQdzzmB+Gnsn+e5E7BgLLayTlI1eav0QhwNEFr+kXpJBCfr7N+i5DamAAkz1FbkgRFEa3weDIdM3AoZ9XNb5m7etvev7t6kh90QGQyTl08591hNso= Received: by 10.142.231.7 with SMTP id d7mr1607023wfh.1192468677719; Mon, 15 Oct 2007 10:17:57 -0700 (PDT) Received: by 10.142.81.19 with HTTP; Mon, 15 Oct 2007 10:17:57 -0700 (PDT) Message-ID: <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> Date: Mon, 15 Oct 2007 10:17:57 -0700 From: "Doug Clements" To: freebsd-hackers@freebsd.org In-Reply-To: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> MIME-Version: 1.0 References: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Interrupt/speed problems with 6.2 NFS server X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 17:18:00 -0000 Hi, I have an new NFS server that is processing roughly 15mbit of NFS traffic that we recently upgraded from an older 4.10 box. It has a 3-ware raid card, and is serving NFS out a single em nic to LAN clients. The machine works great just serving NFS, but when I try to copy data from one raid volume to another for backups, the machine's NFS performance goes way down, and NFSops start taking multiple seconds to perform. The file copy goes quite quickly, as would be expected. The console of the machine also starts to lag pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do the backup. My first impression was that I was having interrupt issues, since during the backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60% CPU processing interrupts). So I recompiled the kernel with polling enabled and enabled it on the NICs. The strange thing is that polling shows enabled in ifconfig, but systat -vm still shows the same amount of interrupts. I get the same performance with polling enabled. I'm looking for some guidance on why the machine bogs so much during what seems to me to be something that should barely impact machine performance at all, and also why polling didn't seem to lower the number of interrupts processed. The old machine was 6 years old running an old intel raid5, and it handled NFS and the concurrent file copies without a sweat. My 3ware is setup as follows: a 2 disk mirror, for the system a 4 disk raid10, for /mnt/data1 a 4 disk raid10, for /mnt/data2 Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007 admin@madonna.linkline.com :/usr/obj/usr/src/sys/MADONNA Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4 Features=0xbfebfbff Features2=0x4e3bd,CX16,,,> AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 2 real memory = 4831838208 (4608 MB) avail memory = 4125257728 (3934 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 17 at device 1.0 on pci2 pci4: on pcib4 pcib5: irq 18 at device 2.0 on pci2 pci5: on pcib5 em0: port 0x3020-0x303f mem 0xf8820000-0xf883ffff,0xf8400000-0xf87fffff irq 18 at device 0.0 on pci5 em0: Ethernet address: 00:15:17:21:bf:30 em1: port 0x3000-0x301f mem 0xf8800000-0xf881ffff,0xf8000000-0xf83fffff irq 19 at device 0.1 on pci5 em1: Ethernet address: 00:15:17:21:bf:31 pcib6: at device 0.3 on pci1 pci6: on pcib6 3ware device driver for 9000 series storage controllers, version: 3.60.02.012 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem 0xfa000000-0xfbffffff,0xf8900000-0xf8900fff irq 26 at device 2.0 on pci6 twa0: [GIANT-LOCKED] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports, Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002 pcib7: at device 3.0 on pci0 pci7: on pcib7 pcib8: at device 4.0 on pci0 pci8: on pcib8 pcib9: at device 5.0 on pci0 pci9: on pcib9 pcib10: at device 6.0 on pci0 pci10: on pcib10 pcib11: at device 7.0 on pci0 pci11: on pcib11 pci0: at device 8.0 (no driver attached) pcib12: irq 16 at device 28.0 on pci0 pci12: on pcib12 uhci0: port 0x4080-0x409f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x4060-0x407f irq 22 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x4040-0x405f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x4020-0x403f irq 22 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xf8c00400-0xf8c007ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub4: 8 ports with 8 removable, self powered pcib13: at device 30.0 on pci0 pci13: on pcib13 pci13: at device 12.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 20 at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem 0xf8c00000-0xf8c003ff irq 20 at device 31.2 on pci0 ata2: on atapci1 ata3: on atapci1 pci0: at device 31.3 (no driver attached) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A orm0: at iomem 0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 2670647184 Hz quality 800 Timecounters tick every 1.000 msec acd0: CDRW at ata0-master UDMA33 da0 at twa0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C) da1 at twa0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 100.000MB/s transfers da1: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) da2 at twa0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) Trying to mount root from ufs:/dev/da0s1a em1: link state changed to UP [root@madonna ~]# mount /dev/da0s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/da0s1e on /usr (ufs, local, soft-updates) /dev/da0s1d on /var (ufs, local, soft-updates) /dev/da1s1d on /mnt/data1 (ufs, NFS exported, local, noatime, soft-updates) /dev/da2s1d on /mnt/data2 (ufs, NFS exported, local, noatime, soft-updates) [root@madonna ~]# df -hi Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/da0s1a 4.8G 337M 4.1G 7% 3648 655806 1% / devfs 1.0K 1.0K 0B 100% 0 0 100% /dev /dev/da0s1e 203G 2.5G 184G 1% 235268 27320570 1% /usr /dev/da0s1d 9.7G 3.3M 8.9G 0% 220 1318690 0% /var /dev/da1s1d 451G 36G 379G 9% 1220061 59920929 2% /mnt/data1 /dev/da2s1d 451G 124G 290G 30% 4143375 56997615 7% /mnt/data2 [root@madonna ~]# more /etc/exports /mnt/data1 -maproot=0 -alldirs courtney bjork cher joan beth kelli miho deborah /mnt/data2 -maproot=0 -alldirs courtney bjork cher joan beth kelli miho deborah [root@madonna ~]# ifconfig -a em0: flags=8802 mtu 1500 options=4b ether 00:15:17:21:bf:30 media: Ethernet autoselect status: no carrier em1: flags=8843 mtu 1500 options=4b inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.x ether 00:15:17:21:bf:31 media: Ethernet 100baseTX status: active lo0: flags=8049 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 Anyone have any clue about what might be going on? --Doug