From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 11:27:44 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B522616A419 for ; Wed, 17 Oct 2007 11:27:44 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id 812AE13C455 for ; Wed, 17 Oct 2007 11:27:44 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.storspeed.com (209-163-168-124.static.twtelecom.net [209.163.168.124]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id l9HBRgZe062701; Wed, 17 Oct 2007 06:27:42 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <4715F1A5.50605@freebsd.org> Date: Wed, 17 Oct 2007 06:27:33 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Doug Clements References: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> In-Reply-To: <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: freebsd-hackers@freebsd.org Subject: Re: Interrupt/speed problems with 6.2 NFS server X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 11:27:44 -0000 Doug Clements wrote: > Hi, > I have an new NFS server that is processing roughly 15mbit of NFS traffic > that we recently upgraded from an older 4.10 box. It has a 3-ware raid card, > and is serving NFS out a single em nic to LAN clients. The machine works > great just serving NFS, but when I try to copy data from one raid volume to > another for backups, the machine's NFS performance goes way down, and > NFSops start taking multiple seconds to perform. The file copy goes > quite > quickly, as would be expected. The console of the machine also starts to lag > pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do > the backup. > > My first impression was that I was having interrupt issues, since during the > backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60% > CPU processing interrupts). So I recompiled the kernel with polling enabled > and enabled it on the NICs. The strange thing is that polling shows enabled > in ifconfig, but systat -vm still shows the same amount of interrupts. I get > the same performance with polling enabled. > > I'm looking for some guidance on why the machine bogs so much during what > seems to me to be something that should barely impact machine performance at > all, and also why polling didn't seem to lower the number of interrupts > processed. The old machine was 6 years old running an old intel raid5, and > it handled NFS and the concurrent file copies without a sweat. > > My 3ware is setup as follows: > a 2 disk mirror, for the system > a 4 disk raid10, for /mnt/data1 > a 4 disk raid10, for /mnt/data2 > > Copyright (c) 1992-2007 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007 > admin@madonna.linkline.com :/usr/obj/usr/src/sys/MADONNA > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class > CPU) > Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4 > Features=0xbfebfbff ,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > Features2=0x4e3bd,CX16,,,> > > AMD Features=0x20100800 > AMD Features2=0x1 > Cores per package: 2 > real memory = 4831838208 (4608 MB) > avail memory = 4125257728 (3934 MB) > ACPI APIC Table: > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-47 on motherboard > lapic0: Forcing LINT1 to edge trigger > kbd1 at kbdmux0 > ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) > acpi0: on motherboard > acpi0: Power Button (fixed) > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > cpu0: on acpi0 > acpi_throttle0: on cpu0 > acpi_button0: on acpi0 > acpi_button1: on acpi0 > pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: at device 2.0 on pci0 > pci1: on pcib1 > pcib2: irq 16 at device 0.0 on pci1 > pci2: on pcib2 > pcib3: irq 16 at device 0.0 on pci2 > pci3: on pcib3 > pcib4: irq 17 at device 1.0 on pci2 > pci4: on pcib4 > pcib5: irq 18 at device 2.0 on pci2 > pci5: on pcib5 > em0: port > 0x3020-0x303f mem 0xf8820000-0xf883ffff,0xf8400000-0xf87fffff irq 18 at > device 0.0 on pci5 > em0: Ethernet address: 00:15:17:21:bf:30 > em1: port > 0x3000-0x301f mem 0xf8800000-0xf881ffff,0xf8000000-0xf83fffff irq 19 at > device 0.1 on pci5 > em1: Ethernet address: 00:15:17:21:bf:31 > pcib6: at device 0.3 on pci1 > pci6: on pcib6 > 3ware device driver for 9000 series storage controllers, version: > 3.60.02.012 > twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem > 0xfa000000-0xfbffffff,0xf8900000-0xf8900fff irq 26 at device 2.0 on pci6 > twa0: [GIANT-LOCKED] > twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports, > Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002 > pcib7: at device 3.0 on pci0 > pci7: on pcib7 > pcib8: at device 4.0 on pci0 > pci8: on pcib8 > pcib9: at device 5.0 on pci0 > pci9: on pcib9 > pcib10: at device 6.0 on pci0 > pci10: on pcib10 > pcib11: at device 7.0 on pci0 > pci11: on pcib11 > pci0: at device 8.0 (no driver attached) > pcib12: irq 16 at device 28.0 on pci0 > pci12: on pcib12 > uhci0: port 0x4080-0x409f irq 23 at device > 29.0 on pci0 > uhci0: [GIANT-LOCKED] > usb0: on uhci0 > usb0: USB revision 1.0 > uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub0: 2 ports with 2 removable, self powered > uhci1: port 0x4060-0x407f irq 22 at device > 29.1 on pci0 > uhci1: [GIANT-LOCKED] > usb1: on uhci1 > usb1: USB revision 1.0 > uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub1: 2 ports with 2 removable, self powered > uhci2: port 0x4040-0x405f irq 23 at device > 29.2 on pci0 > uhci2: [GIANT-LOCKED] > usb2: on uhci2 > usb2: USB revision 1.0 > uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub2: 2 ports with 2 removable, self powered > uhci3: port 0x4020-0x403f irq 22 at device > 29.3 on pci0 > uhci3: [GIANT-LOCKED] > usb3: on uhci3 > usb3: USB revision 1.0 > uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub3: 2 ports with 2 removable, self powered > ehci0: mem 0xf8c00400-0xf8c007ff irq 23 > at device 29.7 on pci0 > ehci0: [GIANT-LOCKED] > usb4: EHCI version 1.0 > usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 > usb4: on ehci0 > usb4: USB revision 2.0 > uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 > uhub4: 8 ports with 8 removable, self powered > pcib13: at device 30.0 on pci0 > pci13: on pcib13 > pci13: at device 12.0 (no driver attached) > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci0: port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 20 at device 31.1 on > pci0 > ata0: on atapci0 > ata1: on atapci0 > atapci1: port > 0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem > 0xf8c00000-0xf8c003ff irq 20 at device 31.2 on pci0 > ata2: on atapci1 > ata3: on atapci1 > pci0: at device 31.3 (no driver attached) > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on > acpi0 > sio0: type 16550A > sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 > sio1: type 16550A > orm0: at iomem > 0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0 > ppc0: cannot reserve I/O port range > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Timecounter "TSC" frequency 2670647184 Hz quality 800 > Timecounters tick every 1.000 msec > acd0: CDRW at ata0-master UDMA33 > da0 at twa0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 100.000MB/s transfers > da0: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C) > da1 at twa0 bus 0 target 1 lun 0 > da1: Fixed Direct Access SCSI-5 device > da1: 100.000MB/s transfers > da1: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) > da2 at twa0 bus 0 target 2 lun 0 > da2: Fixed Direct Access SCSI-5 device > da2: 100.000MB/s transfers > da2: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) > Trying to mount root from ufs:/dev/da0s1a > em1: link state changed to UP > > > [root@madonna ~]# mount > /dev/da0s1a on / (ufs, local) > devfs on /dev (devfs, local) > /dev/da0s1e on /usr (ufs, local, soft-updates) > /dev/da0s1d on /var (ufs, local, soft-updates) > /dev/da1s1d on /mnt/data1 (ufs, NFS exported, local, noatime, soft-updates) > /dev/da2s1d on /mnt/data2 (ufs, NFS exported, local, noatime, soft-updates) > > [root@madonna ~]# df -hi > Filesystem Size Used Avail Capacity iused ifree %iused Mounted > on > /dev/da0s1a 4.8G 337M 4.1G 7% 3648 655806 1% / > devfs 1.0K 1.0K 0B 100% 0 0 100% /dev > /dev/da0s1e 203G 2.5G 184G 1% 235268 27320570 1% /usr > /dev/da0s1d 9.7G 3.3M 8.9G 0% 220 1318690 0% /var > /dev/da1s1d 451G 36G 379G 9% 1220061 59920929 2% > /mnt/data1 > /dev/da2s1d 451G 124G 290G 30% 4143375 56997615 7% > /mnt/data2 > > [root@madonna ~]# more /etc/exports > /mnt/data1 -maproot=0 -alldirs courtney bjork cher joan beth > kelli miho deborah > /mnt/data2 -maproot=0 -alldirs courtney bjork cher joan beth > kelli miho deborah > > [root@madonna ~]# ifconfig -a > em0: flags=8802 mtu 1500 > options=4b > ether 00:15:17:21:bf:30 > media: Ethernet autoselect > status: no carrier > em1: flags=8843 mtu 1500 > options=4b > inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.x > ether 00:15:17:21:bf:31 > media: Ethernet 100baseTX > status: active > lo0: flags=8049 mtu 16384 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 > inet6 ::1 prefixlen 128 > inet 127.0.0.1 netmask 0xff000000 > > Anyone have any clue about what might be going on? Can you also send the output of ps -auxl? Also - do you notice this performance drop when running something like one of the network performance tools? I'd like to isolate the disk activity from the network activity for a clean test.. Eric