Date: Mon, 15 Oct 2007 10:17:57 -0700 From: "Doug Clements" <dclements@gmail.com> To: freebsd-hackers@freebsd.org Subject: Interrupt/speed problems with 6.2 NFS server Message-ID: <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> In-Reply-To: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> References: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
I have an new NFS server that is processing roughly 15mbit of NFS traffic
that we recently upgraded from an older 4.10 box. It has a 3-ware raid card,
and is serving NFS out a single em nic to LAN clients. The machine works
great just serving NFS, but when I try to copy data from one raid volume to
another for backups, the machine's NFS performance goes way down, and
NFSops start taking multiple seconds to perform. The file copy goes
quite
quickly, as would be expected. The console of the machine also starts to lag
pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do
the backup.
My first impression was that I was having interrupt issues, since during the
backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60%
CPU processing interrupts). So I recompiled the kernel with polling enabled
and enabled it on the NICs. The strange thing is that polling shows enabled
in ifconfig, but systat -vm still shows the same amount of interrupts. I get
the same performance with polling enabled.
I'm looking for some guidance on why the machine bogs so much during what
seems to me to be something that should barely impact machine performance at
all, and also why polling didn't seem to lower the number of interrupts
processed. The old machine was 6 years old running an old intel raid5, and
it handled NFS and the concurrent file copies without a sweat.
My 3ware is setup as follows:
a 2 disk mirror, for the system
a 4 disk raid10, for /mnt/data1
a 4 disk raid10, for /mnt/data2
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007
admin@madonna.linkline.com :/usr/obj/usr/src/sys/MADONNA
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class
CPU)
Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4
Features=0xbfebfbff<FPU,VME,DE
,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>,<b18>>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
real memory = 4831838208 (4608 MB)
avail memory = 4125257728 (3934 MB)
ACPI APIC Table: <INTEL S5000PSL>
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
lapic0: Forcing LINT1 to edge trigger
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <INTEL S5000PSL> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xca2,0xca3,0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 1.0 on pci2
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x3020-0x303f mem 0xf8820000-0xf883ffff,0xf8400000-0xf87fffff irq 18 at
device 0.0 on pci5
em0: Ethernet address: 00:15:17:21:bf:30
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x3000-0x301f mem 0xf8800000-0xf881ffff,0xf8000000-0xf83fffff irq 19 at
device 0.1 on pci5
em1: Ethernet address: 00:15:17:21:bf:31
pcib6: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci6: <ACPI PCI bus> on pcib6
3ware device driver for 9000 series storage controllers, version:
3.60.02.012
twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem
0xfa000000-0xfbffffff,0xf8900000-0xf8900fff irq 26 at device 2.0 on pci6
twa0: [GIANT-LOCKED]
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports,
Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002
pcib7: <PCI-PCI bridge> at device 3.0 on pci0
pci7: <PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> at device 7.0 on pci0
pci11: <PCI bus> on pcib11
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib12: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci12: <ACPI PCI bus> on pcib12
uhci0: <UHCI (generic) USB controller> port 0x4080-0x409f irq 23 at device
29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x4060-0x407f irq 22 at device
29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x4040-0x405f irq 23 at device
29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x4020-0x403f irq 22 at device
29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xf8c00400-0xf8c007ff irq 23
at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <EHCI (generic) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci13: <ACPI PCI bus> on pcib13
pci13: <display, VGA> at device 12.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 20 at device 31.1 on
pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
atapci1: <Intel 63XXESB2 SATA300 controller> port
0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem
0xf8c00000-0xf8c003ff irq 20 at device 31.2 on pci0
ata2: <ATA channel 0> on atapci1
ata3: <ATA channel 1> on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2670647184 Hz quality 800
Timecounters tick every 1.000 msec
acd0: CDRW <HL-DT-ST RW/DVD GCC-H21N/1.01> at ata0-master UDMA33
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9550SX-12 DISK 3.08> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C)
da1 at twa0 bus 0 target 1 lun 0
da1: <AMCC 9550SX-12 DISK 3.08> Fixed Direct Access SCSI-5 device
da1: 100.000MB/s transfers
da1: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C)
da2 at twa0 bus 0 target 2 lun 0
da2: <AMCC 9550SX-12 DISK 3.08> Fixed Direct Access SCSI-5 device
da2: 100.000MB/s transfers
da2: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C)
Trying to mount root from ufs:/dev/da0s1a
em1: link state changed to UP
[root@madonna ~]# mount
/dev/da0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/da0s1e on /usr (ufs, local, soft-updates)
/dev/da0s1d on /var (ufs, local, soft-updates)
/dev/da1s1d on /mnt/data1 (ufs, NFS exported, local, noatime, soft-updates)
/dev/da2s1d on /mnt/data2 (ufs, NFS exported, local, noatime, soft-updates)
[root@madonna ~]# df -hi
Filesystem Size Used Avail Capacity iused ifree %iused Mounted
on
/dev/da0s1a 4.8G 337M 4.1G 7% 3648 655806 1% /
devfs 1.0K 1.0K 0B 100% 0 0 100% /dev
/dev/da0s1e 203G 2.5G 184G 1% 235268 27320570 1% /usr
/dev/da0s1d 9.7G 3.3M 8.9G 0% 220 1318690 0% /var
/dev/da1s1d 451G 36G 379G 9% 1220061 59920929 2%
/mnt/data1
/dev/da2s1d 451G 124G 290G 30% 4143375 56997615 7%
/mnt/data2
[root@madonna ~]# more /etc/exports
/mnt/data1 -maproot=0 -alldirs courtney bjork cher joan beth
kelli miho deborah
/mnt/data2 -maproot=0 -alldirs courtney bjork cher joan beth
kelli miho deborah
[root@madonna ~]# ifconfig -a
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
options=4b<RXCSUM,TXCSUM,VLAN_MTU,POLLING>
ether 00:15:17:21:bf:30
media: Ethernet autoselect
status: no carrier
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=4b<RXCSUM,TXCSUM,VLAN_MTU,POLLING>
inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.x
ether 00:15:17:21:bf:31
media: Ethernet 100baseTX <full-duplex>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
Anyone have any clue about what might be going on?
--Doug
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54da514e0710151017t5e12e404uf1258fceeaa3f149>