Date: Thu, 18 Nov 2004 13:31:19 -0800 (PST) From: Matthew Dillon <dillon@apollo.backplane.com> To: Wilko Bulte <wb@freebie.xs4all.nl> Cc: Robert Watson <rwatson@freebsd.org> Subject: Re: serious networking (em) performance (ggate and NFS) problem Message-ID: <200411182131.iAILVJ1T032003@apollo.backplane.com> References: <200411172357.47735.Emanuel.Strobl@gmx.net> <20041118124902.GC75559@freebie.xs4all.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
Polling should not produce any improvement over interrupts for EM0.
The EM0 card will aggregate 8-14+ packets per interrupt, or more.
which is only around 8000 interrupts/sec. I've got a ton of these
cards installed.
# mount_nfs -a 4 dhcp61:/home /mnt
# dd if=/mnt/x of=/dev/null bs=32k
# netstat -in 1
input (Total) output
packets errs bytes packets errs bytes colls
66401 0 93668746 5534 0 962920 0
66426 0 94230092 5537 0 1007108 0
66424 0 93699848 5536 0 963268 0
66422 0 94222372 5536 0 1007290 0
66391 0 93654846 5534 0 962746 0
66375 0 94154432 5532 0 1006404 0
zfod Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt cow 8100 total
1 9 6211 7 75 8100 4 12 88864 wire 7873 mux irq10
10404 act ata0 irq14
19.2%Sys 0.0%Intr 0.0%User 0.0%Nice 80.8%Idl 864476 inact ata1 irq15
| | | | | | | | | | 58152 cache mux irq11
========== 2992 free 227 clk irq0
Note that the interrupt rate is only 7873 interrupts per second
while I am transfering 94 MBytes/sec over NFS (UDP) and receiving
over 66000 packets per second (~8 packets per interrupt).
If I use a TCP mount I get just about the same thing:
# mount_nfs -T -a 4 dhcp61:/home /mnt
# dd if=/mnt/x of=/dev/null bs=32k
# netstat -in 1
input (Total) output
packets errs bytes packets errs bytes colls
61752 0 93978800 8091 0 968618 0
61780 0 93530484 8098 0 904370 0
61710 0 93917880 8093 0 968128 0
61754 0 93491260 8095 0 903940 0
61756 0 93986320 8097 0 968336 0
Proc:r p d s w Csw Trp Sys Int Sof Flt cow 8145 total
5 8 22828 13 5490 8146 13 11 141556 wire 7917 mux irq10
7800 act ata0 irq14
26.4%Sys 0.0%Intr 0.0%User 0.0%Nice 73.6%Idl 244872 inact ata1 irq15
| | | | | | | | | | 8 cache mux irq11
============= 630780 free 228 clk irq0
In this case around 8000 interrupts per second with 61700 packet per
second incoming on the interface (around ~8 packets per interrupt).
The extra interrupts are due to the additional outgoing TCP ack traffic.
If I look at the systat -vm 1 output on the NFS server it also sees
only around 8000 interrupts per second, which isn't saying much other
then it's transmit path (61700 pps outoging) is not creating an undue
interrupt burden relative to the receive path.
-Matt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200411182131.iAILVJ1T032003>