From owner-freebsd-current@FreeBSD.ORG Thu Nov 18 21:31:22 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 996E716A4CF; Thu, 18 Nov 2004 21:31:22 +0000 (GMT) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6D81343D48; Thu, 18 Nov 2004 21:31:22 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) iAILVKcW032004; Thu, 18 Nov 2004 13:31:20 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.9p2/8.12.9/Submit) id iAILVJ1T032003; Thu, 18 Nov 2004 13:31:19 -0800 (PST) (envelope-from dillon) Date: Thu, 18 Nov 2004 13:31:19 -0800 (PST) From: Matthew Dillon Message-Id: <200411182131.iAILVJ1T032003@apollo.backplane.com> To: Wilko Bulte References: <200411172357.47735.Emanuel.Strobl@gmx.net> <20041118124902.GC75559@freebie.xs4all.nl> cc: Emanuel Strobl cc: freebsd-stable@freebsd.org cc: freebsd-current@freebsd.org cc: Robert Watson Subject: Re: serious networking (em) performance (ggate and NFS) problem X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2004 21:31:22 -0000 Polling should not produce any improvement over interrupts for EM0. The EM0 card will aggregate 8-14+ packets per interrupt, or more. which is only around 8000 interrupts/sec. I've got a ton of these cards installed. # mount_nfs -a 4 dhcp61:/home /mnt # dd if=/mnt/x of=/dev/null bs=32k # netstat -in 1 input (Total) output packets errs bytes packets errs bytes colls 66401 0 93668746 5534 0 962920 0 66426 0 94230092 5537 0 1007108 0 66424 0 93699848 5536 0 963268 0 66422 0 94222372 5536 0 1007290 0 66391 0 93654846 5534 0 962746 0 66375 0 94154432 5532 0 1006404 0 zfod Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt cow 8100 total 1 9 6211 7 75 8100 4 12 88864 wire 7873 mux irq10 10404 act ata0 irq14 19.2%Sys 0.0%Intr 0.0%User 0.0%Nice 80.8%Idl 864476 inact ata1 irq15 | | | | | | | | | | 58152 cache mux irq11 ========== 2992 free 227 clk irq0 Note that the interrupt rate is only 7873 interrupts per second while I am transfering 94 MBytes/sec over NFS (UDP) and receiving over 66000 packets per second (~8 packets per interrupt). If I use a TCP mount I get just about the same thing: # mount_nfs -T -a 4 dhcp61:/home /mnt # dd if=/mnt/x of=/dev/null bs=32k # netstat -in 1 input (Total) output packets errs bytes packets errs bytes colls 61752 0 93978800 8091 0 968618 0 61780 0 93530484 8098 0 904370 0 61710 0 93917880 8093 0 968128 0 61754 0 93491260 8095 0 903940 0 61756 0 93986320 8097 0 968336 0 Proc:r p d s w Csw Trp Sys Int Sof Flt cow 8145 total 5 8 22828 13 5490 8146 13 11 141556 wire 7917 mux irq10 7800 act ata0 irq14 26.4%Sys 0.0%Intr 0.0%User 0.0%Nice 73.6%Idl 244872 inact ata1 irq15 | | | | | | | | | | 8 cache mux irq11 ============= 630780 free 228 clk irq0 In this case around 8000 interrupts per second with 61700 packet per second incoming on the interface (around ~8 packets per interrupt). The extra interrupts are due to the additional outgoing TCP ack traffic. If I look at the systat -vm 1 output on the NFS server it also sees only around 8000 interrupts per second, which isn't saying much other then it's transmit path (61700 pps outoging) is not creating an undue interrupt burden relative to the receive path. -Matt