From owner-freebsd-net@FreeBSD.ORG Fri Oct 29 18:12:58 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AA0B106564A for ; Fri, 29 Oct 2010 18:12:58 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id E2B7F8FC1C for ; Fri, 29 Oct 2010 18:12:57 +0000 (UTC) Received: by gwaa18 with SMTP id a18so2302787gwa.13 for ; Fri, 29 Oct 2010 11:12:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=Kk0iZFbR/LVKQmo9FL7wSjRe2qP/16USaGrfB7ryPUc=; b=eJgF44brLifIdJSDWwK1KfuP2XjffJpcx7nFnSg142nYo/emQuAoKoK594hiReeORb MjcGud3cMKjYmgnH7nQbdrM/4uNa8Czse/y/3ArTkWKXqV2SXg7iFVjuZr919l0sqg4y 2jv4gDTbDZAEfOFLolDF79hHhKeiMakqhDzB8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=KSMshyodKqU96mr5bd1x5fbN9J2ZxSf5WbbE8RMbKEKNydxdUN6pYCpk+vDSVMWfho mIZaSfoYgNcMoyVMrXOYZwu8O1gZH5KPebWoxlMBTlCpXjrWeqZq4Fp5V13yfg+NxY7T eL4wLzfniSY9Xy18nL55ttkcTeEIrmOZf3cXw= Received: by 10.42.7.211 with SMTP id f19mr6621982icf.458.1288375976927; Fri, 29 Oct 2010 11:12:56 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 34sm3946344ibi.2.2010.10.29.11.12.52 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 29 Oct 2010 11:12:55 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Fri, 29 Oct 2010 11:11:18 -0700 From: Pyun YongHyeon Date: Fri, 29 Oct 2010 11:11:18 -0700 To: =?utf-8?B?0JrQvtC90YzQutC+0LIg0JXQstCz0LXQvdC40Lk=?= Message-ID: <20101029181118.GB19479@michelle.cdnetworks.com> References: <1519248747.20101028232111@yandex.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1519248747.20101028232111@yandex.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: Polling slows down bandwidth X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Oct 2010 18:12:58 -0000 On Thu, Oct 28, 2010 at 11:21:11PM +0300, ?????????????? ?????????????? wrote: > Hello, > w/0 polling: > > > serv1# ifconfig nfe0 > nfe0: flags=8943 metric 0 mtu 1500 > options=10b > ether 00:13:d4:ce:82:16 > inet 10.11.8.17 netmask 0xfffffc00 broadcast 10.11.11.255 > inet 10.11.15.15 netmask 0xffffff00 broadcast 10.11.15.255 > inet 10.11.8.15 netmask 0xfffffc00 broadcast 10.11.11.255 > media: Ethernet autoselect (1000baseTX ) > status: active > > serv2# ifconfig re0 > re0: flags=8843 metric 0 mtu 1500 > options=389b > ether 00:1c:c0:c8:5a:4e > inet 192.168.255.254 netmask 0xffffffff broadcast 192.168.255.254 > media: Ethernet autoselect (1000baseTX ) > status: active > > > serv1# systat -v > 2 users Load 0.38 0.49 0.38 Oct 28 23:10 > > Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER > Tot Share Tot Share Free in out in out > Act 324452 20196 490400 25112 938576 count > All 346596 21060 1074286k 28104 pages > Proc: Interrupts > r p d s w Csw Trp Sys Int Sof Flt cow 29953 total > 1 47 71k 18 5022 27k 2029 1 zfod atkbd0 1 > ozfod 7 ata0 irq14 > 62.4%Sys 18.7%Intr 0.4%User 0.0%Nice 18.4%Idle %ozfod 27944 nfe0 irq23 > | | | | | | | | | | | daefr 2001 cpu0: time > ===============================++++++++++ prcfr igb0 256 > 1 dtbuf 6 totfr 1 igb0 257 > Namei Name-cache Dir-cache 100000 desvn react igb0 258 > Calls hits % hits % 84106 numvn pdwak > 5 5 100 24824 frevn pdpgs > intrn > Disks ad0 373808 wire > KB/t 16.00 292088 act > tps 7 417568 inact > MB/s 0.10 200 cache > %busy 0 938376 free > > iperf result: > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.3 sec 450 MBytes 368 Mbits/sec > > > after enable POLLING: > serv1# ifconfig nfe0 polling > 2 users Load 0.32 0.39 0.35 Oct 28 23:13 > > Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER > Tot Share Tot Share Free in out in out > Act 324464 20196 490656 25112 938428 count > All 346624 21060 1074286k 28104 pages > Proc: Interrupts > r p d s w Csw Trp Sys Int Sof Flt cow 2006 total > 1 47 28k 19 2691 6 2036 2 zfod atkbd0 1 > ozfod 4 ata0 irq14 > 24.7%Sys 18.6%Intr 0.7%User 0.0%Nice 55.9%Idle %ozfod nfe0 irq23 > | | | | | | | | | | | daefr 2001 cpu0: time > ============++++++++++ prcfr igb0 256 > 10 dtbuf 2 totfr 1 igb0 257 > Namei Name-cache Dir-cache 100000 desvn react igb0 258 > Calls hits % hits % 84106 numvn pdwak > 20 20 100 24824 frevn pdpgs > intrn > Disks ad0 373944 wire > KB/t 16.00 292104 act > tps 4 417564 inact > MB/s 0.07 200 cache > %busy 0 938228 free > > I get bad results ((( > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.3 sec 180 MBytes 147 Mbits/sec > nfe(4) controllers are one of rare gigabit controllers that lacks efficient interrupt moderation mechanism. So it's normal to see high number of interrupts under load. hz controls how frequently checks controller's RX/TX queue so you may have to increase hz to get reasonable performance under high network load with polling(4). One of important performance factor for NIC is how many frames should be processed for given network load pattern. If you have to process more frames you have to increase hz in polling(4). I don't know what nfe(4) controller you have but it seems it's somewhat low-end model because MSI/MSIX is not used at all.