From owner-freebsd-net@FreeBSD.ORG Sat Jan 10 14:54:21 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0DA7A74B for ; Sat, 10 Jan 2015 14:54:21 +0000 (UTC) Received: from mx.aknet.kg (mx.aknet.kg [212.112.96.8]) by mx1.freebsd.org (Postfix) with ESMTP id 5E1F5D67 for ; Sat, 10 Jan 2015 14:54:19 +0000 (UTC) Received: from mx.aknet.kg (localhost.aknet.kg [127.0.0.1]) by mx.aknet.kg (Postfix) with ESMTP id D57F41CE01 for ; Sat, 10 Jan 2015 20:36:10 +0600 (KGT) Received: (from nobody@localhost) by mx.aknet.kg (8.13.8/8.13.8/Submit) id t0AEaAgN071333; Sat, 10 Jan 2015 20:36:10 +0600 (KGT) (envelope-from info@aknet.kg) X-Authentication-Warning: mx.aknet.kg: nobody set sender to info@aknet.kg using -f To: Subject: Netmap-Ipfw: dramatic perfomance degrade after certain load and ruleset insertions X-PHP-Originating-Script: 501:main.inc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 10 Jan 2015 20:36:10 +0600 From: info@aknet.kg Message-ID: X-Sender: info@aknet.kg User-Agent: Roundcube Webmail/0.7.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jan 2015 14:54:21 -0000 Hello, All (NetMap dev. Team) ! We use netmap-ipfw server for traffic pre-processing before the main filtering bridge (ordinary dummynet) and met the situation, that netmap-ipfw dramatically degrades perfomance (point B below) from stable work (point A below) after some actions with ruleset. A. Server with netmap-ipfw shows very stable work with following settings: 1. kipfw starts as ./kipfw netmap:ix0 netmap:ix1 2. current statement of sysctl variables: ... dev.netmap.ring_size: 128000 dev.netmap.ring_curr_size: 128000 dev.netmap.buf_curr_size: 4096 dev.netmap.buf_num: 896000 dev.netmap.buf_curr_num: 896000 dev.netmap.ix_rx_miss: 2020539039 dev.netmap.ix_rx_miss_bufs: 1478979631 .... in /boot/loader.conf : hw.ix.rxd=4096 hw.ix.txd=4096 3. traffic at one of interfaces (ix1): root@bridge-netmap:/usr/local/netmap-ipfw/ipfw # netstat -bdh -w1 -I ix1 input ix1 output packets errs idrops bytes packets errs bytes colls drops 628K 0 0 737M 517K 0 106M 0 0 636K 0 0 749M 522K 0 105M 0 0 633K 0 0 744M 527K 0 105M 0 0 635K 0 0 744M 523K 0 107M 0 0 4. system load: CPU 0: 58.7% user, 0.0% nice, 33.5% system, 3.9% interrupt, 3.9% idle CPU 1: 0.0% user, 0.0% nice, 0.0% system, 3.9% interrupt, 96.1% idle CPU 2: 0.0% user, 0.0% nice, 0.0% system, 3.5% interrupt, 96.5% idle CPU 3: 0.0% user, 0.0% nice, 0.4% system, 2.8% interrupt, 96.9% idle Mem: 22M Active, 243M Inact, 4125M Wired, 814M Buf, 3429M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 853 root 101 0 7087M 415M CPU0 0 17.5H 93.90% kipfw Sometimes kipfw takes all 100% of the first core (20-40 sec) but without any affect to traffic volume. B. Unstable work of the server begins when we add such rules to the ruleset: ./ipfw pipe 10 config mask dst-ip 0xffffffff bw 5120Kbit/s ./ipfw add pipe 10 ip from any to 192.168.0.0/16 by these rules we assign the same bandwidth to each IP's from 192.168.0.0/16 (approx 20K in a whole) After entering rules the server some minutes works good, and we see that in this time RES figure grows (not fast) from 415M to 500M, and than RES jumps from about 510M to 540M and network traffic downs at least twice, kipfw takes 100%, large packet drops (up to 30%) and delay grows. root@bridge-netmap:/usr/local/etc/rc.d # netstat -bdh -w1 -I ix1 input ix1 output packets errs idrops bytes packets errs bytes colls drops 648K 0 0 762M 534K 0 110M 0 0 643K 0 0 755M 529K 0 111M 0 0 648K 0 0 763M 534K 0 112M 0 0 648K 0 0 760M 537K 0 113M 0 0 215K 0 0 240M 211K 0 35M 0 0 !!!! (rules were entered into ruleset) 220K 0 0 235M 210K 0 34M 0 0 212K 0 0 236M 211K 0 35M 0 0 215K 0 0 240M 212K 0 35M 0 0 May be there is any limit (for example 512Mb) of mem usage ? If yes - can it be increased ? We can give any additional information if it be useful for investigating of this issue. Azamat IT Dep AkNet ISP