From owner-freebsd-net@FreeBSD.ORG Fri Apr 11 08:58:48 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A9BE4E12 for ; Fri, 11 Apr 2014 08:58:48 +0000 (UTC) Received: from quix.smartspb.net (quix.smartspb.net [217.119.16.133]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 682B01A6A for ; Fri, 11 Apr 2014 08:58:47 +0000 (UTC) Received: from dyr.smartspb.net ([217.119.16.26] helo=[127.0.0.1]) by quix.smartspb.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.61 (FreeBSD)) (envelope-from ) id 1WYXJ1-000M2M-D0 for freebsd-net@freebsd.org; Fri, 11 Apr 2014 12:59:27 +0400 Message-ID: <5347AEAA.9090801@smartspb.net> Date: Fri, 11 Apr 2014 12:58:18 +0400 From: Dennis Yusupoff User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: "freebsd-net@freebsd.org" Subject: dummynet/ipfw high load? X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Antivirus: avast! (VPS 140410-1, 10.04.2014), Outbound message X-Antivirus-Status: Clean X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2014 08:58:48 -0000 Good day, gurus! We have a servers on the FreeBSD. They do NAT, shaping and traffic accounting for our home (mainly) customers. NAT realized with pf nat, shaping with ipfw dummynet and traffic accounting with ng_netflow via ipfw ng_tee. The problem is performance on (relatively) high traffic. On Xeon E3-1270, whereas use Intel 10Gbit/sec 82599-based NIC(ix) or Intel I350 (82579) in lagg transit traffic in 800 Mbit/sec and 100 kpps [to customers] cause CPU load almost at 100% by interrupts from NIC or, in case of net.isr.dispatch=deferred and net.inet.ip.fastforwarding=0. Deleting ipfw pipe decrease load at ~30% per cpu. Deleting ipfw ng_tee (to ng_netflow) decrease load at 15% per cpu. Turning off ipfw (sysctl net.inet.ip.fw.enable=0) decrease load more, so what server can pass (nat'ed!) traffic on 1600 Mbit/sec and 200 kpps with only load ~40% per cpu. So my questions are: 1. Are there any way to decrease system load caused by dummynet/ipfw? 2. Why dummynet/ipfw increase *interrupts* load, not kernel or something like that? 3. Are there any way to profiling that kind of load? Existing DTrace and pmcstat examples almost useless or I just doesn't know how to do it properly. Huge size of debugging info (including dtrace and pmcstat samples), sysctl settings and so on, I opened appropriate topic at russian network operator's forum: http://forum.nag.ru/forum/index.php?showtopic=93674 In english it's available via google translate: http://translate.google.com/translate?hl=en&sl=auto&tl=en&u=http%3A%2F%2Fforum.nag.ru%2Fforum%2Findex.php%3Fshowtopic%3D93674 Feel free to ask me any question and do actions on the server! I would be VERY appreciate for any help and can take any measuring and debugging on the one server. Moreover, I'm ready to give root access to any of the appropriate person (as I already did it to Gleb Smirnoff when we were investigate pf state problem). -- Best regards, Dennis Yusupoff, network engineer of Smart-Telecom ISP Russia, Saint-Petersburg