From owner-freebsd-net@FreeBSD.ORG Fri Apr 11 16:05:55 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF45F1CE for ; Fri, 11 Apr 2014 16:05:55 +0000 (UTC) Received: from mail-pd0-x22f.google.com (mail-pd0-x22f.google.com [IPv6:2607:f8b0:400e:c02::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6A8C1C05 for ; Fri, 11 Apr 2014 16:05:55 +0000 (UTC) Received: by mail-pd0-f175.google.com with SMTP id x10so5461640pdj.20 for ; Fri, 11 Apr 2014 09:05:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=cApkMMC+DqWO/oQJpHnmqv1MpN8R9ZBRWDbXRoj1WOU=; b=Egxdox7jC7az+2mxP/IDQwNKRH2lt6RlPSShS0ProPQw9pLKCRmWsa6MVxaaKriROS GJdL6mbk6Wllx+vyMIy4DwzFk9miF/Wuc0exNolFjcohVkTWpTDXf1Sfeq7xFESAT+b5 QPOz+rt7GEBvVld66k8dekP4EBFsyAxHAKdNHpTN8tj1DsuONCfmorItqgv4vFamPIfY kLdTucwdF983KrfY7/F0RleOl13sHfRR2t3g/pXGhkE7hHuyAyoi6Y5RB6sxtCum+H4t YIxVJM7eSprJ4yE8267AZxqPXRLNG/Hauy3KHFlNABNFezhoU/remeXe4Jts9dWAbyWG VKDw== MIME-Version: 1.0 X-Received: by 10.66.140.104 with SMTP id rf8mr27962930pab.107.1397232355307; Fri, 11 Apr 2014 09:05:55 -0700 (PDT) Received: by 10.70.127.136 with HTTP; Fri, 11 Apr 2014 09:05:55 -0700 (PDT) In-Reply-To: <5347AEAA.9090801@smartspb.net> References: <5347AEAA.9090801@smartspb.net> Date: Fri, 11 Apr 2014 19:05:55 +0300 Message-ID: Subject: Re: dummynet/ipfw high load? From: Sami Halabi To: Dennis Yusupoff Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2014 16:05:55 -0000 Hi, I had similar problem on the past and it turned to be the ammount of rules in ipfe. Using reduced subset with tables actually reduced the load. Sami =E2=80=8F=D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=A9=D7= =99=D7=A9=D7=99, 11 =D7=91=D7=90=D7=A4=D7=A8=D7=99=D7=9C 2014, Dennis Yusup= off =D7=9B=D7=AA=D7=91: > Good day, gurus! > > We have a servers on the FreeBSD. They do NAT, shaping and traffic > accounting for our home (mainly) customers. > NAT realized with pf nat, shaping with ipfw dummynet and traffic > accounting with ng_netflow via ipfw ng_tee. > The problem is performance on (relatively) high traffic. > On Xeon E3-1270, whereas use Intel 10Gbit/sec 82599-based NIC(ix) or > Intel I350 (82579) in lagg transit traffic in 800 Mbit/sec and 100 kpps > [to customers] cause CPU load almost at 100% by interrupts from NIC or, > in case of net.isr.dispatch=3Ddeferred and net.inet.ip.fastforwarding=3D0= . > Deleting ipfw pipe decrease load at ~30% per cpu. > Deleting ipfw ng_tee (to ng_netflow) decrease load at 15% per cpu. > Turning off ipfw (sysctl net.inet.ip.fw.enable=3D0) decrease load more, s= o > what server can pass (nat'ed!) traffic on 1600 Mbit/sec and 200 kpps > with only load ~40% per cpu. > > So my questions are: > 1. Are there any way to decrease system load caused by dummynet/ipfw? > 2. Why dummynet/ipfw increase *interrupts* load, not kernel or > something like that? > 3. Are there any way to profiling that kind of load? Existing DTrace > and pmcstat examples almost useless or I just doesn't know how to do it > properly. > > Huge size of debugging info (including dtrace and pmcstat samples), > sysctl settings and so on, I opened appropriate topic at russian network > operator's forum: http://forum.nag.ru/forum/index.php?showtopic=3D93674 > In english it's available via google translate: > > http://translate.google.com/translate?hl=3Den&sl=3Dauto&tl=3Den&u=3Dhttp%= 3A%2F%2Fforum.nag.ru%2Fforum%2Findex.php%3Fshowtopic%3D93674 > > Feel free to ask me any question and do actions on the server! > > I would be VERY appreciate for any help and can take any measuring and > debugging on the one server. Moreover, I'm ready to give root access to > any of the appropriate person (as I already did it to Gleb Smirnoff when > we were investigate pf state problem). > > > -- > Best regards, > Dennis Yusupoff, > network engineer of > Smart-Telecom ISP > Russia, Saint-Petersburg > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > --=20 Sami Halabi Information Systems Engineer NMS Projects Expert FreeBSD SysAdmin Expert