From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 11:29:23 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E60E1065692 for ; Wed, 7 Oct 2009 11:29:23 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from n21.bullet.mail.mud.yahoo.com (n21.bullet.mail.mud.yahoo.com [68.142.206.160]) by mx1.freebsd.org (Postfix) with SMTP id D1E528FC25 for ; Wed, 7 Oct 2009 11:29:22 +0000 (UTC) Received: from [68.142.194.243] by n21.bullet.mail.mud.yahoo.com with NNFMP; 07 Oct 2009 11:29:21 -0000 Received: from [76.13.13.26] by t1.bullet.mud.yahoo.com with NNFMP; 07 Oct 2009 11:29:21 -0000 Received: from [76.13.10.165] by t3.bullet.mail.ac4.yahoo.com with NNFMP; 07 Oct 2009 11:29:21 -0000 Received: from [127.0.0.1] by omp106.mail.ac4.yahoo.com with NNFMP; 07 Oct 2009 11:29:21 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 682497.15264.bm@omp106.mail.ac4.yahoo.com Received: (qmail 12248 invoked by uid 60001); 7 Oct 2009 11:29:21 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1254914961; bh=wVMRyV1DwKEjvuV95Et6/9edxRtGe9ZLqSVXV2BT0uI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=lnWrEd+DsDmFI/Rl1VusvKlOCuDK7YVU+eUm/IfyPM0Jbase4UowmrOGggDuS8DoY+nqEs30xuiSAC41Vydxx7a+wv00T2JemXZHeDkzpBqBxRyDN3hg8DTLfb2inGNPSaEYSDT5YL8s9ChKk+OHtPY7y4kPhY/WLn24fJITego= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=atpaeAe60i9LBYWqt782mhkoyS4ni9DPsQIe6q1qCm4iGYgGHV342zmnDU9m8QNqPX2z8d3hGhhDabHH8eBoHbdNFWs/3xsDRehRYzRJDbUx2/xIPekp8C5Z7l5AbE/CRPOO/MldPHhsvk/5b7xXDL8o00+wBI26Dm311ef/Npg=; Message-ID: <397638.11040.qm@web63902.mail.re1.yahoo.com> X-YMail-OSG: OFVnZWgVM1n4xe2MCOOH4KckpugU.meEAI3sNmiq2n2yt46q7YJxNxjEj4HzpKvzd.KPtBJNCnJQoSvLU34uaezpo6jm2_mamQuBDu4zX2Sa2nrDvUHHfLejqI0PqJJAv.TD_ZyrPiRihyiIdEqRQOLmbATBrFAsjzWCb7_6drgQo3yxDCY1yDSKAalh1MSlT.KExMaAxL82NaJ2.E8SMtogtgjuotpwNV50ZeUNAErfX2.pea67ahmYCGkOsnjL3nbLpM1hD64YqOQFAZCi.VzwB8v5wZKILGVXSCgHonGPD4QOUB7j56NT3YLXp5W5unEz3APdK3hx5hSGtxFuUg-- Received: from [98.203.21.152] by web63902.mail.re1.yahoo.com via HTTP; Wed, 07 Oct 2009 04:29:21 PDT X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.347.3 Date: Wed, 7 Oct 2009 04:29:21 -0700 (PDT) From: Barney Cordoba To: rihad In-Reply-To: <4ACC7A38.50809@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 11:29:23 -0000 =0A=0A--- On Wed, 10/7/09, rihad wrote:=0A=0A> From: rihad = =0A> Subject: Re: dummynet dropping too many packets=0A> To:= "Oleg Bulyzhin" =0A> Cc: freebsd-net@freebsd.org=0A> Dat= e: Wednesday, October 7, 2009, 7:23 AM=0A> rihad wrote:=0A> > Oleg Bulyzhin= wrote:=0A> >> On Wed, Oct 07, 2009 at 03:16:27PM +0500, rihad=0A> wrote:= =0A> >>> Oleg Bulyzhin wrote:=0A> >>>> On Wed, Oct 07, 2009 at 02:23:47PM += 0500,=0A> rihad wrote:=0A> >>>> =0A> >>>> Few questions:=0A> >>>> 1) why ar= e you not using fastforwarding?=0A> >>>> 2) search_steps/searches ratio is = not that=0A> good, are you using 'buckets'=0A> >>>>=A0 =A0 keyword in your = pipe=0A> configuration?=0A> >>>> 3) you have net.inet.ip.fw.one_pass =3D 0,= =0A> is it intended?=0A> >>>> =0A> >>> 1) and 3): the box does traffic acco= unting and=0A> shaping, so I need one_pass=3D0 to do both ngtee and pipes.= =0A> >> Still can not see any objection for not using=0A> fastforwarding, a= nd usually=0A> >> ipfw ruleset can be rearranged for using dummynet=0A> & n= etgraph with one_pass=3D1.=0A> > =0A> > You probably have some special sour= ces of=0A> documentation ;-) According to man ipfw, both=0A> "netgraph/ngte= e" and "pipe" decide the fate of the packet=0A> unless one_pass=3D0. Or do = you mean sprinkling smart skiptos=0A> here and there? ;-)=0A> > =0A> >> Cou= ld you show your 'ipfw show' output? (hide ip=0A> addresses if you wish but= =0A> >> keep counters please).=0A> >> =0A> > Here it is, in its whole glory= :=0A> > =0A> >=0A> 00100=A0=A0=A010434423=A0=A0=A01484891105=0A> allow ip f= rom any to any via lo0=0A> > 00200=A0 =A0 =A0 =A0 =A0 2=A0 =A0=0A> =A0 =A0 = =A0=A0=A014 deny ip from any to=0A> 127.0.0.0/8=0A> > 00300=A0 =A0 =A0 =A0 = =A0 1=A0 =A0=0A> =A0 =A0 =A0 =A0 4 deny ip from 127.0.0.0/8 to=0A> any=0A> = > 01000 3300039938 327603104711 allow ip from any to any=0A> in=0A> > 01010= =A0=A0=A026214900=A0 =A0 421138433=0A> allow ip from me to any out=0A> > 01= 020=A0 =A0 5453857=A0=0A> =A0=A0=A046806278 allow icmp from any to any out= =0A> > 01030 3268289053 327224694165 ngtee 1 ip from any to=0A> any out=0A>= >=0A> 01040=A0=A0=A018681181=A0=A0=A01089636054=0A> skipto 1100 ip from ta= ble(127) to any out recv bce0 xmit=0A> bce1=0A> > 01060=A0 777488848=A0 767= 43392754 pipe tablearg=0A> ip from any to table(0) out recv bce0 xmit bce1= =0A> > 01070=A0 776831109=A0 76682499457 allow ip from=0A> any to table(0) = out recv bce0 xmit bce1=0A> > 01100=A0=A0=A013102697=A0 =A0 808411842=0A> p= ipe tablearg ip from any to table(2) out=0A> > 65535=A0 662648946=A0 667114= 87830 allow ip from=0A> any to any=0A> > =0A> > table(127) is static in nat= ure and is under 100=0A> entries.=0A> > table(0) and table(2) have the same= IP clients'=0A> addresses but different pipe IDs.=0A> > =0A> >>> 2) Hm, I'= m not using "buckets", but rather=0A> net.inet.ip.dummynet.hash_size. It's = at default, 64. I've=0A> tried setting net.inet.ip.dummynet.hash_size=3D655= 36 in=0A> sysctl.conf but somehow it was still 64 after reboot, so I=0A> le= ft it at 64. Should I make it 128? 256? Does it matter=0A> that much? The l= oad is at approx. 70-120 consumers per pipe,=0A> so I thought 64 bucket siz= e was enough.=0A> >> It depends on traffic pattern, try to increase it=0A> = and watch=0A> >> search_steps/searches ratio (~1.001 is good=0A> enough)=0A= > >> =0A> After reconfiguring all pipes with hash_size=3D256 (4 times=0A> a= s much as before), the ratio has started decreasing slowly.=0A> Run by me e= very 5-100 seconds:=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.i= p.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | = bc -l=0A> 1.10639566354978963640=0A> [rihad@billing ~]$ echo "$(sysctl -n= =0A> net.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummyne= t.searches)" | bc -l=0A> 1.10638988711274017516=0A> [rihad@billing ~]$ echo= "$(sysctl -n=0A> net.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.in= et.ip.dummynet.searches)" | bc -l=0A> 1.10637649664889937145=0A> [rihad@bil= ling ~]$ echo "$(sysctl -n=0A> net.inet.ip.dummynet.search_steps)/$(sysctl = -n=0A> net.inet.ip.dummynet.searches)" | bc -l=0A> 1.10636898392044547569= =0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.ip.dummynet.search_s= teps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | bc -l=0A> 1.1063479= 8328730542254=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.ip.dumm= ynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | bc -l= =0A> 1.10608591323771604268=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> ne= t.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searc= hes)" | bc -l=0A> 1.10600110020578292697=0A> =0A> =0A> =0A> but the number = of drops is still there. Run every minute:=0A> Wed Oct=A0 7 11:00:44 UTC 20= 09=A0 =A0 34630 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:01:44 UTC 2009=A0 =A0 34630 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:02:44 UTC 2009=A0 =A0 34729 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:03:44 UTC 2009=A0 =A0 34= 729 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:04:= 44 UTC 2009=A0 =A0 34861 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:05:44 UTC 2009=A0 =A0 34932 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:06:44 UTC 2009=A0 =A0 35499 output= =0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:07:45 UTC 200= 9=A0 =A0 35780 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:08:45 UTC 2009=A0 =A0 35841 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:09:45 UTC 2009=A0 =A0 36348 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:10:45 UTC 2009=A0 =A0 36= 568 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:11:= 45 UTC 2009=A0 =A0 36673 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:12:45 UTC 2009=A0 =A0 36673 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:13:46 UTC 2009=A0 =A0 36673 output= =0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:14:46 UTC 200= 9=A0 =A0 36673 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:15:46 UTC 2009=A0 =A0 36673 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:16:46 UTC 2009=A0 =A0 36849 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:17:46 UTC 2009=A0 =A0 37= 234 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:18:= 46 UTC 2009=A0 =A0 37949 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:19:47 UTC 2009=A0 =A0 38043 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:20:47 UTC 2009=A0 =A0 38549 output= =0A> packets dropped due to no bufs, etc.=0A> =0A> 2200-2350 users online (= ipfw table load). I'll wait and see=0A> if the drop rate approaches 500-100= 0 per second as the=0A> number of online users comes close to 3-4K.=0A> =0A= > net.isr.direct=3D0=0A=0A=0AIts frightening to me that someone is managing= such a large network=0Awith dummynet. Talk about stealing your customer's = money.=0A=0ABC=0A=0A=0A