Date: Wed, 7 Oct 2009 04:29:21 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: rihad <rihad@mail.ru> Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets Message-ID: <397638.11040.qm@web63902.mail.re1.yahoo.com> In-Reply-To: <4ACC7A38.50809@mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
=0A=0A--- On Wed, 10/7/09, rihad <rihad@mail.ru> wrote:=0A=0A> From: rihad = <rihad@mail.ru>=0A> Subject: Re: dummynet dropping too many packets=0A> To:= "Oleg Bulyzhin" <oleg@FreeBSD.org>=0A> Cc: freebsd-net@freebsd.org=0A> Dat= e: Wednesday, October 7, 2009, 7:23 AM=0A> rihad wrote:=0A> > Oleg Bulyzhin= wrote:=0A> >> On Wed, Oct 07, 2009 at 03:16:27PM +0500, rihad=0A> wrote:= =0A> >>> Oleg Bulyzhin wrote:=0A> >>>> On Wed, Oct 07, 2009 at 02:23:47PM += 0500,=0A> rihad wrote:=0A> >>>> =0A> >>>> Few questions:=0A> >>>> 1) why ar= e you not using fastforwarding?=0A> >>>> 2) search_steps/searches ratio is = not that=0A> good, are you using 'buckets'=0A> >>>>=A0 =A0 keyword in your = pipe=0A> configuration?=0A> >>>> 3) you have net.inet.ip.fw.one_pass =3D 0,= =0A> is it intended?=0A> >>>> =0A> >>> 1) and 3): the box does traffic acco= unting and=0A> shaping, so I need one_pass=3D0 to do both ngtee and pipes.= =0A> >> Still can not see any objection for not using=0A> fastforwarding, a= nd usually=0A> >> ipfw ruleset can be rearranged for using dummynet=0A> & n= etgraph with one_pass=3D1.=0A> > =0A> > You probably have some special sour= ces of=0A> documentation ;-) According to man ipfw, both=0A> "netgraph/ngte= e" and "pipe" decide the fate of the packet=0A> unless one_pass=3D0. Or do = you mean sprinkling smart skiptos=0A> here and there? ;-)=0A> > =0A> >> Cou= ld you show your 'ipfw show' output? (hide ip=0A> addresses if you wish but= =0A> >> keep counters please).=0A> >> =0A> > Here it is, in its whole glory= :=0A> > =0A> >=0A> 00100=A0=A0=A010434423=A0=A0=A01484891105=0A> allow ip f= rom any to any via lo0=0A> > 00200=A0 =A0 =A0 =A0 =A0 2=A0 =A0=0A> =A0 =A0 = =A0=A0=A014 deny ip from any to=0A> 127.0.0.0/8=0A> > 00300=A0 =A0 =A0 =A0 = =A0 1=A0 =A0=0A> =A0 =A0 =A0 =A0 4 deny ip from 127.0.0.0/8 to=0A> any=0A> = > 01000 3300039938 327603104711 allow ip from any to any=0A> in=0A> > 01010= =A0=A0=A026214900=A0 =A0 421138433=0A> allow ip from me to any out=0A> > 01= 020=A0 =A0 5453857=A0=0A> =A0=A0=A046806278 allow icmp from any to any out= =0A> > 01030 3268289053 327224694165 ngtee 1 ip from any to=0A> any out=0A>= >=0A> 01040=A0=A0=A018681181=A0=A0=A01089636054=0A> skipto 1100 ip from ta= ble(127) to any out recv bce0 xmit=0A> bce1=0A> > 01060=A0 777488848=A0 767= 43392754 pipe tablearg=0A> ip from any to table(0) out recv bce0 xmit bce1= =0A> > 01070=A0 776831109=A0 76682499457 allow ip from=0A> any to table(0) = out recv bce0 xmit bce1=0A> > 01100=A0=A0=A013102697=A0 =A0 808411842=0A> p= ipe tablearg ip from any to table(2) out=0A> > 65535=A0 662648946=A0 667114= 87830 allow ip from=0A> any to any=0A> > =0A> > table(127) is static in nat= ure and is under 100=0A> entries.=0A> > table(0) and table(2) have the same= IP clients'=0A> addresses but different pipe IDs.=0A> > =0A> >>> 2) Hm, I'= m not using "buckets", but rather=0A> net.inet.ip.dummynet.hash_size. It's = at default, 64. I've=0A> tried setting net.inet.ip.dummynet.hash_size=3D655= 36 in=0A> sysctl.conf but somehow it was still 64 after reboot, so I=0A> le= ft it at 64. Should I make it 128? 256? Does it matter=0A> that much? The l= oad is at approx. 70-120 consumers per pipe,=0A> so I thought 64 bucket siz= e was enough.=0A> >> It depends on traffic pattern, try to increase it=0A> = and watch=0A> >> search_steps/searches ratio (~1.001 is good=0A> enough)=0A= > >> =0A> After reconfiguring all pipes with hash_size=3D256 (4 times=0A> a= s much as before), the ratio has started decreasing slowly.=0A> Run by me e= very 5-100 seconds:=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.i= p.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | = bc -l=0A> 1.10639566354978963640=0A> [rihad@billing ~]$ echo "$(sysctl -n= =0A> net.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummyne= t.searches)" | bc -l=0A> 1.10638988711274017516=0A> [rihad@billing ~]$ echo= "$(sysctl -n=0A> net.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.in= et.ip.dummynet.searches)" | bc -l=0A> 1.10637649664889937145=0A> [rihad@bil= ling ~]$ echo "$(sysctl -n=0A> net.inet.ip.dummynet.search_steps)/$(sysctl = -n=0A> net.inet.ip.dummynet.searches)" | bc -l=0A> 1.10636898392044547569= =0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.ip.dummynet.search_s= teps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | bc -l=0A> 1.1063479= 8328730542254=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.ip.dumm= ynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | bc -l= =0A> 1.10608591323771604268=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> ne= t.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searc= hes)" | bc -l=0A> 1.10600110020578292697=0A> =0A> =0A> =0A> but the number = of drops is still there. Run every minute:=0A> Wed Oct=A0 7 11:00:44 UTC 20= 09=A0 =A0 34630 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:01:44 UTC 2009=A0 =A0 34630 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:02:44 UTC 2009=A0 =A0 34729 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:03:44 UTC 2009=A0 =A0 34= 729 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:04:= 44 UTC 2009=A0 =A0 34861 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:05:44 UTC 2009=A0 =A0 34932 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:06:44 UTC 2009=A0 =A0 35499 output= =0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:07:45 UTC 200= 9=A0 =A0 35780 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:08:45 UTC 2009=A0 =A0 35841 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:09:45 UTC 2009=A0 =A0 36348 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:10:45 UTC 2009=A0 =A0 36= 568 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:11:= 45 UTC 2009=A0 =A0 36673 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:12:45 UTC 2009=A0 =A0 36673 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:13:46 UTC 2009=A0 =A0 36673 output= =0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:14:46 UTC 200= 9=A0 =A0 36673 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:15:46 UTC 2009=A0 =A0 36673 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:16:46 UTC 2009=A0 =A0 36849 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:17:46 UTC 2009=A0 =A0 37= 234 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:18:= 46 UTC 2009=A0 =A0 37949 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:19:47 UTC 2009=A0 =A0 38043 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:20:47 UTC 2009=A0 =A0 38549 output= =0A> packets dropped due to no bufs, etc.=0A> =0A> 2200-2350 users online (= ipfw table load). I'll wait and see=0A> if the drop rate approaches 500-100= 0 per second as the=0A> number of online users comes close to 3-4K.=0A> =0A= > net.isr.direct=3D0=0A=0A=0AIts frightening to me that someone is managing= such a large network=0Awith dummynet. Talk about stealing your customer's = money.=0A=0ABC=0A=0A=0A
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?397638.11040.qm>