Date: Fri, 20 Apr 2012 18:29:10 +0200 From: "K. Macy" <kmacy@freebsd.org> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: Andre Oppermann <andre@freebsd.org>, current@freebsd.org, net@freebsd.org Subject: Re: Some performance measurements on the FreeBSD network stack Message-ID: <CAHM0Q_P3XvOZrfJW7dUa23H%2BYUMe608hoKY41DZ7BGGc=cKniQ@mail.gmail.com> In-Reply-To: <20120420144410.GA3629@onelab2.iet.unipi.it> References: <20120419133018.GA91364@onelab2.iet.unipi.it> <4F907011.9080602@freebsd.org> <20120419204622.GA94904@onelab2.iet.unipi.it> <CAHM0Q_M4wcEiWGkjWxE1OjLeziQN0vM%2B4_EYS_WComZ6=j5xhA@mail.gmail.com> <20120419212224.GA95459@onelab2.iet.unipi.it> <CAHM0Q_Md4M1YRA=RJD7-xVxehvwWFjU07PdA5vWFBR6PXE14Zw@mail.gmail.com> <20120420144410.GA3629@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
Comments inline below: On Fri, Apr 20, 2012 at 4:44 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote: > On Thu, Apr 19, 2012 at 11:06:38PM +0200, K. Macy wrote: >> On Thu, Apr 19, 2012 at 11:22 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote= : >> > On Thu, Apr 19, 2012 at 10:34:45PM +0200, K. Macy wrote: >> >> >> This is indeed a big problem. ?I'm working (rough edges remain) on >> >> >> changing the routing table locking to an rmlock (read-mostly) whic= h >> >> > >> >> >> >> This only helps if your flows aren't hitting the same rtentry. >> >> Otherwise you still convoy on the lock for the rtentry itself to >> >> increment and decrement the rtentry's reference count. >> >> >> >> > i was wondering, is there a way (and/or any advantage) to use the >> >> > fastforward code to look up the route for locally sourced packets ? >> > >> > actually, now that i look at the code, both ip_output() and >> > the ip_fastforward code use the same in_rtalloc_ign(...) >> > >> >> > >> >> >> >> If the number of peers is bounded then you can use the flowtable. Max >> >> PPS is much higher bypassing routing lookup. However, it doesn't scal= e >> >> to arbitrary flow numbers. >> > >> > re. flowtable, could you point me to what i should do instead of >> > calling in_rtalloc_ign() ? >> >> If you build with it in your kernel config and enable the sysctl >> ip_output will automatically use it for TCP and UDP connections. If >> you're doing forwarding you'll need to patch the forwarding path. > > cool. > For the records, with "netsend 10.0.0.2 ports 18 0 5" on an ixgbe > talking to a remote host i get the following results (with a single > port netsend does a connect() and then send(), otherwise it > loops around a sendto() ) > Sorry, 5000 vs 5000-5001 means 1 vs 2 streams? Does this mean for a single socket the overhead is less without it compiled in than with it compiled in but enabled? That is certainly different from what I see with TCP where I see a 30% increase in aggregate throughput the last time I tried this (on IPoIB). For the record the M_FLOWID is used to pick the transmit queue so with multiple streams you're best of setting it if your device has more than one hardware device queue. > =A0 =A0 =A0 =A0net.flowtable.enabled =A0 port =A0 =A0 =A0 =A0 =A0 =A0ns/p= kt > =A0 =A0 =A0 =A0----------------------------------------------------- > =A0 =A0 =A0 =A0not compiled in =A0 =A0 =A0 =A0 5000 =A0 =A0 =A0 =A0 =A0 = =A0 944 =A0 =A0M_FLOWID not set > =A0 =A0 =A0 =A00 (disable) =A0 =A0 =A0 =A0 =A0 =A0 5000 =A0 =A0 =A0 =A0 = =A0 =A01004 > =A0 =A0 =A0 =A01 (enable) =A0 =A0 =A0 =A0 =A0 =A0 =A05000 =A0 =A0 =A0 =A0= =A0 =A0 980 > > =A0 =A0 =A0 =A0not compiled in =A0 =A0 =A0 =A0 5000-5001 =A0 =A0 =A0 3400= =A0 =A0M_FLOWID not set > =A0 =A0 =A0 =A00 (disable) =A0 =A0 =A0 =A0 =A0 =A0 5000-5001 =A0 =A0 =A0 = 1418 > =A0 =A0 =A0 =A01 (enable) =A0 =A0 =A0 =A0 =A0 =A0 =A05000-5001 =A0 =A0 = =A0 1230 > > The small penalty when flowtable is disabled but compiled in is > probably because the net.flowtable.enable flag is checked > a bit deep in the code. > > The advantage with non-connect()ed sockets is huge. I don't > quite understand why disabling the flowtable still helps there. Do you mean having it compiled in but disabled still helps performance? Yes, that is extremely strange. -Kip --=20 =A0 =A0=93The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don=92t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won=92t take measure of their own strength, for fear of antagonizing their own weakness. Those who don=92t like to make waves=97or enemies. =A0 =A0Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It=92s the reductionist approach to life: if you keep it small, you=92ll keep it under control. If you don=92t make any noise, the bogeyman won=92t find you. =A0 =A0But it=92s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! >From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. =A0 =A0I choose my own way to burn.=94 =A0 =A0Sophie Scholl
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHM0Q_P3XvOZrfJW7dUa23H%2BYUMe608hoKY41DZ7BGGc=cKniQ>