Date: Fri, 20 Apr 2012 16:44:10 +0200 From: Luigi Rizzo <rizzo@iet.unipi.it> To: "K. Macy" <kmacy@freebsd.org> Cc: Andre Oppermann <andre@freebsd.org>, current@freebsd.org, net@freebsd.org Subject: Re: Some performance measurements on the FreeBSD network stack Message-ID: <20120420144410.GA3629@onelab2.iet.unipi.it> In-Reply-To: <CAHM0Q_Md4M1YRA=RJD7-xVxehvwWFjU07PdA5vWFBR6PXE14Zw@mail.gmail.com> References: <20120419133018.GA91364@onelab2.iet.unipi.it> <4F907011.9080602@freebsd.org> <20120419204622.GA94904@onelab2.iet.unipi.it> <CAHM0Q_M4wcEiWGkjWxE1OjLeziQN0vM%2B4_EYS_WComZ6=j5xhA@mail.gmail.com> <20120419212224.GA95459@onelab2.iet.unipi.it> <CAHM0Q_Md4M1YRA=RJD7-xVxehvwWFjU07PdA5vWFBR6PXE14Zw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Apr 19, 2012 at 11:06:38PM +0200, K. Macy wrote: > On Thu, Apr 19, 2012 at 11:22 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote: > > On Thu, Apr 19, 2012 at 10:34:45PM +0200, K. Macy wrote: > >> >> This is indeed a big problem. ?I'm working (rough edges remain) on > >> >> changing the routing table locking to an rmlock (read-mostly) which > >> > > >> > >> This only helps if your flows aren't hitting the same rtentry. > >> Otherwise you still convoy on the lock for the rtentry itself to > >> increment and decrement the rtentry's reference count. > >> > >> > i was wondering, is there a way (and/or any advantage) to use the > >> > fastforward code to look up the route for locally sourced packets ? > > > > actually, now that i look at the code, both ip_output() and > > the ip_fastforward code use the same in_rtalloc_ign(...) > > > >> > > >> > >> If the number of peers is bounded then you can use the flowtable. Max > >> PPS is much higher bypassing routing lookup. However, it doesn't scale > >> to arbitrary flow numbers. > > > > re. flowtable, could you point me to what i should do instead of > > calling in_rtalloc_ign() ? > > If you build with it in your kernel config and enable the sysctl > ip_output will automatically use it for TCP and UDP connections. If > you're doing forwarding you'll need to patch the forwarding path. cool. For the records, with "netsend 10.0.0.2 ports 18 0 5" on an ixgbe talking to a remote host i get the following results (with a single port netsend does a connect() and then send(), otherwise it loops around a sendto() ) net.flowtable.enabled port ns/pkt ----------------------------------------------------- not compiled in 5000 944 M_FLOWID not set 0 (disable) 5000 1004 1 (enable) 5000 980 not compiled in 5000-5001 3400 M_FLOWID not set 0 (disable) 5000-5001 1418 1 (enable) 5000-5001 1230 The small penalty when flowtable is disabled but compiled in is probably because the net.flowtable.enable flag is checked a bit deep in the code. The advantage with non-connect()ed sockets is huge. I don't quite understand why disabling the flowtable still helps there. cheers luigi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120420144410.GA3629>