Date: Fri, 8 Jan 2016 09:02:37 -0800 From: Adrian Chadd <adrian.chadd@gmail.com> To: Bruce Evans <brde@optusnet.com.au> Cc: Mark Delany <c2h@romeo.emu.st>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <CAJ-Vmom26mukSv3JmsmNiAONvpc6f1bQ%2BujO25qefGHY=5przA@mail.gmail.com> In-Reply-To: <20160108204606.G2420@besplex.bde.org> References: <20160104101747.58347.qmail@f5-external.bushwire.net> <20160104194044.GD3625@kib.kiev.ua> <20160104210741.32812.qmail@f5-external.bushwire.net> <CAP=KkTwfpjec2Tgnm4PRR3u8t4GEqN9Febm5HRcqapifBG-B6g@mail.gmail.com> <CA%2BhQ2%2Bh4NNz9tgSpjJdv7fXteq5tAR7o3LvjV=u08NHjRLPwmA@mail.gmail.com> <CAP=KkTzFUDsZwDDLD3n97xJW0qLVZMPduZGSX%2BeXC3UuLpVjMg@mail.gmail.com> <20160107161213.GZ3625@kib.kiev.ua> <CA%2BhQ2%2Bg6OB3MmZrW5hzNSnkcqKaKf1XGDraHfWXtSrowxKuL5g@mail.gmail.com> <20160107192840.GF3625@kib.kiev.ua> <20160108172323.W1815@besplex.bde.org> <20160108075815.3243.qmail@f5-external.bushwire.net> <CAJ-VmonYPhcN-gikuYQU_k5GaTAqTijoxR_0ORV4BZqsHMRJSg@mail.gmail.com> <20160108204606.G2420@besplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8 January 2016 at 03:02, Bruce Evans <brde@optusnet.com.au> wrote: > On Fri, 8 Jan 2016, Adrian Chadd wrote: > >> On 7 January 2016 at 23:58, Mark Delany <c2h@romeo.emu.st> wrote: >>> >>> On 08Jan16, Bruce Evans allegedly wrote: >>>> >>>> If the NIC can't reach line rate >>> >>> >>>> Network stack overheads are also enormous. >>> >>> >>> Bruce makes some excellent points. >>> >>> I challenge anyone to get line rate UDP out of FBSD (or Linux) for a >>> 1G NIC yet alone a 10G NIC listening to a single port. It was exactly >>> my frustration with UDP performance that led me down the path of >>> *mmsg() and netmap. >>> >>> Frankly this is an opportunity for FBSD as UDP performance appears to >>> be a neglected area. >> >> >> I'm there, on 16 threads. >> >> I'd rather we do it on two or three, as a lot of time is wasted in >> producer/consumer locking. but yeah, 500k tx/rx should be doable per >> CPU with only locking changes. .. and I did mean "kernel producer/consumer locking changes." > > Line rate for 1 Gbps is about 1500 kpps (small packets). > > With I218V2 (em), I see enormous lock contention above 3 or 4 (user) > threads, and 8 are slightly slower than 1. 1 doesn't saturate the NIC, > and 2 is optimal. > The RSS support in -HEAD lets you get away with parallelising UDP streams very nicely. The framework is pretty simple (!): * drivers ask the RSS code for the RSS config and RSS hash to use, and configure the hardware appropriately; * the netisr input paths check the existence of the RSS hash and will calculte it in software if reqiured; * v4/v6 reassembly is done (at the IP level, /not/ at the protocol level) and if it needs a new RSS hash / netisr reinjection, that'll happen; * the PCB lookup code for listen sockets now allows one listen socket per RSS bucket - as the RSS / PCBGROUPS code already extended the PCB to have one PCB table per RSS bucket (as well as a global one); So: * userland code queries RSS for the CPU and RSS bucket setup; * you then create one listen socket per RSS bucket, bind it to the local thread (if you want) and tell it "you're in RSS bucket X"; * .. and then in the UDP case for local-bound sockets, the transmit/receive path does not require modifying the global PCB state, so the locking is kept per-RSS bucket, and scales linearly with the number of CPUs you have (until you hit the NIC queue limits.) https://github.com/erikarn/freebsd-rss/ and: http://adrianchadd.blogspot.com/2014/06/hacking-on-receive-side-scaling-rss-on.html http://adrianchadd.blogspot.com/2014/07/application-awareness-of-receive-side.html http://adrianchadd.blogspot.com/2014/08/receive-side-scaling-figuring-out-how.html http://adrianchadd.blogspot.com/2014/09/receive-side-scaling-testing-udp.html http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmom26mukSv3JmsmNiAONvpc6f1bQ%2BujO25qefGHY=5przA>