Date: Fri, 8 Jan 2016 18:24:26 +1100 (EST) From: Bruce Evans <brde@optusnet.com.au> To: Konstantin Belousov <kostikbel@gmail.com> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, Mark Delany <c2h@romeo.emu.st>, Boris Astardzhiev <boris.astardzhiev@gmail.com>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <20160108172323.W1815@besplex.bde.org> In-Reply-To: <20160107192840.GF3625@kib.kiev.ua> References: <20160104091108.50654.qmail@f5-external.bushwire.net> <20160104093859.GV3625@kib.kiev.ua> <20160104101747.58347.qmail@f5-external.bushwire.net> <20160104194044.GD3625@kib.kiev.ua> <20160104210741.32812.qmail@f5-external.bushwire.net> <CAP=KkTwfpjec2Tgnm4PRR3u8t4GEqN9Febm5HRcqapifBG-B6g@mail.gmail.com> <CA%2BhQ2%2Bh4NNz9tgSpjJdv7fXteq5tAR7o3LvjV=u08NHjRLPwmA@mail.gmail.com> <CAP=KkTzFUDsZwDDLD3n97xJW0qLVZMPduZGSX%2BeXC3UuLpVjMg@mail.gmail.com> <20160107161213.GZ3625@kib.kiev.ua> <CA%2BhQ2%2Bg6OB3MmZrW5hzNSnkcqKaKf1XGDraHfWXtSrowxKuL5g@mail.gmail.com> <20160107192840.GF3625@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 7 Jan 2016, Konstantin Belousov wrote: > On Thu, Jan 07, 2016 at 10:31:13AM -0800, Luigi Rizzo wrote: >> ... >> What we need first is experimental data that shows a performance benefit >> compared to looping around the single-packet syscall. Then we can decide >> how to proceed. > This is about performance. I don't really need the performance, but have done a lot of measurements of the the sendto() loop in an old version of ttcp and know how it works. It works poorly. It is very slow, and good error handling for ENOBUFS is impossible since select() doesn't work so the only ways to handle ENOBUFS are to busy-wait or to try to use a timeout that is not too long or so short that the kernel doesn't support it or it reduces to busy-waiting. All of the netrate utilities and newer versions of ttcp have similar problems. netblast produces interesting statistics by dropping packets differently. netrate tends to reach the limits of timeout granularity before producing anything interesting. Syscall overheads and other per-packet costs are enormous. I get a 10% speedup just by changing the malloc() in sendit() + getsockaddr() to a stack variable. The full 10% only occurs for the non-useful case of _dropped_ packets (ones that are passed to the driver but dropped due to ENOBUFS there). Counting dropped (output) packets is not completely useless for packet-blasting benchmarks. If the NIC can't reach line rate, then dropped packets are your main measure of spare capacity in the network stack. Network stack overheads are also enormous. They seem to have approximately doubled since FreeBSD-5. FreeBSD-5 drops packets better by peeking at the ifq early and not sending the packets down to the driver if they would be dropped there. This frees resources for doing more useful things. Another 10-20%. Batching won't help much here, except it almost requires better ENOBUFS handling which can be done much more easily in the kernel. sm@ pointed me to kttcp in /usr/src/tools. I didn't get around to trying it and it doesn't seem to have been maintained there. I thought that it tests at the driver level, but it actually loops doing sosend/receive() so it is too high level for driver testing but at about the level of kernel sendmmsg(). Anyway: - there must be an easy 10-30% to be gained by doing sendmmsg() in the kernel for just a few more than 1 message at a time - Someone should update and test kttcp. This gives a quick test for the batch performance that can be obtained without changes deep in the stack. Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160108172323.W1815>