Date: Mon, 4 Jan 2016 05:32:36 +0800 (CST) From: HuanHuan <mailhuanhuan@gmail.com> To: Rui Paulo <rpaulo@me.com> Cc: HuanHuan <mailhuanhuan@gmail.com>, freebsd-net@freebsd.org, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <alpine.BSF.2.20.1601040524170.84701@localhost.my.domain> In-Reply-To: <1451841004.10139.34.camel@me.com> References: <alpine.BSF.2.20.1601031833130.84701@localhost.my.domain> <1451841004.10139.34.camel@me.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rui, There are no existing applications, but these two calls are for developing new application on 10G links. Currently I use netgraph, especially ng_socket node. And a simple recvfrom() on a ng_socket costs ~5us or so (200K per second). And there are many netgraph sockets. So it's good to reduce the time by ultilizing send/recvmmsg() if there are these two syscalls. Even a simple-loop like implmentation like linux's will be good as Luigi has suggested. On Sun, 3 Jan 2016, Rui Paulo wrote: > On Sun, 2016-01-03 at 18:34 +0800, HuanHuan wrote: >> NetBSD 7.0 has just introduced these two syscalls. >> And Linux also has them. >> >> Does FreeBSD have them? Or plan to support them in the future? > > FreeBSD does not have them. It doesn't seem especially hard to > implement, though. Do you know any major application already using > them? > > -- > Rui Paulo > > From owner-freebsd-net@freebsd.org Sun Jan 3 21:47:22 2016 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4AA95A60FAF for <freebsd-net@mailman.ysv.freebsd.org>; Sun, 3 Jan 2016 21:47:22 +0000 (UTC) (envelope-from c2h@romeo.emu.st) Received: from f5.bushwire.net (f5.bushwire.net [IPv6:2607:fc50:1000:5b00::2]) by mx1.freebsd.org (Postfix) with ESMTP id 316671999 for <freebsd-net@freebsd.org>; Sun, 3 Jan 2016 21:47:21 +0000 (UTC) (envelope-from c2h@romeo.emu.st) Received: by f5.bushwire.net (Postfix, from userid 1001) id 03718AC908; Sun, 3 Jan 2016 13:47:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/simple; d=emu.st; s=2015; t=1451857641; bh=HkFM0YnXVjWd+c2Og6SCgVNr6P8=; h=Comments:Received:Date:Message-ID:From:To:Subject:References: MIME-Version:Content-Type:Content-Disposition:In-Reply-To; b=p0OUahspntnNYEAXLPa1Igq7E+LLDGpSe9vEzONxdum2wTgCv+n7Si7IYzCCM3gkP FWPRHlTW8160Wgl01Pbd59h2hze02pByYpdxumXMDYDKFgp6T349TvtAUp+d+SCVsd qToAX6s+vqSVzSjJ9OhUVpeCQzhdJlVJKOVc5vyQ=c5vyQ= Comments: QMDA 0.3 Received: (qmail 72015 invoked by uid 1001); 3 Jan 2016 21:47:20 -0000 Date: 3 Jan 2016 21:47:20 +0000 Message-ID: <20160103214720.72014.qmail@f5-external.bushwire.net> From: "Mark Delany" <c2h@romeo.emu.st> To: freebsd-net@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? References: <alpine.BSF.2.20.1601031833130.84701@localhost.my.domain> <1451841004.10139.34.camel@me.com> <alpine.BSF.2.20.1601031744040.20884@fledge.watson.org> <CAJ-Vmomxcn+iYJAzNViL8WnepsCihrkTuHd8=0O6vONKsTExCA@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <CAJ-Vmomxcn+iYJAzNViL8WnepsCihrkTuHd8=0O6vONKsTExCA@mail.gmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sun, 03 Jan 2016 21:47:22 -0000 On 03Jan16, Adrian Chadd allegedly wrote: > It doesn't help at low connection rates. It helps at high connection / > concurrency rates as the time going in/out of the kernel and getting > back to steady state execution changes. This is what I found when I did a comparison at $dayjob. You obviously need more than one message per *mmsg() call to benefit from the syscall amortization which can only occur at very high message rates. It also depends on whether the *mmsg() calls are implemented as mere wrappers around recvmsg() and sendmsg() or whether the implementation goes deeper into the driver to get better batching benefits. (I vaguely thought that one or two drivers in Linux had real recvmmsg() support.) But there are a number of other factors that come into play beyond message rates. The first is that a typical UDP application does very little work per message thus the syscall costs are proportionally more significant than for TCP applications. Eg. DNS or NTP vs HTTP. The second is that a typical syscall sequence of a high performance program likely involves 4 syscalls per packet: select/kqueue/epoll, recvmsg, gettimeofday and sendmsg. The third is that a high performance program probably is multi-threaded so add in a couple of locking syscalls around select/kqueue and you're up to 6 syscalls per request packet. That's a lot of syscalls for an application that might only need a small number of memory references to process a request. The final thing is that on both FBSD and Linux I found a surprising amount of serialization within the UDP socket stack so having multiple threads sit on a blocking recvmsg() at high packet rates is not as productive as you might expect. This level of serialization surprized me given that UDP processing within the kernel should be pretty minimal. Combining all these factors I found it impossible to construct any sort of UDP application that would run any where near line rates on modern beefy machines. It it's not apparent, these are all arguments for implementing *mmsg() calls and for looking at UDP stack serialization. Fortunately for FBSD there is netmap which is moderately well suited to UDP applications... though you still have to do a fair amount of packet decode and encode and it's only applicable in dedicated or specialized deployments. A final observation is that the semantics of *mmsg() may benefit from a bit of scrutiny as they cannot accurately represent some conditions possible with multiple *msg() calls. Eg, if a signal arrives after more than zero packets have been processed by recvmmsg() what is the correct return value? -1 or the count of messages returned? Mark.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.20.1601040524170.84701>