From owner-freebsd-hackers@FreeBSD.ORG Fri Aug 18 13:06:29 2006 Return-Path: <owner-freebsd-hackers@FreeBSD.ORG> X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 24EEE16A4DA for <freebsd-hackers@freebsd.org>; Fri, 18 Aug 2006 13:06:29 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id C49B843D53 for <freebsd-hackers@freebsd.org>; Fri, 18 Aug 2006 13:06:28 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 65DF246D61; Fri, 18 Aug 2006 09:06:28 -0400 (EDT) Date: Fri, 18 Aug 2006 14:06:28 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> X-X-Sender: robert@fledge.watson.org To: Martin Eugen <martin.eugen@gmail.com> In-Reply-To: <966ba91e0608180041v3cfd9dcfh80ef89aab5404f48@mail.gmail.com> Message-ID: <20060818140316.J45647@fledge.watson.org> References: <966ba91e0608180041v3cfd9dcfh80ef89aab5404f48@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: SOCK_DGRAM optimization needed... X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD <freebsd-hackers.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers> List-Post: <mailto:freebsd-hackers@freebsd.org> List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe> X-List-Received-Date: Fri, 18 Aug 2006 13:06:29 -0000 On Fri, 18 Aug 2006, Martin Eugen wrote: > I have a simple application, that deals with lots of dgram sockets (UDP). > Thousands of them. Basically, its purpose is to maintain pairs of sockets > and when data is received on one of the sockets it peeks through it (doing > some simple statistic calculations) and then forwards it to the other > socket. Because of the hudge number of reads and writes (probably about a 10 > packets per second per socket pair) it generates a significant load on the > system, that I would like to minimize. I'm currently evaluating if it would > be possible to add simple 'routing' functionality in the socket layer in the > kernel, because frankly I'm not able to think of anything else. There are some SOCK_DGRAM optimizations present in 7.x for fast UDP send not currently present in 6.x that may be relevant, as they reduce the overhead associated with socket send on a pure datagram socket, as well as contention if simultaneous sends occur on the same socket. How to manage events and context switches is presumably also critical -- using kqueue() instead of poll() or select() may make a big difference, and you want to avoid force context switches per packet, so using an event based model rather than a threaded model makes a great deal of sense (if it meets other security and architectural requirements), as for small amounts of calculation, context switch cost will out-weight the benefits of concurrency unless concurrency is designed into the system very carefully. However, if performance is really a critical issue here, it sounds like you might want to think about pushing this into the kernel. When you say SOCK_DGRAM, do you mean UNIX domain sockets or UDP sockets? (Or something else, for that matter)? Robert N M Watson Computer Laboratory University of Cambridge