From owner-freebsd-hackers@FreeBSD.ORG  Fri Aug 18 13:06:29 2006
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
X-Original-To: freebsd-hackers@freebsd.org
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 24EEE16A4DA
	for <freebsd-hackers@freebsd.org>; Fri, 18 Aug 2006 13:06:29 +0000 (UTC)
	(envelope-from rwatson@FreeBSD.org)
Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42])
	by mx1.FreeBSD.org (Postfix) with ESMTP id C49B843D53
	for <freebsd-hackers@freebsd.org>; Fri, 18 Aug 2006 13:06:28 +0000 (GMT)
	(envelope-from rwatson@FreeBSD.org)
Received: from fledge.watson.org (fledge.watson.org [209.31.154.41])
	by cyrus.watson.org (Postfix) with ESMTP id 65DF246D61;
	Fri, 18 Aug 2006 09:06:28 -0400 (EDT)
Date: Fri, 18 Aug 2006 14:06:28 +0100 (BST)
From: Robert Watson <rwatson@FreeBSD.org>
X-X-Sender: robert@fledge.watson.org
To: Martin Eugen <martin.eugen@gmail.com>
In-Reply-To: <966ba91e0608180041v3cfd9dcfh80ef89aab5404f48@mail.gmail.com>
Message-ID: <20060818140316.J45647@fledge.watson.org>
References: <966ba91e0608180041v3cfd9dcfh80ef89aab5404f48@mail.gmail.com>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: freebsd-hackers@freebsd.org
Subject: Re: SOCK_DGRAM optimization needed...
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
	<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
	<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 18 Aug 2006 13:06:29 -0000


On Fri, 18 Aug 2006, Martin Eugen wrote:

> I have a simple application, that deals with lots of dgram sockets (UDP). 
> Thousands of them. Basically, its purpose is to maintain pairs of sockets 
> and when data is received on one of the sockets it peeks through it (doing 
> some simple statistic calculations) and then forwards it to the other 
> socket. Because of the hudge number of reads and writes (probably about a 10 
> packets per second per socket pair) it generates a significant load on the 
> system, that I would like to minimize. I'm currently evaluating if it would 
> be possible to add simple 'routing' functionality in the socket layer in the 
> kernel, because frankly I'm not able to think of anything else.

There are some SOCK_DGRAM optimizations present in 7.x for fast UDP send not 
currently present in 6.x that may be relevant, as they reduce the overhead 
associated with socket send on a pure datagram socket, as well as contention 
if simultaneous sends occur on the same socket.  How to manage events and 
context switches is presumably also critical -- using kqueue() instead of 
poll() or select() may make a big difference, and you want to avoid force 
context switches per packet, so using an event based model rather than a 
threaded model makes a great deal of sense (if it meets other security and 
architectural requirements), as for small amounts of calculation, context 
switch cost will out-weight the benefits of concurrency unless concurrency is 
designed into the system very carefully.

However, if performance is really a critical issue here, it sounds like you 
might want to think about pushing this into the kernel.

When you say SOCK_DGRAM, do you mean UNIX domain sockets or UDP sockets?  (Or 
something else, for that matter)?

Robert N M Watson
Computer Laboratory
University of Cambridge