From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 12 19:35:32 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 65C6C16A401 for ; Thu, 12 Apr 2007 19:35:32 +0000 (UTC) (envelope-from daniel_h_taylor@yahoo.co.uk) Received: from web27705.mail.ukl.yahoo.com (web27705.mail.ukl.yahoo.com [217.146.177.239]) by mx1.freebsd.org (Postfix) with SMTP id D2A1E13C483 for ; Thu, 12 Apr 2007 19:35:31 +0000 (UTC) (envelope-from daniel_h_taylor@yahoo.co.uk) Received: (qmail 63357 invoked by uid 60001); 12 Apr 2007 19:08:49 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.co.uk; h=Message-ID:X-YMail-OSG:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=2x2bnKtyS4SjXke2dcKDWr8f9Dhe5ekwG23iO42W3WzKiAjYLhVlx9ngTiZa7zk3svAjA+reoykcdHjkFXuDmLI1yuQaGHXGUwKXm7PN44sFijx7x2QGX8+lNr3Uu1OVi3vq9+i+29hVGfVf/K5K5VtI7KRTkhkihP6gc1D/5AA= ; Message-ID: <20070412190849.63355.qmail@web27705.mail.ukl.yahoo.com> X-YMail-OSG: F850BhQVM1lWRo3o6nMkPbx7pYq.cVEmZg84.QN28e1qzQWoApAD1F0oyXAGplyi7dIISkISkAWq6dYqrxJoJ6rLmb6QRU64K4EyUzM8b3p8Kk5rcEWNmjVilc0- Received: from [38.117.185.138] by web27705.mail.ukl.yahoo.com via HTTP; Thu, 12 Apr 2007 20:08:49 BST Date: Thu, 12 Apr 2007 20:08:49 +0100 (BST) From: Daniel Taylor To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: tcp connection splitter X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Apr 2007 19:35:32 -0000 Not a FreeBSD question per se but I thought I'd ask anyway, as there are many knowledgeable people here. I am writing a server (which btw is going to run on FreeBSD), acting as a streaming gateway for multiple TCP clients. The server connects to a source of data (which only accepts a single connection) and needs to distribute this data to multiple TCP clients. With the exception of the initial login sequence, communication is mostly one-way (data source to server, and server to clients). What is the most efficient data structure for this scenario? I need something that would scale to many client connections. The simplest solution is to maintain a separate packet queue for each client, and copy each packet received from the source to all N client queues. Of course, this solution involves a lot of memory allocations (the server receives 1-10MB of data/second), a lot of memcpy()s, and doesn't scale very well. Also, adding a packet to N queues is expensive because it needs to acquire and release N mutex locks (one for each client queue.) At the moment, I have a solution which allocates a reference-counted packet, and adds a *pointer* to it to each of the N client queues. This saves on memcpy because copying a pointer is very fast. Each enqueue bumps the refcount, each dequeue decreases it; when the refcount drops to 0, the packet is free()'d (by whomever happened to dequeue it last). If the clients are "slow" (can't keep up with the source), it's ok to drop them; hence I'm toying with the idea of having a single circular queue accessed by N clients. It would certainly save on memory allocation, but at the same time increase lock contention, because the server and N clients will all be locking the same queue mutex to access it. I could use read-write locks here (single writer, multiple readers) but it doesn't help with write lock contention. There must be some clever lockless algorithm to achieve the same effect. Any suggestions? Thanks, Dan ___________________________________________________________ To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com