From owner-freebsd-arch@FreeBSD.ORG Fri Dec 1 10:09:36 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E845816A40F; Fri, 1 Dec 2006 10:09:36 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 559BE43CA3; Fri, 1 Dec 2006 10:09:23 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.13.7/8.13.4) with ESMTP id kB1A9WZL064232; Fri, 1 Dec 2006 02:09:32 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.13.7/8.13.4/Submit) id kB1A9VA8064231; Fri, 1 Dec 2006 02:09:31 -0800 (PST) Date: Fri, 1 Dec 2006 02:09:31 -0800 (PST) From: Matthew Dillon Message-Id: <200612011009.kB1A9VA8064231@apollo.backplane.com> To: Robert Watson References: <200611292147.kATLll4m048223@apollo.backplane.com> <11606.1164837711@critter.freebsd.dk> <20061201012221.J79653@fledge.watson.org> Cc: Ivan Voras , freebsd-arch@freebsd.org Subject: Re: a proposed callout API X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2006 10:09:37 -0000 : : http://www.ece.rice.edu/~willmann/pubs/paranet_tr06-872.pdf : :Robert N M Watson Oh, that paper. You know, I talked to Alan about that paper a while back, but it isn't really possible to compare DragonFly side by side with FreeBSD yet in an SMP environment because we still have a lot of BGL junk in the network path, and because our interrupts are still going to cpu #0. The code itself is mostly MP safe, and Jeff has actually turned off the BGL in some of his own testing, but I can't do it officially yet. In anycase, that is why DragonFly wasn't used. I like the paper but I'm not sure it is possible to compare particulars of the network implementation in such different operating systems (FreeBSD vs Linux). The basic issue is, in a nutshell: per-packet per-connection ---------- ------------- long code paths short code paths long data paths shorter data paths (more cache contention) (less cache contention, work aggregation has a better change of fitting in the L1/L2, much less 'shared' data between cpus) lots of mutexes fewer mutexes or no mutexes (eats time, potential contention) no thread switches or more thread switches - protocol threads fewer thread switches process packets, not the interface interurpt (remember, interrupts (~500ns-~1us to switch threads) may be threaded) I'm sure I can think of more differences. The per-packet model almost certainly has better synchronous performance and almost certainly has better performance when operating on a limited number of connections. The basic premise is that you can make better use of any available cpu by using a fine-grained locking model. The per-connection model has the potential to process data in bulk with a far smaller cache footprint due to the separation of work, but requires a large number of connections to aggregate enough work to absorb the thread switching overhead. The basic premise is that you only care about peformance if you are actually running your machine flat out. This is largly borne out by the paper, though I again caution that the operating systems are so different from each other I don't think the graphs can be taken as an indication of the relative benefits of the networking model. -Matt Matthew Dillon