From owner-freebsd-arch@FreeBSD.ORG Tue Aug 1 13:27:56 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A138716A4DD; Tue, 1 Aug 2006 13:27:56 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4522243D8C; Tue, 1 Aug 2006 13:27:49 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id CE2D246BB0; Tue, 1 Aug 2006 09:27:49 -0400 (EDT) Date: Tue, 1 Aug 2006 14:27:49 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Andrew Gallatin In-Reply-To: <17615.18793.700752.342809@grasshopper.cs.duke.edu> Message-ID: <20060801142558.M64452@fledge.watson.org> References: <20060730141642.D16341@fledge.watson.org> <17615.18793.700752.342809@grasshopper.cs.duke.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@FreeBSD.org, net@FreeBSD.org Subject: Re: Changes in the network interface queueing handoff model X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2006 13:27:56 -0000 On Tue, 1 Aug 2006, Andrew Gallatin wrote: > > - The ifnet send queue is a separately locked object from the device driver, > > meaning that for a single enqueue/dequeue pair, we pay an extra four lock > > operations (two for insert, two for remove) per packet. > > Going forward, especially now that we support sun4v CoolThreads hardware, > we're going to want to rethink the "single lock" per transmit routine model > that most drivers have. The most expensive operation in transmit routines > is bus_dmamap_load_mbuf_sg(), especially when there is an IOMMU involved > (like on CoolThreads machines) and there is no reason why this needs to be > called with a driver's transmit lock held. I have hard data (from Solaris) > about how much fine grained locking in a 10GbE driver's transmit routine > helps. Right now, with the exception of locking for the ifnet dispatch queue, I believe our ifnet API pretty much leaves decisions about the nature and granularity of synchronization to the device driver author. The ifnet queue is high on my list to address (hence this thread) -- are there any other parts of our device driver framework that stand in the way from a device driver being modified to support greater parallelism in sending? Robert N M Watson Computer Laboratory University of Cambridge