From owner-cvs-all@FreeBSD.ORG Sat Dec 23 22:08:15 2006 Return-Path: X-Original-To: cvs-all@FreeBSD.org Delivered-To: cvs-all@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A1F316A40F; Sat, 23 Dec 2006 22:08:15 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id BEAB513C434; Sat, 23 Dec 2006 22:08:12 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id B364C46DFD; Sat, 23 Dec 2006 16:36:33 -0500 (EST) Date: Sat, 23 Dec 2006 21:36:33 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: John Polstra In-Reply-To: Message-ID: <20061223213014.U35809@fledge.watson.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: cvs-src@FreeBSD.org, Scott Long , src-committers@FreeBSD.org, cvs-all@FreeBSD.org Subject: Re: cvs commit: src/sys/dev/bge if_bge.c X-BeenThere: cvs-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CVS commit messages for the entire tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Dec 2006 22:08:15 -0000 On Sat, 23 Dec 2006, John Polstra wrote: >> That said, dropping and regrabbing the driver lock in the rxeof routine of >> any driver is bad. It may be safe to do, but it incurs horrible >> performance penalties. It essentially allows the time-critical, high >> priority RX path to be constantly preempted by the lower priority if_start >> or if_ioctl paths. Even without this preemption and priority inversion, >> you're doing an excessive number of expensive lock ops in the fast path. > > We currently make this a lot worse than it needs to be by handing off the > received packets one at a time, unlocking and relocking for every packet. > It would be better if the driver's receive interrupt handler would harvest > all of the incoming packets and queue them locally. Then, at the end, hand > off the linked list of packets to the network stack wholesale, unlocking and > relocking only once. (Actually, the list could probably be handed off at > the very end of the interrupt service routine, after the driver has already > dropped its lock.) We wouldn't even need a new primitive, if ether_input() > and the other if_input() functions were enhanced to deal with a possible > list of packets instead of just a single one. I try this experiement every few years, and generally don't measure much improvement. I'll try it again with 10gbps early next year once back in the office again. The more interesting transition is between the link layer and the network layer, which is high on my list of topics to look into in the next few weeks. In particular, reworking the ifqueue handoff. The tricky bit is balancing latency, overhead, and concurrency... FYI, there are several sets of patches floating around to modify if_em to hand off queues of packets to the link layer, etc. They probably need updating, of course, since if_em has changed quite a bit in the last year. In my implementaiton, I add a new input routine that accepts mbuf packet queues. Robert N M Watson Computer Laboratory University of Cambridge