From owner-freebsd-current@FreeBSD.ORG Wed Aug 7 20:00:26 2013 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4F59177B for ; Wed, 7 Aug 2013 20:00:26 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B1A0A2522 for ; Wed, 7 Aug 2013 20:00:25 +0000 (UTC) Received: (qmail 55691 invoked from network); 7 Aug 2013 20:46:00 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 7 Aug 2013 20:46:00 -0000 Message-ID: <5202A74E.4060602@freebsd.org> Date: Wed, 07 Aug 2013 22:00:14 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Luigi Rizzo Subject: Re: [net] protecting interfaces from races between control and data ? References: <20130805215319.GA43271@onelab2.iet.unipi.it> <201308070326.r773QCLD035541@mail.karels.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Adrian Chadd , current@freebsd.org, mike@karels.net, Bryan Venteicher , Navdeep Parhar , net@freebsd.org, Giuseppe Lettieri X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Aug 2013 20:00:26 -0000 On 07.08.2013 09:18, Luigi Rizzo wrote: > On Wed, Aug 7, 2013 at 5:26 AM, Mike Karels > wrote: > Jumping to (near) the end of the thread, I like most of Andre's proposal. > Running with minimal locks at this layer is an admirable goal, and I agree > with most of what was said. I have a few observations on the general changes, > or related issues: > > There was mention of taskqueues. I think that with MSI-X, taskqueues > should not be needed or used. More specifically, having separate ithreads > and taskqueues, with ithreads deferring to taskqueues after some limit, makes > sense only for MSI and legacy interrupts. With MSI-X, an interrupt thread > should be able to process packets indefinitely with sufficient CPU resources, > and there is no reason to context switch to a different thread periodically. > A periodic "yield" might be reasonable, but if it is necessary, small packet > performance will suffer. However, most of this is internal to the driver. > > > i am not completely clear on what is the difference between ithreads and taskqueues. The difference between ithreads and taskqueues is actually very small and mostly in name and how they're set up and kicked off when work is to be done. Both are real kernel threads. Most of the confusion, also in Mikes response, seems to come from the name ithreads for interrupt-threads. However an ithread is not running in interrupt context, only the interrupt handler (called filter) does. Scheduling a taskqueue from an ithread is a bit round-about but commonly done. The bus_setup_intr(9) man page isn't helping to clear up the confusion. The idea is that a (legacy) interrupt is handled in two stages: 1) a small function reading the hardware interrupt status register to determine if this hardware actually raised an interrupt, and acknowledge it (to prevent additional interrupts from firing while this one is handled). If it wasn't this card, it hands off to the handler if this interrupt line is shared; 2) the actual function/thread handling the data that was indicated by the interrupt. Only step 1 runs in interrupt context and thus must be very small and avoid any form of blocking. Step 2 is a normal kernel thread set up as an ithread running at an elevated priority compared to user-space processes/threads. MSI and MSI-X always are exclusive and non-shared interrupts. Thus a handler running in interrupt context isn't necessary for non-legacy interrupt sources. The ithread can be scheduled right away to do its work. > Also, Andre's proposal requires to force-kill the ithread, but i am unclear on how to do it > safely (i.e. without leaving the data structures in some inconsistent state), unless ithread > periodically yields the CPU when it is in a safe state. While this is internal to the driver, > we should probably provide some template code to avoid that each driver implements > its own way to shutdown the ithread. Yes, when done a well-tested, stable and performant template will be provided. -- Andre