From owner-freebsd-current@FreeBSD.ORG Mon Aug 5 17:17:30 2013 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 48085886; Mon, 5 Aug 2013 17:17:30 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-x235.google.com (mail-we0-x235.google.com [IPv6:2a00:1450:400c:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 81BD7295A; Mon, 5 Aug 2013 17:17:29 +0000 (UTC) Received: by mail-we0-f181.google.com with SMTP id p58so2687490wes.40 for ; Mon, 05 Aug 2013 10:17:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=53AuICVCdcZlIghBYscnVB5whDr/1xVLfYIPIe4O16U=; b=h/gmYHTWNFnwUvRBYGPA6SDt4GD2n/jo+fzv4NUxKgnwpxKInBTDfuSRWJ/dfZfj10 2l/nySiD+H1tqhMll727a2vf9LChWGUzma5UVnEBmWCqdsxKqPWR8TFL1aNWHmJQ/TlH CiHBER8erGa1RRj97541I6dHet1Ohfai0+0Yi51Qr6kbQdN0iIs9DGKDKXcjrDHGo/O3 8RmnABsBelJw8lgnU8n+y2SBNcziTuzwnnVqDwDIF/FTxUEht+ylabFsLKVY2UbSLbHs 4AG7njqS8+4BXPzvqvwVKoey9fN4YSG42NBJ2/mx6DHyaDKcS60l30HnndMGtAors8S1 hjMw== MIME-Version: 1.0 X-Received: by 10.180.185.148 with SMTP id fc20mr7612847wic.0.1375723047877; Mon, 05 Aug 2013 10:17:27 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.116.136 with HTTP; Mon, 5 Aug 2013 10:17:27 -0700 (PDT) In-Reply-To: <51FFDD1E.1000206@FreeBSD.org> References: <20130805082307.GA35162@onelab2.iet.unipi.it> <2034715395.855.1375714772487.JavaMail.root@daemoninthecloset.org> <51FFDD1E.1000206@FreeBSD.org> Date: Mon, 5 Aug 2013 10:17:27 -0700 X-Google-Sender-Auth: xvHHdnO8uRlzvUdaJTkNkn7V8Fo Message-ID: Subject: Re: [net] protecting interfaces from races between control and data ? From: Adrian Chadd To: Navdeep Parhar Content-Type: text/plain; charset=ISO-8859-1 Cc: Bryan Venteicher , Giuseppe Lettieri , Luigi Rizzo , current@freebsd.org, net@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Aug 2013 17:17:30 -0000 I'm travelling back to San Jose today; poke me tomorrow and I'll brain dump what I did in ath(4) and the lessons learnt. The TL;DR version - you don't want to grab an extra lock in the read/write paths as that slows things down. Reuse the same per-queue TX/RX lock and have: * a reset flag that is set when something is resetting; that says to the queue "don't bother processing anything, just dive out"; * 'i am doing Tx / Rx' flags per queue that is set at the start of TX/RX servicing and finishes at the end; that way the reset code knows if there's something pending; * have the reset path grab each lock, set the 'reset' flag on each, then walk each queue again and make sure they're all marked as 'not doing TX/RX'. At that point the reset can occur, then the flag cna be cleared, then TX/RX can resume. -adrian On 5 August 2013 10:13, Navdeep Parhar wrote: > On 08/05/13 09:15, Luigi Rizzo wrote: >> On Mon, Aug 5, 2013 at 5:46 PM, Adrian Chadd wrote: >> >>> On 5 August 2013 07:59, Bryan Venteicher >>> wrote: >>> >>>> What I've done in my drivers is: >>>> * Lock the core mutex >>>> * Clear IFF_DRV_RUNNING >>>> * Lock/unlock each queue's lock >>> >>> .. and I think that's the only sane way of doing it. >>> >> >> yeah, this was also the solution we had in mind, i was surprised >> not find this pattern in the drivers i have looked at. >> >> Also there are drivers (chelsio ?) which do not seem to have locks on the >> receive interrupt handlers ? > > This is correct. cxgbe(4) does not have any locks on rx, just a "state" > for each rx queue that's maintained with atomic ops. > > Regards, > Navdeep > > >> >> Does anyone know how linux copes with the same problem ? >> >> They seem to have an rtnl_lock() which is a global lock for all >> configuration >> of netdevices (would replace our per-interface 'core lock' above), >> but i am totally unclear on how individual tx threads and interrupt handlers >> acknowledge that they have read the change in status. >> >> cheers >> luigi >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >