From owner-freebsd-wireless@FreeBSD.ORG Thu Feb 14 19:50:25 2013 Return-Path: Delivered-To: freebsd-wireless@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 850C7560; Thu, 14 Feb 2013 19:50:25 +0000 (UTC) (envelope-from monthadar@gmail.com) Received: from mail-we0-x233.google.com (we-in-x0233.1e100.net [IPv6:2a00:1450:400c:c03::233]) by mx1.freebsd.org (Postfix) with ESMTP id D5DFB9C5; Thu, 14 Feb 2013 19:50:24 +0000 (UTC) Received: by mail-we0-f179.google.com with SMTP id p43so491605wea.38 for ; Thu, 14 Feb 2013 11:50:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=iKoeADO4k0xj3LUTaS4l48I1cyr512Jgl2AiUuDN9ew=; b=fkjyhziu6dnoytUG0XjkdeMoA39yCSz6QAbvYPJaD82MALNg/5tAoFj/F8vulHEQob vNKhUSYNsUwiSHiLjVIXm5NyX9jcDoQpsJU3i87LWgwtoFUli9yj9bWZ7dUiAPMmpo/E pZlpY2mQx6061WnwYbfb2Yq0Lx7doYSKdZs8fY51zOgTKhb0DVgDZD/gvq3/R5hVvJ5s 1qe7bQ4ubmK3NoZnGzwSBGTFvqFafGpkdKdIV0RBwmtb1OJ0n28kpPPdicdmt7N5RL2z CqeV/BoL1LXlY2yUoRp6cVI8OMeudU30A2rxRvr0TQ2hIOgLy2Oj/W62ZTMfBPXdv4i8 NJLQ== MIME-Version: 1.0 X-Received: by 10.194.57.206 with SMTP id k14mr48091742wjq.26.1360871423802; Thu, 14 Feb 2013 11:50:23 -0800 (PST) Received: by 10.227.59.19 with HTTP; Thu, 14 Feb 2013 11:50:23 -0800 (PST) In-Reply-To: References: Date: Thu, 14 Feb 2013 20:50:23 +0100 Message-ID: Subject: Re: [RFC] serialising net80211 TX From: Monthadar Al Jaberi To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-wireless@freebsd.org X-BeenThere: freebsd-wireless@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussions of 802.11 stack, tools device driver development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Feb 2013 19:50:25 -0000 On Thu, Feb 14, 2013 at 6:14 AM, Adrian Chadd wrote: > Hi, > > I'd like to work on the net80211 TX serialisation now. I'll worry > about driver serialisation and if_transmit methods later. > > The 30 second version - it happens in parallel, which means preemption > and multi-core devices can and will hit lots of subtle and > hard-to-debug races in the TX path. > > We actually need an end-to-end serialisation method - not only for the > 802.11 state (sequence number, correct aggregation handling, etc) but > to line up 802.11 sequence number allocation with the encryption IV/PN > values. Otherwise you end up with lots of crazy subtle out of order > packets occuring. The other is the seqno/CCMP IV race between the raw > transmit path and the normal transmit path. There are other nagging > issues that I'm trying to resolve - but, one thing at a time. > > So there are three current contenders: > > * wrap everything in net80211 TX in a per-vap TX lock; grab it at the > beginning of ieee80211_output() and ieee80211_start(), and don't > release it until the frame is queued to something (a power save queue, > an age queue, the driver.) That guarantees that the driver is called > in lock-step with each frame being processed. > * do deferred transmit- ie, the net80211 entry points simply queue > mbufs to a queue, and a taskqueue runs over the ifnet queue and runs > those frames in-order. There's no need for a lock here as there's only > one sending context (either per-VAP or per-IC). Seems like the best architectural wise, first-in first out. I am just thinking of one can extend this too have like more than one queue, more like the QoS concept, and each packet have a time-stamp assigned to it.Would that help? What I am thinking is more of an optimization maybe. the Wifi can run in different modes BSS (AP/STA), IBSS and MBSS. And for the MBSS mode there is not just data comming from up the stack, or sending beacons. The MBSS also needs to handle data forwarding and management frame propagation. Would it help to have different queues? A mesh path discovery would be delayed too much if the complete system is highly active? Should the path request frames have higher priority to lets say data frames? Would that help for when sending beacons from the stack too. Maybe I am way off :) > * A hybrid setup - use a per-vap TX lock; do a try-acquire on it and > direct dispatch from the queue head if we have it; otherwise defer > frames into a queue and have a taskqueue handle those. That maybe gives the most speed up? > > 1) is what drivers like iwn(4) do internally. > 2) is what I've tinkered with - but we become a slave to the > scheduler. Task switching is expensive and unpredictable; doubly so > for a non-preemption kernel. We'd have to run the TX taskqueue at some > very high priority to get something resembling direct-dispatch > behaviour. > 3) is what the gige/10ge drivers do. They hold a big TX lock for each > TX (from xxx_transmit() to hardware dispatch) and if they can't > acquire the TX lock, they defer it to a drbd lockless ring buffer and > service that via a taskqueue. > > I can implement any of the above. architecturally I'd prefer 2) - it > massively simplifies and streamlines things, but the scheduling > latency is just plain stab-worthy.I'm tempted to just do 1) for now > and turn it into 3) if we need to. > > The main reason against doing 1) (and why 2) is nicer) is recursion - > if the TX path wants to call the net80211 TX code for some odd reason, > we'll hit lock recursion. I'd rather have the system crash at this > point (and then fix the misbehaving driver) but that's just me. > > So - what do people think? > > Once this is done I'd like to make sure that the wifi chipset drivers > do the same - ie, ensure that the frame order is preserved both > between the normal and the raw xmit paths. > That should fix all of the odd CCMP out of order crap that I see under > heavy, heavy test conditions. > > Thanks, > > > Adrian > _______________________________________________ > freebsd-wireless@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-wireless > To unsubscribe, send any mail to "freebsd-wireless-unsubscribe@freebsd.org" -- Monthadar Al Jaberi