From owner-freebsd-wireless@FreeBSD.ORG Sat Oct 6 06:44:11 2012 Return-Path: Delivered-To: freebsd-wireless@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 72C341065670 for ; Sat, 6 Oct 2012 06:44:11 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4737D8FC18 for ; Sat, 6 Oct 2012 06:44:11 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id rp8so2947893pbb.13 for ; Fri, 05 Oct 2012 23:44:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=fRlP1+e8wPsy1ziTP3w/hw2zdU0zB3OfNY0fWDQnyD4=; b=d1Yw9R4tR9gY8rfrq78QtYH8ZRTi3gt1qxbxgfx35fKHmtUsS3rl1deSG/nIq3A6Qq gA8RxhFtFBldb5CgRtcxUZpaHdRUtizR+vbbjWm1L6ZvS2dh0TbBlJ5EcWW+aY0PyoT6 w4wvHlW7hWWAMIzCcAjtNrQhGMHBrrGthkZ04oaeIXrNDHT8k68G5hVP2oOtmoieC4A4 VzTPSFGU/uJqSBG3dEik/H9buMZuYnfn0KOP2lGMYH8mp/fBNOjaqVQWSzAlGUA5hAS7 TCVNTJcFJlOSit8r7kOjFSlIWhBCm/HLK4iaskFlzRyaRHhGoPguiJQIA1vqI2cbi3MV Ss2Q== MIME-Version: 1.0 Received: by 10.66.72.132 with SMTP id d4mr27253005pav.61.1349505850576; Fri, 05 Oct 2012 23:44:10 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.223.136 with HTTP; Fri, 5 Oct 2012 23:44:10 -0700 (PDT) Date: Fri, 5 Oct 2012 23:44:10 -0700 X-Google-Sender-Auth: f6zctYyF1tD2Gz5eDGsh5mA-Fms Message-ID: From: Adrian Chadd To: freebsd-wireless@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: [RFC] How the TX path works; I'd like to fix this before 10.x X-BeenThere: freebsd-wireless@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussions of 802.11 stack, tools device driver development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2012 06:44:11 -0000 Hi all, It seems the net80211 TX path is a bit .. racey. Specifically: * There's three entry points - the vap netif start at ieee80211_start(), and ic_raw_xmit() method, and ieee80211_output() for raw frames * the vap netif path will do 802.11 encapsulation, including seqno allocation * raw_xmit (iirc) doesn't do seqno allocation, but it's for management frames * ieee80211_output() does the 802.11 encap, seqno allocation and encryption if needed. * Then, the driver does the encryption, including CCMP IV. Now, there's problems: * there's multiple, overlapping entry points into ieee80211_start(), as the if_start() method doesn't _at all_ guarantee only one if_start() method is running at once. Sigh. * ic_raw_xmit() can be called at any time, and bypasses the vap ifp entirely. * ieee80211_output() can either use or bypass the vap ifp queue, depending upon whether it's a raw 802.11 or 802.3 frame. Then, to make matters worse, the whole TX path shift in the 7.x days broke fragment TX handling, as the method of using m_nextpt doesn't work at all with the ifnet macros. That's definitely a problem as it means we can't get any wifi certification _at all_. Before I continue - if_transmit() is not the answer. if_transmit() guarantess _no_ serialisation at all. So we'd stlil have to stuff things into queues. And ensure only one thing is running at once. So the question is - how do we fix this? We could do the dfbsd approach and just giant-lock the whole wifi subsystem and drivers. I don't want to do that. The ic_raw_xmit() interface needs to go away. Wireless frames should just be pushed onto the output queue like any other frames and the queue discipline should ensure management frames don't get dropped. But as long as they're not encrypted (for now, there _is_ management frame encryption as an optional part of 802.11) and don't require sequence/CCMP IVs, we're ok. Still; I think we should attach an optional "tx params" to the mbuf and send that along. Then we teach the driver to check for that particular mbuf parameter set and use it, just like how ic_raw_xmit() works right at the moment. The ic_raw_xmit() stuff is dangerous and scary, mostly because it right now does bypass a full ifnet queue and lets us send those management frames doirectly. I could wrap a big VAP TX lock around ieee80211_output() and ieee80211_start(), ensuring they don't run over each other. But long-held locks need to go away and die. yes, even the ones in the ath(4) driver that I've introduced. They're there because of a lack of synchronisation and queue design. A lot of the gige/10ge drivers use long-held locks.. sigh. I could create a net80211 TX tasklet for each vap (or ic, _maybe_) which serialises the TX path. ieee80211_start() would just schedule the tasklet to run. That would serialise all of the parallel TX entry points and solve a bunch of the subtle sequence number and other TX state races that are occuring. That doesn't solve the ic_raw_xmit() or ieee80211_output() problem, as both of those can also do TX 802.11 encapsulation and kick off parts of the state machine. It doesn't solve the same issues with the drivers .Yes, even if we converted them to use if_transmit(). iwn(4) solves this by just having a big IWN_LOCK() but it releases it when doing anything stack related. I'm not sure if it holds it across the whole TX path through iwn_start(). In any case, it's a big lock. ath(4) can and does have multiple ath_start() instances running in multiple kernel threads, fighting to dequeue frames from the ifnet. This still can cause issues with non-aggregate sequence number and CCMP IV counter allocation. Sigh. God even knows what to do about USB devices in all of this. So, now that I've ranted about the problem and some poking about solutions, I'm opening the floor for comment. I'd like to have this worked on parallel (whilst I'm doing things to the net80211 stack to support better 802.11n legacy power save and fix ps/ps-poll support) and get it implemented/debugged before 10.0. ... your turn. :) Adrian