From owner-freebsd-wireless@FreeBSD.ORG Sat Oct 6 19:00:28 2012 Return-Path: Delivered-To: freebsd-wireless@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8F7CD1065690 for ; Sat, 6 Oct 2012 19:00:28 +0000 (UTC) (envelope-from moonlightakkiy@yahoo.ca) Received: from nm18.bullet.mail.gq1.yahoo.com (nm18.bullet.mail.gq1.yahoo.com [98.136.217.1]) by mx1.freebsd.org (Postfix) with SMTP id 46D4F8FC24 for ; Sat, 6 Oct 2012 19:00:28 +0000 (UTC) Received: from [98.137.12.61] by nm18.bullet.mail.gq1.yahoo.com with NNFMP; 06 Oct 2012 19:00:22 -0000 Received: from [208.71.42.213] by tm6.bullet.mail.gq1.yahoo.com with NNFMP; 06 Oct 2012 19:00:22 -0000 Received: from [127.0.0.1] by smtp224.mail.gq1.yahoo.com with NNFMP; 06 Oct 2012 19:00:22 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.ca; s=s1024; t=1349550022; bh=XlFk0FSr66rzoA1sieFuWJD0InrXhAGHHVa9pciSJEw=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:Received:MIME-Version:Received:Received:Date:Message-ID:Subject:From:To:Content-Type; b=4S8+9rV8s2i7Qdgn4H1IK1Agaz5YDY6+hoozGjRjfoZye4ZzxgRNqSVu1RBPpJNeGyV4Nruzt+RDpxDbOiVsgWm5y5DOpVch2jb2ocRfQNdbRE4+0xL/FK3/yZmRq8bPuZl0LuxgP2SsqZFI/dLg33bAi686a6EusqfxA4uXtss= X-Yahoo-Newman-Id: 485837.9237.bm@smtp224.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: aNGPN4AVM1lYcdzepptxosF9Oqi.DRbcV_Fejy5al2Bcyap pxPEMr_e7h2ZK.QNjN__f.6Qp82dcr31DbXFeYTj1yvmclvlZ8oyozS_IgYA XSQaDtetbmUv7xXgZnxx7r7GKHTqhvvQqI9KNPTAA9G9wNggQIb1rIq7hqCR Cu5LVFmxRGBNpg6xrCAIm4gq81qDBKZ0.0g18o3ErhJ9wWu1iaVBEMeuhBIT B856WjKFw6iaVBw9iHuaUfT9GnQMm4UESbeymQOxK8xyG6pthUENZhOdNXr8 KnZtHODjm3tN9pDoeYcTW5WNdAzFOOmouX.OpeRhXZYhOCWekoMdInYHama7 3Lsgj43kEzRGQqO.lscx9_O.yoDeAW05tWU9RT9eYSahM.MsQF.HFkVhuJzn VmqC42u0yMEngJ5IFPh8XHz9QoH8IZziTNB3_3Rb3e0Wc6MLInKFER8Vog6C dFUY9HbFQ5cD5PhiMHVBgUOMyzHcqwbyxGnFmB.bzqOPUxJeqiELKMeVUzZM fuX20YYD7znKpbIJ4PLxidqrcBQTwjcBwWDw.e9B7jGVwxZbUjnALUdNDNk6 O0SZeuA2cjrNZ X-Yahoo-SMTP: Xr6qjFWswBAEmd20sAvB4Q3keqXvXsIH9TjJ Received: from mail-vb0-f54.google.com (moonlightakkiy@209.85.212.54 with plain) by smtp224.mail.gq1.yahoo.com with SMTP; 06 Oct 2012 12:00:22 -0700 PDT Received: by mail-vb0-f54.google.com with SMTP id v11so3815479vbm.13 for ; Sat, 06 Oct 2012 12:00:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.23.100 with SMTP id l4mr7392201vef.46.1349550021326; Sat, 06 Oct 2012 12:00:21 -0700 (PDT) Received: by 10.58.182.72 with HTTP; Sat, 6 Oct 2012 12:00:21 -0700 (PDT) Date: Sat, 6 Oct 2012 13:00:21 -0600 Message-ID: From: PseudoCylon To: Adrian Chadd , freebsd-wireless@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: Re: [RFC] How the TX path works; I'd like to fix this before 10.x X-BeenThere: freebsd-wireless@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussions of 802.11 stack, tools device driver development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2012 19:00:28 -0000 > ------------------------------ > > Message: 4 > Date: Fri, 5 Oct 2012 23:44:10 -0700 > From: Adrian Chadd > Subject: [RFC] How the TX path works; I'd like to fix this before 10.x > To: freebsd-wireless@freebsd.org > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > Before I continue - if_transmit() is not the answer. if_transmit() > guarantess _no_ serialisation at all. So we'd stlil have to stuff > things into queues. And ensure only one thing is running at once. As far as I know, if_transmit/if_start are options for drivers, not for the serialisation. if_transmit to use device specific queue if_start to use if_snd queue i.e. http://fxr.watson.org/fxr/source/dev/e1000/if_em.c#L2941 > > I could wrap a big VAP TX lock around ieee80211_output() and > ieee80211_start(), ensuring they don't run over each other. But > long-held locks need to go away and die. yes, even the ones in the > ath(4) driver that I've introduced. They're there because of a lack of > synchronisation and queue design. A lot of the gige/10ge drivers use > long-held locks.. sigh. > > I could create a net80211 TX tasklet for each vap (or ic, _maybe_) > which serialises the TX path. ieee80211_start() would just schedule > the tasklet to run. That would serialise all of the parallel TX entry > points and solve a bunch of the subtle sequence number and other TX > state races that are occuring. That doesn't solve the ic_raw_xmit() or > ieee80211_output() problem, as both of those can also do TX 802.11 > encapsulation and kick off parts of the state machine. I prefer taskqueue over new lock. I have had enough of LORs already. We just need to decide where/how to funnel all tx entries. Currently, pseudo devices (i.e. if_bridge) \ IP stack --> ieee80211 stack --> driver ic_raw_xmit / so we need to ensure serialization (by lock or taskqueue) at 2 points, 1) between upper layer and ieee80211, 2) driver and ieee80211. If we solve the raw_xmit problem, there will be only 1 point to take care. An idea Guarantee only one thread is running at any moment. So, first queued first dequeued and tx'd without a lock. if_start() { if (sc_task == NOT_RUNNING) wakeup(sc_txtask); } if_txtask() { for (;sc_txtask_exit != DONE;) { IFQ_DRV_DEQUEUE(); if (m == NULL) { sc_txtask = NOT_RUNNING; tsleep(); sc_txtask = RUNNING; } else tx(); } } tx_callback() { ... if (sc_task == NOT_RUNNING) wakeup(sc_txtask); } iv_vap_create() { ... taskqueue_enqueue(sc_taskqueue); } iv_vap_delete() { ... sc_txtask_exit = DONE; if (sc_task == NOT_RUNNING) wakeup(sc_txtask); } > > It doesn't solve the same issues with the drivers .Yes, even if we > converted them to use if_transmit(). iwn(4) solves this by just having > a big IWN_LOCK() but it releases it when doing anything stack related. > I'm not sure if it holds it across the whole TX path through > iwn_start(). In any case, it's a big lock. ath(4) can and does have > multiple ath_start() instances running in multiple kernel threads, > fighting to dequeue frames from the ifnet. This still can cause issues > with non-aggregate sequence number and CCMP IV counter allocation. > Sigh. > > God even knows what to do about USB devices in all of this. > Similar to other drivers, i.e. iwn(4). The difference is USB drivers create per-USB-pipe queue in addition to if_snd queue. So that, tx path look like if_transmit -> queue -> if_start -> dequeue -> driver_tx -> usb_queue -> usb_stack It seems redundant. I'd like to change that to (for run(4)) if_transmit -> driver_tx -> usb_queue -> usb_stack AK