From owner-freebsd-wireless@FreeBSD.ORG Tue Oct 30 17:39:55 2012 Return-Path: Delivered-To: freebsd-wireless@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 25741F15 for ; Tue, 30 Oct 2012 17:39:55 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by mx1.freebsd.org (Postfix) with ESMTP id D74C78FC17 for ; Tue, 30 Oct 2012 17:39:54 +0000 (UTC) Received: by mail-pa0-f54.google.com with SMTP id bi1so364615pad.13 for ; Tue, 30 Oct 2012 10:39:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=W7FMLaKn2qIVqjYMmukY4ml6VVFzluKEmk4ZIZEgonE=; b=AjoSdvTzDUqzaVyQPn33Z3rC5FU5Teo5aBXxwWugZLFLZiyVVJC6M5pKxUz7/oW4YG tdGlQqunTHprD2y7HSOlTucgbSKWxCp+huYYHo/dTkcluIhi0TNFF3z6qJRNFnfsTCrX IDD3zuS9ZqgGaSSzImwwlPnNHuGukFH+GLpEWKYchmWOspxUZsAB0HQGfWcSl1wDcrKH tATjFJCVXrDMK4DR2WxGrhuUnFHo8rPFPLlWqUgrLTnu+0mWhDrmF6GOYepnDg9QJCXD WCIjBx2xnlHUNQMiOeIcFaEKKqK3YA6jRn4LCuErqQ73kujAO89jx+Kb8qSyLcIcO10V dUZA== MIME-Version: 1.0 Received: by 10.68.229.201 with SMTP id ss9mr104124731pbc.80.1351618794252; Tue, 30 Oct 2012 10:39:54 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.146.233 with HTTP; Tue, 30 Oct 2012 10:39:54 -0700 (PDT) In-Reply-To: References: Date: Tue, 30 Oct 2012 10:39:54 -0700 X-Google-Sender-Auth: fXZA0X0a_UnWzdgrYXBalSgHbG4 Message-ID: Subject: Re: updates: net80211/ath now do AP power save "better" (except ps-poll); I broke performance From: Adrian Chadd To: freebsd-wireless@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-wireless@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussions of 802.11 stack, tools device driver development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Oct 2012 17:39:55 -0000 On 28 October 2012 20:48, Adrian Chadd wrote: > Now for the problem: I broke throughput. Instead of getting > 150Mbit/sec TCP, I now get ~ 100MBit/sec TCP. The culprit is almost > exclusively going to be the TX serialisation. Now, that's easily > tested and I'll do that tomorrow - I can just undo the TX > serialisation and make ath_start() direct dispatch. If this _isn't_ > the case, I'll have to spend whatever time needed to figure it out. > But if it is the case, I'll need to figure out how to serialise TX > without that performance drop. So, if you do update to -HEAD, please > keep that in mind. Yes, it's the TX taskqueue changes. If I go back to direct dispatch, I actually now get 150mbit -> 170mbit TCP throughput. Yes, it's that crazy. So at least for the station-side running iperf, here's the problem: * iperf queues some traffic; * it waits for an ACK before it can queue more; * ath(4) RX interrupt occurs; * RX tasklet gets scheduled - and if nothing else is going on, it runs; * So.. RX frames are handled, and I guess one is an ACK, as it wakes up iperf - so whilst the RX taskqueue is running, the scheduler switches to iperf (maybe it hits a lock that needs waking up? Not sure.) * The iperf thread sends more data - so a whole lot of ath_start() is called, which in the TX taskqueue implementation just schedules the TX tasklet to run; * .. but as the RX tasklet is running - it can't run. * Then the rest of the RX tasklet runs to completion; * Once that's done, TX occurs. That extra latency is costing like, half the performance. Now, the new if_bridge code is likely making that worse in the bridging path - since it's now direct-dispatching from wifi to the arge interface, if arge blocks at all, RX is going to stall; which means any traffic the other way is going to have to wait. Whereas before it'd just populate the bridge ifnet->if_snd queue. It's very likely more complicated than that, but I can totally see that happening. So! Given this is the behaviour of the IP/TCP stack, I'm not entirely sure what to do next. * I can go back to direct-dispatch, but that introduces lots of synchronisation issues again, especially with preemption and SMP. * I could go to "only schedule ath_start() if it's not already running" but there's a small race window with that - specifically, if you do this: ath_start() if (! running) { set run=1; run; set run=0; } That seems innocuous - but if you have a second thread that gets run just as set run=0 is about to be run, the second thread won't run - but the first thread will terminate. So it's not _that- easy to do. * I could just go the linux path and create a wifi RX and TX lock - and hold that across any RX or TX respectively. Thoughts are welcome. :-) Thanks, Adrian