From owner-freebsd-net@FreeBSD.ORG Wed Oct 17 01:26:41 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F410B436; Wed, 17 Oct 2012 01:26:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by mx1.freebsd.org (Postfix) with ESMTP id A73438FC0C; Wed, 17 Oct 2012 01:26:40 +0000 (UTC) Received: by mail-pa0-f54.google.com with SMTP id bi1so6983971pad.13 for ; Tue, 16 Oct 2012 18:26:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=bmRWDQgAacYcgW3VizvX9o+evO04s1b8d0cJgL5So04=; b=MHVlLiuSQdqTM3ZE8/tTznTHRf0zABwVBNh9LhNoOozPYrugsgabD9PpyZ4SfsO4ZZ ebU5mpNi/VeeHJIAvDHeln4VGQCsEtNYSMr1Eh5DWF0GD4LSJtpBAuPaljYmFIjKZLdc Lj+Nzb4fmdKVvfg5Nz8XJn/lo5jDCfKGsQY61DyvqlN7Xf764jvfD9uz1+3qSJdNvHlg HBnts6BlDoxG/p7qecU2S0r5XRjlCWFnc0i0qvBwMUXASpiN+MvDQDypaplcDHuPKMW/ PgiWMqvs5TWt0HQSTjAjDZ2MSSde6dSdieLTHSzNMG92+wzEHK0YqC6C0TAOC7vApwUi rfrA== MIME-Version: 1.0 Received: by 10.66.86.129 with SMTP id p1mr6638350paz.39.1350437200422; Tue, 16 Oct 2012 18:26:40 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.146.233 with HTTP; Tue, 16 Oct 2012 18:26:40 -0700 (PDT) In-Reply-To: <201210160838.17741.jhb@freebsd.org> References: <5079A9A1.4070403@FreeBSD.org> <201210151414.27318.jhb@freebsd.org> <201210160838.17741.jhb@freebsd.org> Date: Tue, 16 Oct 2012 18:26:40 -0700 X-Google-Sender-Auth: GoMxUKHM6DlvJHVOZ6ge9-Pu1BU Message-ID: Subject: Re: ixgbe & if_igb RX ring locking From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: "Alexander V. Chernikov" , freebsd-net@freebsd.org, Jack Vogel , net@freebsd.org, Luigi Rizzo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 01:26:41 -0000 On 16 October 2012 05:38, John Baldwin wrote: > I don't follow how this is related to this thread at all (which has more to do > with ixgbe scheduling duplicate work). However, is your issue that the stack > locks (e.g. socket and protocol layer locks) are held across > if_start/if_transmit? It's a comment on the larger scale architectural problem. Since if_transmit and if_start are called from multiple thread contexts, the current ways drivers implement this are: * support direct dispatch to hardware, but wrap the whole sending process in one enormous lock, to prevent packet reordering issues; or * drop TX and TX completion into a TX taskqueue (or multiple, one per hardware send queue) and push frames into that taskqueue via some queue and then wake said taskqueue up; or * some bastardised version of both. For the intel drivers, the locks are held for a (potentially) very long time. Both igb and ixgb both hold the locks for the entirety of the TX process. It's not protecting something like a queue operation, it's effectively serialising the entirety of the TX and TX completion process. That works ok-ish for ethernet drivers which are "send and ignore", but for wireless drivers where the stack implements a lot more state, it really does quite suck. And since wireless drivers have a top level idea of sequence and encryption (ie, it's not per-TCP stream, it's across multiple sending streams to a given node), I can't model the locking and serialisation on what the TCP/UDP code does. I wish we had a better way of implementing "serialisation without long, long held locks" but short of stuffing everything into a taskqueue and only locking the send queue involved, I can't really think of anything. Adrian