From owner-freebsd-drivers@FreeBSD.ORG Mon Sep 19 18:10:20 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E76C106566C for ; Mon, 19 Sep 2011 18:10:20 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-ew0-f48.google.com (mail-ew0-f48.google.com [209.85.215.48]) by mx1.freebsd.org (Postfix) with ESMTP id AAE888FC19 for ; Mon, 19 Sep 2011 18:10:18 +0000 (UTC) Received: by ewy23 with SMTP id 23so864643ewy.7 for ; Mon, 19 Sep 2011 11:10:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=BL2HsWz0ub7xCaogZuokjo8yrazvtOtB7V9MUA09aE8=; b=gpwF3e7TpW2JUXiXQfCSDCzlPqSBg8mx4E7wXAvOL75yOSX1G+Y0NG8w6mNrCnGNQI LrjC7VdJ1Eci0oSUDBSdqgVfZc+PEgkr4Rp0B4C6gXaX5lorXuF/3R8VoWcTkaUC62in DeDExPET/2jl05i2QiSpf7utA8WJQ/kk216wc= MIME-Version: 1.0 Received: by 10.52.96.166 with SMTP id dt6mr2550023vdb.345.1316454224550; Mon, 19 Sep 2011 10:43:44 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.52.113.202 with HTTP; Mon, 19 Sep 2011 10:43:44 -0700 (PDT) In-Reply-To: References: <201109190813.37817.jhb@freebsd.org> Date: Mon, 19 Sep 2011 19:43:44 +0200 X-Google-Sender-Auth: O9Ou1KTry7WwWcirPzK6C5qIVhc Message-ID: From: "K. Macy" To: freebsd-drivers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Fwd: Multiple TX queue support in nic driver X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2011 18:10:20 -0000 forwarding response to the list > ---------- Forwarded message ---------- > From:=A0Naresh > To:=A0"freebsd-drivers@freebsd.org" > Date:=A0Thu, 15 Sep 2011 20:26:16 +0530 > Subject:=A0Multiple TX queue support in nic driver > Hi All, > > I am working on a 10G Ethernet driver. =A0I have a requirement to support= multiple TX queues. I looked into IXGBE and CXGB drivers for reference, I = have some questions regarding the implementation. > > 1) For supporting multiple TXQs, I see the above mentioned drivers create= a ring buffer in the driver for each hardware TX queue, and queue the pack= ets from stack to the ring buffer before queueing on the hardware queue. I = don't understand why to create a ring buffer for supporting multiple TXQS. = There might be a very good reason for them to do that way, but I am unable = to get that. The software queues are only used if the txq lock is already held by another thread. What alternative are you thinking of? Waiting for the txq lock would be much slower than a simple ring buffer enqueue. A global queue where threads randomly pulled off and put on an available txq would cause packet reording within individual flows. > Is it ok to queue the packets to hardware queue directly based on the flo= wid supplied in the mbuf =A0packet header in if_transmit routine ? Yes. This is what those drivers already do if the txq lock is not held. > 2) Regarding flowid, for the first TX packet the flowid is not set and th= e packet goes out on the default TX queue and when the corresponding RX res= ponse comes to hardware it queues =A0 the packet to appropriate RXQ based o= n the rss hash. The driver then before posting RX packet to stack it should= set the flowid in packet to RXQ index. > After that the TX packets from stack for that TCP connection will come wi= th flowid supplied by us. Is this understanding correct ? Yes. For using multiple queues with UDP unidirectionally one needs to either use the flowtable or change the inpcb creation routine. Cheers