From owner-freebsd-net@FreeBSD.ORG Tue Apr 7 21:48:59 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CF17106566B for ; Tue, 7 Apr 2009 21:48:59 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63904.mail.re1.yahoo.com (web63904.mail.re1.yahoo.com [69.147.97.119]) by mx1.freebsd.org (Postfix) with SMTP id DB67D8FC1D for ; Tue, 7 Apr 2009 21:48:58 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: (qmail 8939 invoked by uid 60001); 7 Apr 2009 21:48:58 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1239140938; bh=edKaXXlheTRCU7a0Hb5pxmKQ2ERAT4MbHjCQEYi7ygs=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=ap2rM8h0M51xEXthOsTANWjnGiPoD+kjw5015F2W3Ns736JanDrpJS0v64p8zJzzfouEuK/eUnyenN3fpupw8zy9OxgnXuwiEhit4lIPLJKcxZnJJ7ITdeL2zEZNzP0aVncU18G1sehJWG+YP15YLpm0SVXp/PM99R/NclQ2uPQ= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=qJbvV67MgqUsc9v7u8Pep/lKk+M4r90MEg4PXMGBHzYdoU273V0Bw3D8mHj0ZWCtDUcYBLrT2AYp8uwz43y7mHk+UGk8fAIJCUzZzJ7QcDyKJwWP6a9u6szqRub9u8spnVUn/MYBxxlvKaQmMLrRD4gplm/06vWOP9ezfajw5fQ=; Message-ID: <409843.2186.qm@web63904.mail.re1.yahoo.com> X-YMail-OSG: dL.jcakVM1k6TmJf3INAATWN1iNL8FVjrttDiroG4bbTPvxKraJAX2hujlMVzNKZxUNKOvi5YSFsmzlHn2QcmQOkIQTU9x_cUPDxLm8RZ.WDuCO9cT65a.4wt5jqbecm.YmLJgT8jA8BZJeqpwLNVmxrcd42e1FXbJrki1TrMc.opHujVQPEHUp1HHDj0Jux1oMbIPmI1qUQj.objoYLcwQb6Ve06xS7JPLjZyVA0tX3WmLeZSmQuK3DfGffl1t2iuouKo8rLuqabS3lIF3yDnfpTftWHtT71oHHY8XCEQw6IzrU3s13nA5YGJAR Received: from [98.242.222.229] by web63904.mail.re1.yahoo.com via HTTP; Tue, 07 Apr 2009 14:48:58 PDT X-Mailer: YahooMailWebService/0.7.289.1 Date: Tue, 7 Apr 2009 14:48:58 -0700 (PDT) From: Barney Cordoba To: Robert Watson , Sepherosa Ziehau In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-net@freebsd.org, Ivan Voras Subject: Re: Advice on a multithreaded netisr patch? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: barney_cordoba@yahoo.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Apr 2009 21:48:59 -0000 --- On Tue, 4/7/09, Sepherosa Ziehau wrote: > From: Sepherosa Ziehau > Subject: Re: Advice on a multithreaded netisr patch? > To: "Robert Watson" > Cc: freebsd-net@freebsd.org, "Ivan Voras" > Date: Tuesday, April 7, 2009, 9:57 AM > On Tue, Apr 7, 2009 at 8:54 PM, Robert Watson > wrote: > > > > On Tue, 7 Apr 2009, Sepherosa Ziehau wrote: > > > >> On Sun, Apr 5, 2009 at 9:34 PM, Ivan Voras > wrote: > >>> > >>> Robert Watson wrote: > >>>> > >>>> On Sun, 5 Apr 2009, Ivan Voras wrote: > >>>> > >>>>> I thought this has something to deal > with NIC moderation (em) but > >>>>> can't really explain it. The bad > performance part (not the jump) is > >>>>> also visible over the loopback > interface. > >>>> > >>>> FYI, if you want high performance, you > really want a card supporting > >>>> multiple input queues -- igb, cxgb, mxge, > etc. if_em-only cards are > >> > >> PCI-E em(4) supports 2 RX queues. 82571/82572 > support 2 TX queues. I have > >> not tested multi-TX queues, but em(4) multi-RX > queues work well in dfly > >> (tested with 82573 and 82571) > > > > You may not have seen, but in FreeBSD 7.x and higher, > we have a new if_igb > > driver to support more recent Intel gigabit devices, > which now probes a few > > of the devices historically associated with if_em. > For example, on one of > > the boxes I use: > > If I understand the code correctly, it only takes 82575 and > 82576; I > don't have the hardware, else I would have already > added dfly support > (with multi rx queues at least, it seems 82576 supports 16 > RX queues > :) Regarding if_igb: 1) Multiple TX queues are not supported. There's some hokey code to test, but it doesn't properly separate flows to the queues. 2) 2 Rx queues don't work, so only 1 and 4 work 3) With 4 queues, it just sucks up CPU under heavy load on 4 cpus. It will blow 4 cpus at a lower load than em will with 1 4) You'll need to fix DMA setup, as it sets the alignment requirement to PAGE_SIZE. I haven't been able to convince Jack that its wrong, not that I've tried very hard since its easy to just fix myself. Barney