From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 7 12:15:34 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75FDD106566B for ; Tue, 7 Jun 2011 12:15:34 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2CFD38FC08 for ; Tue, 7 Jun 2011 12:15:34 +0000 (UTC) Received: by vxc34 with SMTP id 34so4986362vxc.13 for ; Tue, 07 Jun 2011 05:15:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=aGJ0+/a5cRckLE6I72sy816TLmuUY1FytzyL1Xy+e7g=; b=G0NDG1eoXE1aCiXalrGSZXLZrFBlwhn7s2hzxDjDYp/M56daMCDNqqcajpRMJwRfuI OQpJhYjpMu+I5wMOV12TeL4oab4ZwyGK33sSpRXYu8ViMr0Td0Lny+Jc43ysCOtgn2X1 yD13+dw0DdL+IpKec21Rz6ic2KacIvhZHXctI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=J9gA2pM0ewHJ71gxY/l6q4HYwa/V0ewrwhh+F9AFThnH5jfA8ubz+w3aRXcZU9gilL YCvgqmCoH6p1zKhfYARKGhfChSfsqZCGN4CfzdOPhV0esRVntewoaDme/PfzVSwUzqU3 7E5kz7jOSOHzs5I6HkfBLEqK1FDJWV+1YeCbM= MIME-Version: 1.0 Received: by 10.52.177.234 with SMTP id ct10mr5383646vdc.2.1307447516943; Tue, 07 Jun 2011 04:51:56 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.52.187.74 with HTTP; Tue, 7 Jun 2011 04:51:56 -0700 (PDT) In-Reply-To: References: Date: Tue, 7 Jun 2011 13:51:56 +0200 X-Google-Sender-Auth: I-iQZjUd_oolg_CNn8ERImaG6ys Message-ID: From: "K. Macy" To: grarpamp Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, freebsd-net@freebsd.org Subject: Re: FreeBSD I/OAT (QuickData now?) driver X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jun 2011 12:15:34 -0000 All 10GigE NICs and some newer 10 GigE NICs have multiple hardware queues with a separate MSI-x vector per queue, where each vector is directed to a different CPU. The current operating model is to have a separate interrupt thread per vector. This obviously gets bogged down if one has multiple cards as the interrupt threads end up requiring the scheduler to distribute work fairly between cards as multiple threads will end up running on the same CPUs. Nokia had a reasonable interface for coping with this that was reminiscent of NAPI whereby cooperative sharing between interfaces was provided by having a single taskqueue thread per-core and the cards would queue tasks (which would be re-queued if more than a certain amount of work were required) as interrupts were delivered. There has been talk off and on of porting this "net_task" interface to freebsd. None of this addresses PF_RING's facility for pushing packets in to userland - but presumably Rizzo's netmap work addresses those in need of that sufficiently. Cheers, Kip On Tue, Jun 7, 2011 at 4:13 AM, grarpamp wrote: > Is this work part of what's needed to enable the FreeBSD > equivalent of TNAPI? > > I know we've got polling. And probably MSI-X in a couple drivers. > Pretty sure there is still one CPU doing the interrupt work? > And none of the multiple queue thread spreading tech exists? > > http://www.ntop.org/blog > http://www.ntop.org/TNAPI.html > TNAPI attempts to solve the following problems: > =A0 =A0* Distribute the traffic across cores (i.e. the more core the more > scalable is your networking application) for improving scalability. > =A0 =A0* Poll packets simultaneously from each RX queue (contraty to > sequential NAPI polling) for fetching packets as fast as possible > hence improve performance. > =A0 =A0* Through PF_RING, expose the RX queues to the userland so that > the application can spawn one thread per queue hence avoid using > semaphores at all. > TNAPI achieves all this by starting one thread per RX queue. Received > packets are then pushed to PF_RING (if available) or through the > standard Linux stack. However in order to fully exploit this > technology it is necessary to use PF_RING as it provides a straight > packet path from kernel to userland. Furthermore it allows to create a > virtual ethernet card per RX queue. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >