From owner-freebsd-net@FreeBSD.ORG Fri May 14 16:22:30 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78AE01065673; Fri, 14 May 2010 16:22:30 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.freebsd.org (Postfix) with ESMTP id 3608A8FC14; Fri, 14 May 2010 16:22:30 +0000 (UTC) Received: from [172.31.193.10] (rrcs-98-101-145-84.midsouth.biz.rr.com [98.101.145.84]) (authenticated bits=0) by duke.cs.duke.edu (8.14.2/8.14.2) with ESMTP id o4EGMKsQ001920 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 14 May 2010 12:22:20 -0400 (EDT) X-DKIM: Sendmail DKIM Filter v2.8.3 duke.cs.duke.edu o4EGMKsQ001920 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail; t=1273854144; bh=wr56XOlwsERw69EPXpJL+Sjo3Vw4rfhjkik0vEY2SV4=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=wBBZ09/PqOw/pz2+c+vAcbaDviFXC4fUKEDsSk27EF0yN6wi0G+ybNSHbXO86Kpkr 5C+jGxW9PdIBHr0k0YrZPrmPETMJkgVHmDrim15zKgFKBuKxIByB8nDnBcESseln8x 5gdeUL60JbpVqmVAit1RItnDKPdhDzXO80qARunc= Message-ID: <4BED78B5.8000906@cs.duke.edu> Date: Fri, 14 May 2010 12:22:13 -0400 From: Andrew Gallatin User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: Alexander Sack References: <4BE52856.3000601@unsane.co.uk> <1273323582.3304.31.camel@efe> <20100511135103.GA29403@grapeape2.cs.duke.edu> <4BED5929.5020302@cs.duke.edu> <4BED6F1B.7070602@cs.duke.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Murat Balaban , freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Intel 10Gb X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 May 2010 16:22:30 -0000 Alexander Sack wrote: >> To use DCA you need: >> >> - A DCA driver to talk to the IOATDMA/DCA pcie device, and obtain the tag >> table >> - An interface that a client device (eg, NIC driver) can use to obtain >> either the tag table, or at least the correct tag for the CPU >> that the interrupt handler is bound to. The basic support in >> a NIC driver boils down to something like: >> >> nic_interrupt_handler() >> { >> if (sc->dca.enabled && (curcpu != sc->dca.last_cpu)) { >> sc->dca.last_cpu = curcpu; >> tag = dca_get_tag(curcpu); >> WRITE_REG(sc, DCA_TAG, tag); >> } >> } > > Drew, at least in the Intel documentation, it seems the NIC uses the > LAPIC id to tell the PCIe TLPs where to put inbound NIC I/O (in the > TLP the DCA info is stored) to the appropriate core's cache. i.e. the > heuristic you gave above is more granular than what I think Intel The pseudo-code above was intended to be the MSI-X interrupt handler for a single queue, not some dispatcher for multiple queues. Sorry that wasn't clear. So yes, the DCA tag value may be different per queue. > does. I could be wrong, maybe Jack can chime in and correct me. But > it seems with Intel chipsets it is a per queue parameter which allows > you to bind a core cache's to a queue via DCA. The added piece to > this for at least bpf(4) consumers is to have bpf(4) subscribe to > these queues AND to allow an interface for libpcap applications to > know where what queue is on what core and THEN bind to it. Yes, everything associated with a queue must be bound to the same core (or at least to cores which share a cache). Drew