From owner-freebsd-net@FreeBSD.ORG Fri May 14 14:07:54 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A107106564A; Fri, 14 May 2010 14:07:54 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.freebsd.org (Postfix) with ESMTP id ED92F8FC0A; Fri, 14 May 2010 14:07:53 +0000 (UTC) Received: from [172.31.193.10] (rrcs-98-101-145-84.midsouth.biz.rr.com [98.101.145.84]) (authenticated bits=0) by duke.cs.duke.edu (8.14.2/8.14.2) with ESMTP id o4EE7jas024681 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 14 May 2010 10:07:45 -0400 (EDT) X-DKIM: Sendmail DKIM Filter v2.8.3 duke.cs.duke.edu o4EE7jas024681 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail; t=1273846067; bh=kXi5tWSmoSa8QcZzXchvHOCewVYqctK7J0WUDkPf6hc=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=sL95rcXK77xDld358fA5oQ7Z5vCe/Xag1v6oR05mOPKvVieK20O94ANwkhWcBvdiH Z3oijdWQeM7pIzFQinoS1Q1NAbxiYxPBqDBuHWBy66KhsC6yciiXA8zDGU6MacKsIv v5H6RMOZNH8N9UtNq7vIab5fbEp4moRxeV0SHnz8= Message-ID: <4BED5929.5020302@cs.duke.edu> Date: Fri, 14 May 2010 10:07:37 -0400 From: Andrew Gallatin User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: Alexander Sack References: <4BE52856.3000601@unsane.co.uk> <1273323582.3304.31.camel@efe> <20100511135103.GA29403@grapeape2.cs.duke.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Murat Balaban , freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Intel 10Gb X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 May 2010 14:07:54 -0000 Alexander Sack wrote: <...> >> Using this driver/firmware combo, we can receive minimal packets at >> line rate (14.8Mpps) to userspace. You can even access this using a >> libpcap interface. The trick is that the fast paths are OS-bypass, >> and don't suffer from OS overheads, like lock contention. See >> http://www.myri.com/scs/SNF/doc/index.html for details. > > But your timestamps will be atrocious at 10G speeds. Myricom doesn't > timestamp packets AFAIK. If you want reliable timestamps you need to > look at companies like Endace, Napatech, etc. I see your old help ticket in our system. Yes, our timestamping is not as good as a dedicated capture card with a GPS reference, but it is good enough for most people. > PS I am not sure but Intel also supports writing packets directly in > cache (yet I thought the 82599 driver actually does a prefetch anyway > which had me confused on why that helps) You're talking about DCA. We support DCA as well (and I suspect some other 10G NICs do to). There are a few barriers to using DCA on FreeBSD, not least of which is that FreeBSD doesn't currently have the infrastructure to support it (no IOATDMA or DCA drivers). DCA is also problematic because support from system/motherboard vendors is very spotty. The vendor must provide the correct tag table in BIOS such that the tags match the CPU/core numbering in the system. Many motherboard vendors don't bother with this, and you cannot enable DCA on a lot of systems, even though the underlying chipset supports DCA. I've done hacks to force-enable it in the past, with mixed results. The problem is that DCA depends on having the correct tag table, so that packets can be prefetched into the correct CPU's cache. If the tag table is incorrect, DCA is a big pessimization, because it blows the cache in other CPUs. That said, I would *love* it if FreeBSD grew ioatdma/dca support. Jack, does Intel have any interest in porting DCA support to FreeBSD? Drew