From owner-freebsd-net@FreeBSD.ORG Fri May 14 15:18:38 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F8261065673; Fri, 14 May 2010 15:18:38 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 360AB8FC15; Fri, 14 May 2010 15:18:37 +0000 (UTC) Received: by gyh20 with SMTP id 20so1516731gyh.13 for ; Fri, 14 May 2010 08:18:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=ZJmkRaUGSWNAOue41Aq+/ib7rODr3nrVpztvpGlNPgE=; b=Ou8l2DMizPpyp2L0B3BFrt8eSi9sR36qCpfuCfjHBnD/8HuzzYmK349cOpN5pfg6nX mj9YbQl4KcoJnXx3UwidmszOjPaHTAp/2WZ2Ib+X7JC1uFrL9UswyX2HTFxRAxj7zfoJ OzgKeIM4M/uClgsBM1Vf3vkYHevTKY/6awyQA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=NaYpCVpB7tBWMPuklzuYqHUyOhzNKJZi70jEWpX8Dc66lXqw5R345GopzTrvgt5AP8 4dTXnMRILh1ojPl/k0+MydBg8FbrpTa8f6MZImECxL8Ru9uClNc0zpV5ZbmBAy4WQtsD h/fXZoK7NDCAtFocB9erajUfsXcx+nSML8+6A= MIME-Version: 1.0 Received: by 10.101.203.9 with SMTP id f9mr1446280anq.208.1273850314273; Fri, 14 May 2010 08:18:34 -0700 (PDT) Received: by 10.100.58.2 with HTTP; Fri, 14 May 2010 08:18:33 -0700 (PDT) In-Reply-To: <4BED5929.5020302@cs.duke.edu> References: <4BE52856.3000601@unsane.co.uk> <1273323582.3304.31.camel@efe> <20100511135103.GA29403@grapeape2.cs.duke.edu> <4BED5929.5020302@cs.duke.edu> Date: Fri, 14 May 2010 11:18:33 -0400 Message-ID: From: Alexander Sack To: Andrew Gallatin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Murat Balaban , freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Intel 10Gb X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 May 2010 15:18:38 -0000 On Fri, May 14, 2010 at 10:07 AM, Andrew Gallatin wr= ote: > Alexander Sack wrote: > <...> >>> Using this driver/firmware combo, we can receive minimal packets at >>> line rate (14.8Mpps) to userspace. =A0You can even access this using a >>> libpcap interface. =A0The trick is that the fast paths are OS-bypass, >>> and don't suffer from OS overheads, like lock contention. =A0See >>> http://www.myri.com/scs/SNF/doc/index.html for details. >> >> But your timestamps will be atrocious at 10G speeds. =A0Myricom doesn't >> timestamp packets AFAIK. =A0If you want reliable timestamps you need to >> look at companies like Endace, Napatech, etc. > > I see your old help ticket in our system. =A0Yes, our timestamping > is not as good as a dedicated capture card with a GPS reference, > but it is good enough for most people. I was told btw that it doesn't timestamp at ALL. I am assuming NOW that is incorrect. Define *most* people. I am not knocking the Myricom card. In fact I so wish you guys would just add the ability to latch to a 1PPS for timestamping and it would be perfect. We use I think an older version of the card internally for replay. Its a great multi-purpose card. However with IPG at 10G in the nanoseconds, anyone trying to do OWDs or RTT will find it difficult compared to an Endace or Napatech card. Btw, I was referring to bpf(4) specifically, so please don't take my comments as a knock against it. >> PS I am not sure but Intel also supports writing packets directly in >> cache (yet I thought the 82599 driver actually does a prefetch anyway >> which had me confused on why that helps) > > You're talking about DCA. =A0We support DCA as well (and I suspect some > other 10G NICs do to). =A0There are a few barriers to using DCA on > FreeBSD, not least of which is that FreeBSD doesn't currently have the > infrastructure to support it (no IOATDMA or DCA drivers). Right. > DCA is also problematic because support from system/motherboard > vendors is very spotty. =A0The vendor must provide the correct tag table > in BIOS such that the tags match the CPU/core numbering in the system. > Many motherboard vendors don't bother with this, and you cannot enable > DCA on a lot of systems, even though the underlying chipset supports > DCA. =A0I've done hacks to force-enable it in the past, with mixed > results. The problem is that DCA depends on having the correct tag > table, so that packets can be prefetched into the correct CPU's cache. > If the tag table is incorrect, DCA is a big pessimization, because it > blows the cache in other CPUs. Right. > That said, I would *love* it if FreeBSD grew ioatdma/dca support. > Jack, does Intel have any interest in porting DCA support to FreeBSD? Question for Jack or Drew, what DOES FreeBSD have to do to support DCA? I thought DCA was something you just enable on the NIC chipset and if the system is IOATDMA aware, it just works. Is that not right (assuming cache tags are correct and accessible)? i.e. I thought this was hardware black magic than anything specific the OS has to do. -aps