From owner-freebsd-net@FreeBSD.ORG  Thu Feb 13 18:15:11 2014
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 5B4AA346
 for <freebsd-net@freebsd.org>; Thu, 13 Feb 2014 18:15:11 +0000 (UTC)
Received: from hergotha.csail.mit.edu
 (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id E5B661169
 for <freebsd-net@freebsd.org>; Thu, 13 Feb 2014 18:15:10 +0000 (UTC)
Received: from hergotha.csail.mit.edu (localhost [127.0.0.1])
 by hergotha.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s1DIF1nA047692;
 Thu, 13 Feb 2014 13:15:02 -0500 (EST)
 (envelope-from wollman@hergotha.csail.mit.edu)
Received: (from wollman@localhost)
 by hergotha.csail.mit.edu (8.14.7/8.14.4/Submit) id s1DIF1eO047689;
 Thu, 13 Feb 2014 13:15:01 -0500 (EST) (envelope-from wollman)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <21245.2981.568824.19800@hergotha.csail.mit.edu>
Date: Thu, 13 Feb 2014 13:15:01 -0500
From: Garrett Wollman <wollman@bimajority.org>
To: John-Mark Gurney <jmg@funkthat.com>
Subject: Re: Use of contiguous physical memory in cxgbe driver
In-Reply-To: <20140213180736.GA34851@funkthat.com>
References: <21216.22944.314697.179039@hergotha.csail.mit.edu>
 <201402111348.52135.jhb@freebsd.org>
 <CAJ-VmonCdNQPUCQwm0OhqQ3Kt_7x6-g-JwGVZQfzWTgrDYfmqw@mail.gmail.com>
 <201402121446.19278.jhb@freebsd.org>
 <21244.20212.423983.960018@hergotha.csail.mit.edu>
 <20140213075651.GY34851@funkthat.com>
 <21245.1163.754141.154430@hergotha.csail.mit.edu>
 <20140213180736.GA34851@funkthat.com>
X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3
 (hergotha.csail.mit.edu [127.0.0.1]); Thu, 13 Feb 2014 13:15:02 -0500 (EST)
X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED
 autolearn=disabled version=3.3.2
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on
 hergotha.csail.mit.edu
Cc: FreeBSD Net <freebsd-net@freebsd.org>
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 13 Feb 2014 18:15:11 -0000

<<On Thu, 13 Feb 2014 10:07:36 -0800, John-Mark Gurney <jmg@funkthat.com> said:

>> [I wrote:]
>> It's very easy to get to that state if the driver insists on getting
>> three physically contiguous pages (which is what it takes to allocate
>> a single 9k cluster)

> Well, if you're using a cheap NIC that can't do scatter/gather DMA,
> then you get what you pay for...

No, I'm using expensive 10G NICs, like the Chelsio ones mentioned in
the subject header.  They have no trouble doing s/g (subject to some
reasonable limits on chain length).  The problem is when the drivers
insist on physical contiguity anyway.

I finally gave up and hacked the cxgbe driver never to use anything
longer than a physical page.

-GAWollman