From owner-freebsd-net@FreeBSD.ORG Thu Feb 13 18:36:25 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 92A97AB7 for ; Thu, 13 Feb 2014 18:36:25 +0000 (UTC) Received: from mail-qc0-x22b.google.com (mail-qc0-x22b.google.com [IPv6:2607:f8b0:400d:c01::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 482CB133E for ; Thu, 13 Feb 2014 18:36:25 +0000 (UTC) Received: by mail-qc0-f171.google.com with SMTP id n7so18409884qcx.2 for ; Thu, 13 Feb 2014 10:36:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=2n3lZv2ob4axOl1Z0ZQ52rl37trNUOip+2MuO5nKQ50=; b=koq5lRoLwp3BruudrDMWbiCrDahj8FmINjiLLSwDYH/euuroRy8ZNB2E/h9Deqh6Kf tYgrCyOo69o+Ug3e0X+6mNdlzlEj+IBDnsf18z4hb29dqPJGgeDPh433CQcI5r+hiMgF 6DFvyNa5kYGDVDI3nNwQHjULuE1xIeMiJqnr5r73mwkBQ1mdN63Gz98IbOac9lO7oqtm Rf5p5H6WGQaS1mZLGQ75BNtbSg5ycXqTOiZ3TH42RAm2NHXQJNsLjL8PnvSX/vuUILyU JFm6P9mktKcJD/1myK5hZVVBfc9jfbn3yhh/l3N4qYVlidm7JAAIaZrJz06ussnPeNE9 iPOQ== MIME-Version: 1.0 X-Received: by 10.140.98.135 with SMTP id o7mr4859294qge.102.1392316584449; Thu, 13 Feb 2014 10:36:24 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.16.10 with HTTP; Thu, 13 Feb 2014 10:36:24 -0800 (PST) In-Reply-To: <21245.2981.568824.19800@hergotha.csail.mit.edu> References: <21216.22944.314697.179039@hergotha.csail.mit.edu> <201402111348.52135.jhb@freebsd.org> <201402121446.19278.jhb@freebsd.org> <21244.20212.423983.960018@hergotha.csail.mit.edu> <20140213075651.GY34851@funkthat.com> <21245.1163.754141.154430@hergotha.csail.mit.edu> <20140213180736.GA34851@funkthat.com> <21245.2981.568824.19800@hergotha.csail.mit.edu> Date: Thu, 13 Feb 2014 10:36:24 -0800 X-Google-Sender-Auth: xduvhYHEYMnWejHvtM5PRNzuOQk Message-ID: Subject: Re: Use of contiguous physical memory in cxgbe driver From: Adrian Chadd To: Garrett Wollman Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Net , John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Feb 2014 18:36:25 -0000 On 13 February 2014 10:15, Garrett Wollman wrote: > < said: > >>> [I wrote:] >>> It's very easy to get to that state if the driver insists on getting >>> three physically contiguous pages (which is what it takes to allocate >>> a single 9k cluster) > >> Well, if you're using a cheap NIC that can't do scatter/gather DMA, >> then you get what you pay for... > > No, I'm using expensive 10G NICs, like the Chelsio ones mentioned in > the subject header. They have no trouble doing s/g (subject to some > reasonable limits on chain length). The problem is when the drivers > insist on physical contiguity anyway. > > I finally gave up and hacked the cxgbe driver never to use anything > longer than a physical page. Have you tried it at 40G? ISTR np@ commenting on there being benefits to larger (up to 64K) allocations for the 40G T5 NICs that do packet -> buffer packing on receive side. -a