From owner-freebsd-net@freebsd.org Fri Jul 27 22:18:54 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E82CB1058ABF for ; Fri, 27 Jul 2018 22:18:53 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 56D908BED6 for ; Fri, 27 Jul 2018 22:18:52 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id w6RMIid0004282 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Jul 2018 15:18:44 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id w6RMIiiw004281; Fri, 27 Jul 2018 15:18:44 -0700 (PDT) (envelope-from jmg) Date: Fri, 27 Jul 2018 15:18:43 -0700 From: John-Mark Gurney To: Ryan Moeller Cc: freebsd-net@freebsd.org Subject: Re: 9k jumbo clusters Message-ID: <20180727221843.GZ2884@funkthat.com> Mail-Followup-To: Ryan Moeller , freebsd-net@freebsd.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Fri, 27 Jul 2018 15:18:44 -0700 (PDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2018 22:18:54 -0000 Ryan Moeller wrote this message on Fri, Jul 27, 2018 at 12:45 -0700: > There is a long-standing issue with 9k mbuf jumbo clusters in FreeBSD. > For example: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=183381 > https://lists.freebsd.org/pipermail/freebsd-net/2013-March/034890.html > > This comment suggests the 16k pool does not have the fragmentation problem: > https://reviews.freebsd.org/D11560#239462 > I???m curious whether that has been confirmed. > > Is anyone working on the pathological case with 9k jumbo clusters in the > physical memory allocator? There was an interesting discussion started a > few years ago but I???m not sure what ever came of it: > http://docs.freebsd.org/cgi/mid.cgi?21225.20047.947384.390241 > > I have seen some work in the direction of avoiding larger than page size > jumbo clusters in 12-CURRENT. Many existing drivers avoid the 9k cluster > size already. The code for larger cluster sizes in iflib is #ifdef'd out > so it maxes out at the page size jumbo clusters until "CONTIGMALLOC_WORKS" > (apparently it doesn't). > > With all the changes due to iflib, is there any chance some of this will > get MFC'd to address the serious problem that remains in 11-STABLE? > > Otherwise, would it be feasible to disable the use of the 9k cluster pool > in at least some of the popular NIC drivers as a solution for the stable > branches? > > Finally, I have studied some of the driver code in 11-STABLE and posted the > gist of my notes in relation to this problem. If anyone spots a mistake or > has something else to contribute, comments on the gist would be greatly > appreciated! > https://gist.github.com/freqlabs/eba9b755f17a223260246becfbb150a1 Drivers need to be fixed to use 4k pages instead of cluster. I really hope no one is using a card that can't do 4k pages, or if they are, then they should get a real card that can do scatter/gather on 4k pages for jumbo frames.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."