From owner-freebsd-net@freebsd.org Fri Jul 27 19:45:09 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D89761054D0A for ; Fri, 27 Jul 2018 19:45:08 +0000 (UTC) (envelope-from ryan@ixsystems.com) Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 556AC84BD7 for ; Fri, 27 Jul 2018 19:45:08 +0000 (UTC) (envelope-from ryan@ixsystems.com) Received: by mail-pf1-x42d.google.com with SMTP id x17-v6so2069629pfh.5 for ; Fri, 27 Jul 2018 12:45:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems-com.20150623.gappssmtp.com; s=20150623; h=from:content-transfer-encoding:mime-version:subject:message-id:date :to; bh=Hi3My3h3xeF7rxUw/9EbkxJa7j2O25CTDAATgEibcZ0=; b=uvw3uWB7wu41j2nMAnEBTB0jGHBK1hkaM+WPEcFCe/YrNO5Aue3kbjA/1A7DvbIJ9/ +rXuIjPR5+rKVOIpAd3JIMAHiBvohBSCA15QaJquuG/o17AnlAVwRQPjy5p0PI2QhiAd XB/b6FAjF/AmkElejoOAc20ED7NTg9i/2yVMTsctzlpA9MULoNUZYVjZgyTiEMCotEav 8y96KXp4TDQNTL8OUBMqO+4wRtVL0jfidw3W1izSptUbsy+TADgSJcn2hvF82lt3WGnf P2tLJBl32hCzkZ4S9AWxJYrOxcsHhuQKZa2WafrjclExsnBu5j+ZXKq6B3Eadl9yk/a4 Fbnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=Hi3My3h3xeF7rxUw/9EbkxJa7j2O25CTDAATgEibcZ0=; b=TpRgReonBaJc3fhRvP8grPK+L2dcWZyKRCAL7cYUmkRXAN4O3a9ZKGLqkVPS0a/KdB qcdlXKiVouY4RsJsGF08E3ipRz63Z3QR7uNepzDgh+P2j6QzB2Lo1Vs9PzHkI19KJgOr n5PRgYPjDzK6T+4z6AKP8e88zinLzcHJgWd/lkkcObUHsZHhjl4Zs44maBA4jEW2fx9o 42YNxLBVXhUpgq/Rq8tTqkBc2QtBYyuwE4uP4rU306cFqu0CP45FGkcZwsTYo5BPLlyb PA8POrEFPEl4wGu2K4VGrcpjJ0Ub0ZcaBs+EFANj2wKPTgsCyLHMtj4aRK83A66H9nEY wsaA== X-Gm-Message-State: AOUpUlHC0KOUk001IEKPgfef/0QqAJBkKSJHCOx2Syte93h6TmGI9PjJ OGerj9u9llD6ms+AnUGV1OS0KZcUD63rKyXDsYXctNmTA49rXslhO9j9u+yvatOo1DcKJ6QHfW9 LhGT+LdkxV1sf1SbEDYVnycMO9cBWbaaGuQuyCsNXdwAbS+UR47lcjyOwU2tmh2yOCw== X-Google-Smtp-Source: AAOMgpeIWoA4Y0qLuU7LA4abUzOmRol9ugHaLKS9xptglNLEqkF88QvL/aLmdaGV88AOaJEhMALjxg== X-Received: by 2002:a63:c902:: with SMTP id o2-v6mr7349142pgg.118.1532720706977; Fri, 27 Jul 2018 12:45:06 -0700 (PDT) Received: from [10.0.0.116] (c-67-160-253-209.hsd1.ca.comcast.net. [67.160.253.209]) by smtp.gmail.com with ESMTPSA id u9-v6sm15137206pfi.104.2018.07.27.12.45.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Jul 2018 12:45:06 -0700 (PDT) From: Ryan Moeller Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: 9k jumbo clusters Message-Id: Date: Fri, 27 Jul 2018 12:45:05 -0700 To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2018 19:45:09 -0000 Hello, There is a long-standing issue with 9k mbuf jumbo clusters in FreeBSD. For example: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D183381 https://lists.freebsd.org/pipermail/freebsd-net/2013-March/034890.html This comment suggests the 16k pool does not have the fragmentation = problem: https://reviews.freebsd.org/D11560#239462 I=E2=80=99m curious whether that has been confirmed. Is anyone working on the pathological case with 9k jumbo clusters in the physical memory allocator? There was an interesting discussion started = a few years ago but I=E2=80=99m not sure what ever came of it: http://docs.freebsd.org/cgi/mid.cgi?21225.20047.947384.390241 I have seen some work in the direction of avoiding larger than page size jumbo clusters in 12-CURRENT. Many existing drivers avoid the 9k = cluster size already. The code for larger cluster sizes in iflib is #ifdef'd = out so it maxes out at the page size jumbo clusters until = "CONTIGMALLOC_WORKS" (apparently it doesn't). With all the changes due to iflib, is there any chance some of this will get MFC'd to address the serious problem that remains in 11-STABLE? Otherwise, would it be feasible to disable the use of the 9k cluster = pool in at least some of the popular NIC drivers as a solution for the stable branches? Finally, I have studied some of the driver code in 11-STABLE and posted = the gist of my notes in relation to this problem. If anyone spots a mistake = or has something else to contribute, comments on the gist would be greatly appreciated! https://gist.github.com/freqlabs/eba9b755f17a223260246becfbb150a1 Thanks in advance! Ryan=