From owner-svn-src-head@FreeBSD.ORG Wed Aug 21 19:41:42 2013 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7DF59F58 for ; Wed, 21 Aug 2013 19:41:42 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm23-vm5.bullet.mail.ne1.yahoo.com (nm23-vm5.bullet.mail.ne1.yahoo.com [98.138.91.245]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3C054258F for ; Wed, 21 Aug 2013 19:41:42 +0000 (UTC) Received: from [98.138.226.179] by nm23.bullet.mail.ne1.yahoo.com with NNFMP; 21 Aug 2013 19:41:34 -0000 Received: from [98.138.84.39] by tm14.bullet.mail.ne1.yahoo.com with NNFMP; 21 Aug 2013 19:41:34 -0000 Received: from [127.0.0.1] by smtp107.mail.ne1.yahoo.com with NNFMP; 21 Aug 2013 19:41:33 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1377114093; bh=IC+U8ZNZGDdDDlgdE4rtcRWDAd0N/W+ZfDEhv0z9zoQ=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To:X-Mailer; b=SyWmql0X9IQH9O6YdKYcYx0bptzUs3jFunIo6AgLhTbEtPsmjXn70jahmFETRLQyQ/ZcC0a5sa/LjDbENpjSSYJ6WjZWgzYl9cdjahPh2kynFRtyQ4M4/rejhGV2nWhwZ24tgNMepGtfIMf61VNn6xMDrY/33aGbrTH+GKx9TEI= X-Yahoo-Newman-Id: 978900.93516.bm@smtp107.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: XCKIae4VM1nAQfclg6rerxlHGn94L8CmNs2haylBjpdrdDI g0fjJJc_o3nSQRIKXZ.lzpsE_RFtrgsDPiLtVkOoMh2J5yVd41Hgg0yXWuaN 4gKwZuMrRCQJdvSiWK29sokRpg9GACPeZQKF_tzC07LDMP3q4pRaWjCcs9sS nqd1swOK2hSEgBVkds6G2cq.s2Flv3LwUyO17.7atq.tme7x.dEqdYsf7Mxk FdEx69brbpEK.q0KOePZY6ajG8jUow0X_XY9._LjtjGilU3fcZIqqHMtZlqW upyotTConB9wyjUUuxV.LNna07X7YDN1lDiY3xS8HZA06gNwkjxjwl4SUtGK scR1GQHOnhnIzRNTRsxIb8j7MRPIpNOlk2E_zBkvHKVTnQsBCv9JvNL5FmE5 EuxJ4JfYZ5LFmcVyDR8Kh1jSjdRW8eBag0tAyDalNyfpxOJe8nPP2C1mOUsn HWRQSX8vLuDLP7zNdnDMGEIcHKI8hKx_CPBg_lUhtfB3wKE0KNoHnVyU5ist Ao7ZiBkqeNjRSv6KHxIwv8iO_3HQ3OHxJuiiMXJoJC5B53Cs5XvVU0OWKqDK ECw-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- X-Rocket-Received: from [10.2.176.206] (scott4long@69.53.237.66 with ) by smtp107.mail.ne1.yahoo.com with SMTP; 21 Aug 2013 12:41:33 -0700 PDT Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: svn commit: r254520 - in head/sys: kern sys From: Scott Long In-Reply-To: <5214D5E0.9040002@freebsd.org> Date: Wed, 21 Aug 2013 13:41:32 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <201308191116.r7JBGsc6065793@svn.freebsd.org> <521256CE.6070706@FreeBSD.org> <5212870A.50105@freebsd.org> <521291F1.8060500@FreeBSD.org> <5214D5E0.9040002@freebsd.org> To: Andre Oppermann X-Mailer: Apple Mail (2.1508) Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, Navdeep Parhar X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Aug 2013 19:41:42 -0000 On Aug 21, 2013, at 8:59 AM, Andre Oppermann wrote: > On 19.08.2013 23:45, Navdeep Parhar wrote: >> On 08/19/13 13:58, Andre Oppermann wrote: >>> On 19.08.2013 19:33, Navdeep Parhar wrote: >>>> On 08/19/13 04:16, Andre Oppermann wrote: >>>>> Author: andre >>>>> Date: Mon Aug 19 11:16:53 2013 >>>>> New Revision: 254520 >>>>> URL: http://svnweb.freebsd.org/changeset/base/254520 >>>>>=20 >>>>> Log: >>>>> Remove the unused M_NOFREE mbuf flag. It didn't have any = in-tree >>>>> users >>>>> for a very long time, if ever. >>>>>=20 >>>>> Should such a functionality ever be needed again the = appropriate and >>>>> much better way to do it is through a custom EXT_SOMETHING >>>>> external mbuf >>>>> type together with a dedicated *ext_free function. >>>>>=20 >>>>> Discussed with: trociny, glebius >>>>>=20 >>>>> Modified: >>>>> head/sys/kern/kern_mbuf.c >>>>> head/sys/kern/uipc_mbuf.c >>>>> head/sys/sys/mbuf.h >>>>>=20 >>>>=20 >>>> Hello Andre, >>>>=20 >>>> Is this just garbage collection or is there some other reason for = this? >>>=20 >>> This is garbage collection and removal of not quite right, rotten, >>> functionality. >>>=20 >>>> I recently tried some experiments to reduce the number of mbuf and >>>> cluster allocations in a 40G NIC driver. M_NOFREE and EXT_EXTREF = proved >>>> very useful and the code changes to the kernel were minimal. See >>>> user/np/cxl_tuning. The experiment was quite successful and I was >>>> planning to bring in most of those changes to HEAD. I was hoping = to get >>>> some runtime mileage on the approach in general before tweaking the >>>> ctors/dtors for jumpbo, jumbo9, jumbo16 to allow for an mbuf+refcnt >>>> within the cluster. But now M_NOFREE has vanished without a = warning... >>>=20 >>> I'm looking through your experimental code and that is some really = good >>> numbers you're achieving there! >>>=20 >>> However a couple things don't feel quite right, hackish even, and = not >>> fit for HEAD. This is a bit the same situation we had with some of = the >>> first 1GigE cards quite a number of years back (mostly ti(4)). = There >>> we ended up with a couple of just good enough hacks to make it fast. >>> Most of the remains I've collected today. >>=20 >> If M_NOFREE and EXT_EXTREF are properly supported in the tree (and = I'm >> arguing that they were, before r254520) then the changes are = perfectly >> legitimate. The only hackish part was that I was getting the cluster >> from the jumbop zone while bypassing its normal refcnt mechanism. = This >> I did so as to use the same zone as m_uiotombuf to keep it "hot" for = all >> consumers (driver + network stack). >=20 > If you insist I'll revert the commit removing M_NOFREE. EXT_EXTREF = isn't > touched yet, but should get better support. >=20 > The hackish part for me is that the driver again manages its own = memory > pool. Windows works that way, NetBSD is moving towards it while = FreeBSD > has and remains at a central network memory pool. The latter (our = current) > way of doing it seems more efficient overall especially on heavily = loaded > networked machines. There may be significant queues building (think = app > blocked having many sockets buffer fill up) up delaying the freeing = and > returning of network memory resources. Together with fragmentation = this > can lead to bad very outcomes. Router applications with many = interfaces > also greatly benefit from central memory pools. >=20 > So I'm really not sure that we should move back in the driver owned = pool > direction with lots of code duplication and copy-pasting (see NetBSD). > Also it is kinda weird to have a kernel based pool for data going down > the stack and another one in each driver for those going up. >=20 > Actually I'm of the opinion that we should stay with the central = memory > pool and fix so that it works just as well for those cases a driver = pool > currently performs better. The central memory pool approach is too slow, unfortunately. There's a reason that other OS's are moving to them. At Netflix we are currently working on some approaches to private memory pools in order to achieve better efficiency, and we're closely watching and anticipating = Navdeep's work. Scott