From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 14 02:06:49 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C8240988 for ; Fri, 14 Mar 2014 02:06:49 +0000 (UTC) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 65AF3C57 for ; Fri, 14 Mar 2014 02:06:49 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s2E26kJr075211; Thu, 13 Mar 2014 22:06:46 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.7/8.14.4/Submit) id s2E26ku5075208; Thu, 13 Mar 2014 22:06:46 -0400 (EDT) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21282.25654.342428.860428@hergotha.csail.mit.edu> Date: Thu, 13 Mar 2014 22:06:46 -0400 From: Garrett Wollman To: John-Mark Gurney Subject: Re: kernel memory allocator: UMA or malloc? In-Reply-To: <20140314015021.GN32089@funkthat.com> References: <20140313054659.GG32089@funkthat.com> <1783335610.22308389.1394749339304.JavaMail.root@uoguelph.ca> <20140314015021.GN32089@funkthat.com> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Thu, 13 Mar 2014 22:06:46 -0400 (EDT) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu X-Mailman-Approved-At: Fri, 14 Mar 2014 03:59:12 +0000 Cc: Freebsd hackers list , Rick Macklem X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Mar 2014 02:06:49 -0000 < said: > So, this is where a UMA half alive object could be helpful... Say that > you always need to allocate an iovec + 8 mbuf clusters to populate the > iovec... What you can do is have a uma uminit function that allocates > the memory for the iovec and 8 mbuf clusters, and populates the iovec > w/ the correct addresses... Then when you call uma_zalloc, the iovec > is already initalized, and you just go on your merry way instead of > doing all that work... when you uma_zfree, you don't have to worry > about loosing the clusters as the next uma_zalloc might return the > same object w/ the clusters already present... When the system gets > low on memory, it will call your fini function which will need to > free the clusters.... I thought about this, but I don't think it helps, because the mbufs are going to get handed into the network stack and queued in TCP and then in the interface for potentially a long period of time -- with no callback to NFS that would tell it that the mbufs are now free -- whereas the iovec (and in my implementation, the uio) can get freed immediately and recycled. If we had the ability to get 64k chunks of direct-mapped physmem -- from a boot-time-reserved region of memory -- and use those as mbufs, then it might be a win, because then you could cache a buffer, the mbuf that points to it, the iovec that points to the mbuf, and the uio that points to the iovec all in the same allocation, and get a callback when the last reference to that buffer drops. I expect that would significantly improve performance on high-end servers like the ones I've built, but we'd need to arrange for Rick to get a decent 64-bit server for testing. On a 96-GiB server, I'd be perfectly willing to reserve a couple of 1 GiB superpages for this purpose. -GAWollman