From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 20:51:32 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6733816A401; Mon, 27 Mar 2006 20:51:32 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id F1F5F43D48; Mon, 27 Mar 2006 20:51:31 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id D26925E4906; Mon, 27 Mar 2006 12:51:31 -0800 (PST) Received: from [129.101.167.145] (unknown [129.101.167.145]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id 2A8925E48D7; Mon, 27 Mar 2006 12:51:30 -0800 (PST) In-Reply-To: <20060327185017.GF7001@funkthat.com> References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <20060327185017.GF7001@funkthat.com> Mime-Version: 1.0 (Apple Message framework v746.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <352D1430-CC72-47EE-9E50-B1C4404ACA95@FreeBSD.org> Content-Transfer-Encoding: 7bit From: Jason Evans Date: Mon, 27 Mar 2006 12:52:17 -0800 To: John-Mark Gurney X-Mailer: Apple Mail (2.746.2) X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=failed version=3.0.5 Cc: Robert Watson , Poul-Henning Kamp , John Baldwin , freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 20:51:32 -0000 On Mar 27, 2006, at 10:50 AM, John-Mark Gurney wrote: > Jason Evans wrote this message on Mon, Mar 27, 2006 at 08:34 -0800: >> >> I can think of a few straightforward uses: > > [...] > >> 2) [...] > > This use case is incorrect... Fair enough. That leaves three use cases. >> 4) Porting code from Linux. Case in point: Xara Xtreme, currently >> being >> ported by Vasil Dimov. At the moment, he has to use dlmalloc. > > sad to say, but we seem to always inherit Linux's warts in the name of > portability.. :( I'd say adopt a method similar to the ports tree, > once > we have 10+ ports depending upon it, then we can consider this.. but > importing this function for a single program seems a bit excessive.. malloc_usable_size() is *not* a Linux wart. Instead, it mostly[1] addresses a fundamental flaw in the malloc(3) API. We must go back much farther than Linux if we really want to start disparaging malloc (3). Consider that the allocator must know how large allocations are so that it can do the right thing for realloc() and free(). Consider also that the application must know how large its objects are so that it can stay in bounds. That means that such information has to be stored in two places. Following is a much more reasonable API that doesn't burden the allocator with information that the application should be able to provide: void * malloc(size_t size); void * calloc(size_t size); void * realloc(void *ptr, size_t size, size_t old_size); void free(void *ptr, size_t size); That's not what we have though, and we're certainly not going to completely redesign malloc(3) at this point. So, instead, we're faced with trying to patch things up as well as is feasible without breaking compatibility. Let's re-visit Poul-Henning's suggestion, which involves a parallel API that exposes the internal sizes to the application. The following is not identical to what he proposed, but it has the same semantics: void * malloc_np(size_t *size); void * calloc_np(size_t *size); void * realloc_np(void *ptr, size_t *size, size_t *old_size); void * memalign_np(size_t alignment, size_t *size); void * free_np(void *ptr, size_t *size); This serves much the same purpose as the idealized malloc(3) API I outlined above but with the advantage that it can co-exist with the current malloc(3) API. From an engineering perspective, this is probably actually a better solution than malloc_usable_size(), but 1) it's a much broader API, 2) it's nothing like what any other OSes have done, and 3) in practice it really doesn't gain us anything over malloc_usable_size() (in fact it doesn't solve the problem of porting Linux and OS X software). For my own personal needs, I really don't care whether we add malloc_usable_size() or the broader API described above. However, I do feel like the latter is an over-engineered solution to the problem. To me, this isn't about reinventing a perfect wheel -- it's about filing off a rough edge on the stone wheel we're stuck with. Jason [1] In a slightly more ideal world, we would add malloc_size() instead of malloc_usable_size(), which would return the size that was passed to malloc()/calloc()/realloc(). However, that would impose implementation constraints on the allocator that would have far- reaching implications. I don't want to get into the details here, but suffice it to say that requiring support for malloc_size() would require a lot of extra space overhead for many allocator designs, including for phkmalloc and jemalloc.