From owner-freebsd-arch@FreeBSD.ORG Fri Mar 24 23:17:10 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6E81616A423 for ; Fri, 24 Mar 2006 23:17:10 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9D5F943D68 for ; Fri, 24 Mar 2006 23:17:07 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id 82DB95E48BF; Fri, 24 Mar 2006 15:17:07 -0800 (PST) Received: from [192.168.168.201] (moscow-cuda-gen2-68-64-60-20.losaca.adelphia.net [68.64.60.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id 2F29B5E48BF for ; Fri, 24 Mar 2006 15:17:06 -0800 (PST) Message-ID: <44247DF1.8000002@FreeBSD.org> Date: Fri, 24 Mar 2006 15:17:05 -0800 From: Jason Evans User-Agent: Mozilla Thunderbird 1.0.7-1.4.1 (X11/20050929) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-arch@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.5 Cc: Subject: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2006 23:17:10 -0000 === Proposal === Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: size_t malloc_size_np(const void *ptr); This function would return the usable size of the allocation pointed to by ptr, which is always greater than or equal to the size that was specified in the call to malloc(), calloc(), posix_memalign(), or realloc(), regardless of allocator implementation. It is up to the implementation whether the value returned by malloc_size_np() differs from the size specified during allocation, but the return value must be consistent across multiple calls for the same allocation. === Rationale === The standard malloc(3) API does not provide a mechanism for querying the sizes of existing allocations. This can be useful for a variety of purposes, including introspection and putting wrappers around the system malloc implementation. For a stand-alone application, it is possible to always keep track of allocation sizes, but when integrating with third party libraries, this may not be possible. The lack of such a function causes issues when porting software from Linux or OS X. Linux provides malloc_usable_size(), which returns the usable size of an allocation (may be larger than the allocation request size). OS X provides malloc_size() and malloc_good_size(), which return the precise allocation request size and the usable size of the allocation, respectively. By providing malloc_size_np(), we provide an analogue to these APIs, without placing undue constraints on future malloc implementations. I propose malloc_size_np() rather than malloc_size() (and a separate header file) in order to emphasize the non-portability of this function. As a case in point, the proposed semantics differ from those of OS X's malloc_size(). If we were to use malloc_size(), there would be the potential for undue expectation of equivalence between operating systems. === Comments? Thanks, Jason From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 04:51:51 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B9D0416A400; Sat, 25 Mar 2006 04:51:51 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id DB32243D46; Sat, 25 Mar 2006 04:51:50 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (vby8ne24k4y16qpt@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.3/8.13.3) with ESMTP id k2P4polc015383; Fri, 24 Mar 2006 20:51:50 -0800 (PST) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.3/8.13.3/Submit) id k2P4pnOl015382; Fri, 24 Mar 2006 20:51:49 -0800 (PST) (envelope-from jmg) Date: Fri, 24 Mar 2006 20:51:49 -0800 From: John-Mark Gurney To: Jason Evans Message-ID: <20060325045149.GA7001@funkthat.com> Mail-Followup-To: Jason Evans , freebsd-arch@FreeBSD.org References: <44247DF1.8000002@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <44247DF1.8000002@FreeBSD.org> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 04:51:51 -0000 Jason Evans wrote this message on Fri, Mar 24, 2006 at 15:17 -0800: > === Proposal === > Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: > > size_t > malloc_size_np(const void *ptr); > > This function would return the usable size of the allocation pointed to > by ptr, which is always greater than or equal to the size that was > specified in the call to malloc(), calloc(), posix_memalign(), or > realloc(), regardless of allocator implementation. It is up to the > implementation whether the value returned by malloc_size_np() differs > from the size specified during allocation, but the return value must be > consistent across multiple calls for the same allocation. a) the idea of using memory beyond what you've allocated is bad... b) it should return the maximum size you can realloc to w/o a copy of the memory... and then the way you use the extra memory is to call realloc on the buffer to the value returned by malloc_size_np.. My vote is to never let allocations go outside their area even if they could... It'll just make new programmers lazy, and ready to break something... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 07:46:39 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E690D16A400; Sat, 25 Mar 2006 07:46:39 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 58F6843D45; Sat, 25 Mar 2006 07:46:39 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id DB8971EC319; Sat, 25 Mar 2006 08:46:37 +0100 (CET) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.4/8.13.4) with ESMTP id k2P7kaQQ062733; Sat, 25 Mar 2006 08:46:37 +0100 (CET) (envelope-from phk@critter.freebsd.dk) To: Jason Evans From: "Poul-Henning Kamp" In-Reply-To: Your message of "Fri, 24 Mar 2006 15:17:05 PST." <44247DF1.8000002@FreeBSD.org> Date: Sat, 25 Mar 2006 08:46:36 +0100 Message-ID: <62732.1143272796@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 07:46:40 -0000 In message <44247DF1.8000002@FreeBSD.org>, Jason Evans writes: >=== Proposal === >Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: > > size_t > malloc_size_np(const void *ptr); I'm for the concept, but wonder if it would be smarter to make it void * malloc_np(size_t, size_t *) so we can do it in one go ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 07:47:41 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EE19F16A400; Sat, 25 Mar 2006 07:47:41 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 927C243D45; Sat, 25 Mar 2006 07:47:41 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id AC4061EC318; Sat, 25 Mar 2006 08:47:40 +0100 (CET) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.4/8.13.4) with ESMTP id k2P7ldZr062793; Sat, 25 Mar 2006 08:47:40 +0100 (CET) (envelope-from phk@critter.freebsd.dk) To: John-Mark Gurney From: "Poul-Henning Kamp" In-Reply-To: Your message of "Fri, 24 Mar 2006 20:51:49 PST." <20060325045149.GA7001@funkthat.com> Date: Sat, 25 Mar 2006 08:47:39 +0100 Message-ID: <62792.1143272859@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: Jason Evans , freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 07:47:42 -0000 In message <20060325045149.GA7001@funkthat.com>, John-Mark Gurney writes: >a) the idea of using memory beyond what you've allocated is bad... Not if you get permission. >b) it should return the maximum size you can realloc to w/o a copy >of the memory... and then the way you use the extra memory is to >call realloc on the buffer to the value returned by malloc_size_np.. This is not an easy number to return, and it is dynamic. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 08:06:35 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4694216A401; Sat, 25 Mar 2006 08:06:35 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0E21843D45; Sat, 25 Mar 2006 08:06:34 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.13.4.20060308/8.13.4) with ESMTP id k2P86Y3U011862; Sat, 25 Mar 2006 00:06:34 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.13.4.20060308/8.13.4/Submit) id k2P86YJU011861; Sat, 25 Mar 2006 00:06:34 -0800 (PST) Date: Sat, 25 Mar 2006 00:06:34 -0800 (PST) From: Matthew Dillon Message-Id: <200603250806.k2P86YJU011861@apollo.backplane.com> To: Jason Evans References: <44247DF1.8000002@FreeBSD.org> Cc: freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 08:06:35 -0000 : :=== Proposal === :Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: : : size_t : malloc_size_np(const void *ptr); : :This function would return the usable size of the allocation pointed to :by ptr, which is always greater than or equal to the size that was :specified in the call to malloc(), calloc(), posix_memalign(), or :... That creates a serious race condition for any threaded program, since multiple threads might be malloc()ing memory at the same time and one could wind up using the 'extra' space that abuts an allocation being tested by some other thread. Maybe something like: n = realloc_try(ptr, bytes) This function attempts to extend the size of a previously allocated memory block without changing its pointer. The total size of the (now possibly extended) memory pointed to by pointer is returned. The total size returned will always be at least min(bytes, previous_size) and will never exceed (bytes). The function cannot fail, but may not return the requested number of bytes. This function may also be used to truncate the size of a previously allocated memory block without changing its pointer. This function will always return (bytes) when the passed bytes is less then or equal to the size of the current allocation. The function does NOT guarentee that the memory allocator can actually use the freed memory past the truncation point for other unrelated allocations, but it DOES guarentee that, baring other allocations that reuse the memory, the current allocation can be extended back into the truncated area with another call to realloc_try(). -- That would give you the most flexibility and be relatively opaque to the actual malloc implementation. The degenerate case would only have to determine the size of the current allocation to return a meaningful result (even if it doesn't actually extend or truncate it), and that information clearly must already be attainable since the original allocation size is not specified when you free() the memory. -Matt From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 08:07:41 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C6C8816A41F for ; Sat, 25 Mar 2006 08:07:41 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7661143D46 for ; Sat, 25 Mar 2006 08:07:41 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id 55A4B5E4911; Sat, 25 Mar 2006 00:07:41 -0800 (PST) Received: from [192.168.168.201] (moscow-cuda-gen2-68-64-60-20.losaca.adelphia.net [68.64.60.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id 7158B5E4906; Sat, 25 Mar 2006 00:07:40 -0800 (PST) Message-ID: <4424FA4B.5080201@FreeBSD.org> Date: Sat, 25 Mar 2006 00:07:39 -0800 From: Jason Evans User-Agent: Mozilla Thunderbird 1.0.7-1.4.1 (X11/20050929) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Poul-Henning Kamp References: <62732.1143272796@critter.freebsd.dk> In-Reply-To: <62732.1143272796@critter.freebsd.dk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.5 Cc: freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 08:07:41 -0000 Poul-Henning Kamp wrote: > In message <44247DF1.8000002@FreeBSD.org>, Jason Evans writes: > >>=== Proposal === >>Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: >> >> size_t >> malloc_size_np(const void *ptr); > > > I'm for the concept, but wonder if it would be smarter to make it > > void * > malloc_np(size_t, size_t *) > > so we can do it in one go ? I can see how this would be useful, but I don't think it completely replaces the use case for malloc_size_np(), which can be useful at other times than allocation/deallocation (ex: assertions that detect buffer overflow). Also, if we add malloc_np(), for completeness we should also provide: void * calloc_np(size_t size, size_t *size_ceil); void * realloc_np(void *ptr, size_t size, size_t *size_ceil); (Let's ignore posix_memalign() here.) size_t free_np(void *ptr); Given that these functions are essentially the (somewhat higher performance) fusion of two function calls, it isn't clear to me whether it's worth adding several extra functions. Personally, I would use them if they were available, but I'm not sure if they would be broadly useful. Thoughts? Thanks, Jason From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 08:10:44 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7597E16A422; Sat, 25 Mar 2006 08:10:44 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail18.syd.optusnet.com.au (mail18.syd.optusnet.com.au [211.29.132.199]) by mx1.FreeBSD.org (Postfix) with ESMTP id CAE0143D4C; Sat, 25 Mar 2006 08:10:43 +0000 (GMT) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-19-236.belrs4.nsw.optusnet.com.au [220.239.19.236]) by mail18.syd.optusnet.com.au (8.12.11/8.12.11) with ESMTP id k2P8AcZ3004495 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Sat, 25 Mar 2006 19:10:40 +1100 Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.13.4/8.13.4) with ESMTP id k2P8Acd7005480; Sat, 25 Mar 2006 19:10:38 +1100 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.13.4/8.13.4/Submit) id k2P8AcIZ005479; Sat, 25 Mar 2006 19:10:38 +1100 (EST) (envelope-from peter) Date: Sat, 25 Mar 2006 19:10:38 +1100 From: Peter Jeremy To: Poul-Henning Kamp Message-ID: <20060325081037.GC703@turion.vk2pj.dyndns.org> References: <44247DF1.8000002@FreeBSD.org> <62732.1143272796@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <62732.1143272796@critter.freebsd.dk> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.11 Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 08:10:44 -0000 On Sat, 2006-Mar-25 08:46:36 +0100, Poul-Henning Kamp wrote: >In message <44247DF1.8000002@FreeBSD.org>, Jason Evans writes: >>=== Proposal === >>Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: >> >> size_t >> malloc_size_np(const void *ptr); > >I'm for the concept, but wonder if it would be smarter to make it > > void * > malloc_np(size_t, size_t *) > >so we can do it in one go ? That restricts its use to malloc() wrappers. I can also see the benefit in a function trying to validate the size of an arbitrary piece of memory that has been passed to it. -- Peter Jeremy From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 08:19:30 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 103AE16A400; Sat, 25 Mar 2006 08:19:30 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id A70EA43D46; Sat, 25 Mar 2006 08:19:27 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id 64A861EC30C; Sat, 25 Mar 2006 09:19:26 +0100 (CET) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.4/8.13.4) with ESMTP id k2P8JNCJ062922; Sat, 25 Mar 2006 09:19:25 +0100 (CET) (envelope-from phk@critter.freebsd.dk) To: Peter Jeremy From: "Poul-Henning Kamp" In-Reply-To: Your message of "Sat, 25 Mar 2006 19:10:38 +1100." <20060325081037.GC703@turion.vk2pj.dyndns.org> Date: Sat, 25 Mar 2006 09:19:23 +0100 Message-ID: <62921.1143274763@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 08:19:30 -0000 In message <20060325081037.GC703@turion.vk2pj.dyndns.org>, Peter Jeremy writes: >On Sat, 2006-Mar-25 08:46:36 +0100, Poul-Henning Kamp wrote: >>In message <44247DF1.8000002@FreeBSD.org>, Jason Evans writes: >>>=== Proposal === >>>Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: >>> >>> size_t >>> malloc_size_np(const void *ptr); >> >>I'm for the concept, but wonder if it would be smarter to make it >> >> void * >> malloc_np(size_t, size_t *) >> >>so we can do it in one go ? > >That restricts its use to malloc() wrappers. I can also see the >benefit in a function trying to validate the size of an arbitrary >piece of memory that has been passed to it. No you can't, because you cannot be certain that the pointer passed to you is actually an unadultered malloc pointer. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 08:23:08 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E2A4816A41F for ; Sat, 25 Mar 2006 08:23:07 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8324243D48 for ; Sat, 25 Mar 2006 08:23:07 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id 4E7A45E4906; Sat, 25 Mar 2006 00:23:07 -0800 (PST) Received: from [192.168.168.201] (moscow-cuda-gen2-68-64-60-20.losaca.adelphia.net [68.64.60.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id 17AC05E4906; Sat, 25 Mar 2006 00:23:06 -0800 (PST) Message-ID: <4424FDE9.3080707@FreeBSD.org> Date: Sat, 25 Mar 2006 00:23:05 -0800 From: Jason Evans User-Agent: Mozilla Thunderbird 1.0.7-1.4.1 (X11/20050929) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Matthew Dillon References: <44247DF1.8000002@FreeBSD.org> <200603250806.k2P86YJU011861@apollo.backplane.com> In-Reply-To: <200603250806.k2P86YJU011861@apollo.backplane.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.5 Cc: freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 08:23:08 -0000 Matthew Dillon wrote: > : > :=== Proposal === > :Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: > : > : size_t > : malloc_size_np(const void *ptr); > : > :This function would return the usable size of the allocation pointed to > :by ptr, which is always greater than or equal to the size that was > :specified in the call to malloc(), calloc(), posix_memalign(), or > :... > > That creates a serious race condition for any threaded program, > since multiple threads might be malloc()ing memory at the same > time and one could wind up using the 'extra' space that abuts > an allocation being tested by some other thread. It sounds like I wasn't clear enough about the motivation for the vague specification of malloc_size_np()'s return value. The issue I am concerned about is that an allocator implementation does not actually have to store the precise allocation request size; it can instead round the request size up internally, then treat the allocation as being of this rounded up size. By vaguely specifying the return value of malloc_size_np(), we grant the malloc implementation freedom as to whether the size is precisely what was specified during allocation, or some rounded up value. I had no intention of suggesting that malloc_size_np() should extend existing allocations, nor change the return value depending on the current state of memory following the allocation pointed to by ptr. > Maybe something like: > > n = realloc_try(ptr, bytes) > > This function attempts to extend the size of a previously allocated > memory block without changing its pointer. The total size of the > (now possibly extended) memory pointed to by pointer is returned. The > total size returned will always be at least min(bytes, previous_size) > and will never exceed (bytes). The function cannot fail, but may not > return the requested number of bytes. > > This function may also be used to truncate the size of a previously > allocated memory block without changing its pointer. This function > will always return (bytes) when the passed bytes is less then or > equal to the size of the current allocation. The function does NOT > guarentee that the memory allocator can actually use the freed memory > past the truncation point for other unrelated allocations, but it DOES > guarentee that, baring other allocations that reuse the memory, the > current allocation can be extended back into the truncated area with > another call to realloc_try(). > > -- > > That would give you the most flexibility and be relatively opaque to the > actual malloc implementation. The degenerate case would only have > to determine the size of the current allocation to return a meaningful > result (even if it doesn't actually extend or truncate it), and that > information clearly must already be attainable since the original > allocation size is not specified when you free() the memory. This actually does make some assumptions about the way memory is managed by malloc. It would not be useful in the context of phkmalloc or jemalloc to truncate an allocation via realloc_try(), because each page of memory contains allocations of only one size class. Thus, there is no way to re-use the trailing space (exception: multi-page allocations). This would potentially be useful for dlmalloc (or ptmalloc) though. Thanks, Jason From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 08:37:33 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BF8D416A400; Sat, 25 Mar 2006 08:37:33 +0000 (UTC) (envelope-from mux@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id EE73D43D48; Sat, 25 Mar 2006 08:37:29 +0000 (GMT) (envelope-from mux@freebsd.org) Received: by elvis.mu.org (Postfix, from userid 1920) id 9D1AD1A3C22; Sat, 25 Mar 2006 00:37:29 -0800 (PST) Date: Sat, 25 Mar 2006 09:37:29 +0100 From: Maxime Henrion To: Jason Evans Message-ID: <20060325083729.GN80618@elvis.mu.org> References: <44247DF1.8000002@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <44247DF1.8000002@FreeBSD.org> User-Agent: Mutt/1.4.2.1i Cc: freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 08:37:33 -0000 Jason Evans wrote: > === Proposal === > Add malloc_size_np() to libc, and provide the prototype in malloc_np.h: > > size_t > malloc_size_np(const void *ptr); > > This function would return the usable size of the allocation pointed to > by ptr, which is always greater than or equal to the size that was > specified in the call to malloc(), calloc(), posix_memalign(), or > realloc(), regardless of allocator implementation. It is up to the > implementation whether the value returned by malloc_size_np() differs > from the size specified during allocation, but the return value must be > consistent across multiple calls for the same allocation. > > === Rationale === > The standard malloc(3) API does not provide a mechanism for querying the > sizes of existing allocations. This can be useful for a variety of > purposes, including introspection and putting wrappers around the system > malloc implementation. For a stand-alone application, it is possible to > always keep track of allocation sizes, but when integrating with third > party libraries, this may not be possible. > > The lack of such a function causes issues when porting software from > Linux or OS X. Linux provides malloc_usable_size(), which returns the > usable size of an allocation (may be larger than the allocation request > size). OS X provides malloc_size() and malloc_good_size(), which return > the precise allocation request size and the usable size of the > allocation, respectively. By providing malloc_size_np(), we provide an > analogue to these APIs, without placing undue constraints on future > malloc implementations. > > I propose malloc_size_np() rather than malloc_size() (and a separate > header file) in order to emphasize the non-portability of this function. > As a case in point, the proposed semantics differ from those of OS X's > malloc_size(). If we were to use malloc_size(), there would be the > potential for undue expectation of equivalence between operating systems. > > === I like this idea and, for one, can't see anything to change in your proposal :-). Cheers, MAxime From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 17:40:52 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EEEEF16A426; Sat, 25 Mar 2006 17:40:51 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6850343D5C; Sat, 25 Mar 2006 17:40:48 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.13.4.20060308/8.13.4) with ESMTP id k2PHeloE014341; Sat, 25 Mar 2006 09:40:47 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.13.4.20060308/8.13.4/Submit) id k2PHelT4014340; Sat, 25 Mar 2006 09:40:47 -0800 (PST) Date: Sat, 25 Mar 2006 09:40:47 -0800 (PST) From: Matthew Dillon Message-Id: <200603251740.k2PHelT4014340@apollo.backplane.com> To: Jason Evans References: <44247DF1.8000002@FreeBSD.org> <200603250806.k2P86YJU011861@apollo.backplane.com> <4424FDE9.3080707@FreeBSD.org> Cc: freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 17:40:52 -0000 :... :have to store the precise allocation request size; it can instead round :the request size up internally, then treat the allocation as being of :this rounded up size. By vaguely specifying the return value of :malloc_size_np(), we grant the malloc implementation freedom as to :whether the size is precisely what was specified during allocation, or :some rounded up value. : :I had no intention of suggesting that malloc_size_np() should extend :existing allocations, nor change the return value depending on the :current state of memory following the allocation pointed to by ptr. I'm not sure what benefit this would have, though. When a program calls malloc it generally does so with a good idea as to the number of bytes it wants to allocate. malloc implementations have always been free to return more bytes then were requested. But why would any program want to complicate its life and use an API where the potential additional space (which is extremely algorithm-dependant) becomes visible to the program? I'm trying to imagine the number of mistakes programmers would make, with the resulting bugs winding up in the program(s), that an integrated implementation would cause. An API like malloc_size_np() would have a very limited usefullness, not to mention being inflexible. Programs are far more likely to want to resize an allocation then to use whatever extra slop space the malloc implementation might happened to have included in the original allocation. A reallocation API like realloc_try() makes a lot more sense. Not only does it cover all aspects of malloc_size_np(), it also gives the allocator implementation the flexibility to actually truncate or extend an allocation if that happens to be supported by the allocator, or not if it doesn't. :This actually does make some assumptions about the way memory is managed :by malloc. It would not be useful in the context of phkmalloc or :jemalloc to truncate an allocation via realloc_try(), because each page :of memory contains allocations of only one size class. Thus, there is :no way to re-use the trailing space (exception: multi-page allocations). : This would potentially be useful for dlmalloc (or ptmalloc) though. : :Thanks, :Jason No, it actually doesn't. Or I should say, uf a program is going to actually try to USE the potential extra space for something, something like realloc_try() makes no more assumptions about the malloc implementation than malloc_size_np(). The implementation is totally free to limit the functionality of realloc_try() to the same slop space (if any) that malloc_size_np() returns. The difference is that realloc_try() is far closer to what a programmer would likely need. Even for things like slab allocators which have no ability to extend or truncate small allocations beyond the chunk size of the slab they reside in, many malloc implementations still use a different mechanism for larger multi-page allocations and in those cases the implementation might actually be able to do something with the original allocation. The only real situation that I can think of where a program might want to have the option of (optimally) using more or less space then was originally requested is the situation where a large chunk of memory, such as 32KB, needs to be extended or truncated. For example, a preprocessor or parser (generating post-parsed output) or linker (relocation arrays), or any other program of that nature could benefit. I'm still wondering who in their right mind would actually malloc X bytes and then use an API call to obtain and actually use the (dynamic, ephermal, and completely non-deterministic) Y bytes that the underlying implementation reserved, verses simply allocating Y bytes in the first place. It sounds like a recipe for disaster to me. At least with a reallocation API such as realloc_try() the caller can specify a requirement that the malloc implementation is either able to meet or not able to meet. -Matt Matthew Dillon From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 18:56:13 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 65D1916A41F; Sat, 25 Mar 2006 18:56:13 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id EFC8B43D48; Sat, 25 Mar 2006 18:56:12 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (m2eio4qwgoa7hdwh@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.4/8.13.3) with ESMTP id k2PIuCex039015; Sat, 25 Mar 2006 10:56:12 -0800 (PST) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.4/8.13.3/Submit) id k2PIuCB2039014; Sat, 25 Mar 2006 10:56:12 -0800 (PST) (envelope-from jmg) Date: Sat, 25 Mar 2006 10:56:12 -0800 From: John-Mark Gurney To: Jason Evans Message-ID: <20060325185612.GC7001@funkthat.com> Mail-Followup-To: Jason Evans , freebsd-arch@freebsd.org References: <44247DF1.8000002@FreeBSD.org> <200603250806.k2P86YJU011861@apollo.backplane.com> <4424FDE9.3080707@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4424FDE9.3080707@FreeBSD.org> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 18:56:13 -0000 Jason Evans wrote this message on Sat, Mar 25, 2006 at 00:23 -0800: > have to store the precise allocation request size; it can instead round > the request size up internally, then treat the allocation as being of > this rounded up size. By vaguely specifying the return value of > malloc_size_np(), we grant the malloc implementation freedom as to > whether the size is precisely what was specified during allocation, or > some rounded up value. > > I had no intention of suggesting that malloc_size_np() should extend > existing allocations, nor change the return value depending on the > current state of memory following the allocation pointed to by ptr. Ok, so what you are saying is that the function returns the size of the bucket (if any) that the memory was allocated from... But even though this function may return a larger value, the program is not allowed to use extra space, and it's only useful for further allocations of the same size? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Sat Mar 25 19:11:34 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7C4DE16A420 for ; Sat, 25 Mar 2006 19:11:34 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id D721843D53 for ; Sat, 25 Mar 2006 19:11:33 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id 9D77C5E4912; Sat, 25 Mar 2006 11:11:33 -0800 (PST) Received: from [192.168.168.201] (moscow-cuda-gen2-68-64-60-20.losaca.adelphia.net [68.64.60.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id 420855E48F1; Sat, 25 Mar 2006 11:11:32 -0800 (PST) Message-ID: <442595E2.5080804@FreeBSD.org> Date: Sat, 25 Mar 2006 11:11:30 -0800 From: Jason Evans User-Agent: Mozilla Thunderbird 1.0.7-1.4.1 (X11/20050929) X-Accept-Language: en-us, en MIME-Version: 1.0 To: John-Mark Gurney References: <44247DF1.8000002@FreeBSD.org> <200603250806.k2P86YJU011861@apollo.backplane.com> <4424FDE9.3080707@FreeBSD.org> <20060325185612.GC7001@funkthat.com> In-Reply-To: <20060325185612.GC7001@funkthat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.5 Cc: freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 19:11:34 -0000 John-Mark Gurney wrote: > Jason Evans wrote this message on Sat, Mar 25, 2006 at 00:23 -0800: > >>have to store the precise allocation request size; it can instead round >>the request size up internally, then treat the allocation as being of >>this rounded up size. By vaguely specifying the return value of >>malloc_size_np(), we grant the malloc implementation freedom as to >>whether the size is precisely what was specified during allocation, or >>some rounded up value. >> >>I had no intention of suggesting that malloc_size_np() should extend >>existing allocations, nor change the return value depending on the >>current state of memory following the allocation pointed to by ptr. > > Ok, so what you are saying is that the function returns the size of > the bucket (if any) that the memory was allocated from... But even > though this function may return a larger value, the program is not > allowed to use extra space, and it's only useful for further > allocations of the same size? I'm saying that malloc_size_np() returns the size of the allocation, to the best of the allocator's knowledge. If you malloc(17), and malloc_size_np() returns 32 for that allocation, then you can treat it as a 32-byte allocation. However, the malloc implementation could conceivably return any value >=17. Jason