From owner-freebsd-current@FreeBSD.ORG Fri Nov 29 19:17:04 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9B209AE4 for ; Fri, 29 Nov 2013 19:17:04 +0000 (UTC) Received: from mail-lb0-x22c.google.com (mail-lb0-x22c.google.com [IPv6:2a00:1450:4010:c04::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 113031A56 for ; Fri, 29 Nov 2013 19:17:03 +0000 (UTC) Received: by mail-lb0-f172.google.com with SMTP id z5so7211401lbh.17 for ; Fri, 29 Nov 2013 11:17:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:cc:content-type; bh=DSpN88Js5jO6SUJDV5EUkv31bQIso8r2SBJmTwTJKzI=; b=ytrc6A1QVe/RF4xjJLiNIDTCCU79ud1xgN2SPHouxAiWcWp9bszIGpaQkT5e2rFAPs PhB+xs+pqucOLmLhPiJD+3WTc9R2cUhcxFduy96+M3GpkOTHSkyZWFP1EWkn5AvDIngC 8/AdA0pRcwX7FU05A+iGNgy9EFQefDfwdoI1QllCuPXYPqVkrLlfC/nfoG/UnF3CIi6R 206ons5s7Y7hwXyEWcqSRjO3n2hRTE05hjVCH1vk1RQNwqnuZdOcnW2816YOIScFh6nb C51fMhi1zzvk+IgCUlaerGyXgnfJkjxrdPMEhMurbf7KaGMqU2nZ660y81vPDOAL3Ene RRLA== MIME-Version: 1.0 X-Received: by 10.112.219.99 with SMTP id pn3mr18810138lbc.24.1385752621717; Fri, 29 Nov 2013 11:17:01 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.175.180 with HTTP; Fri, 29 Nov 2013 11:17:01 -0800 (PST) Date: Fri, 29 Nov 2013 11:17:01 -0800 X-Google-Sender-Auth: o_YVzGJDZ5Ms8BkLj8gfB6jOcWE Message-ID: Subject: Re: [RFC] how to get the size of a malloc(9) block ? From: Luigi Rizzo To: jb Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.16 Cc: freebsd-current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2013 19:17:04 -0000 On Thu, Nov 28, 2013 at 7:13 AM, jb wrote: > Luigi Rizzo iet.unipi.it> writes: > > > ... > > But I don't understand why you find ksize()/malloc_usable_size() > dangerous. > > ... > > The original crime is commited when *usable size* (an implementation > detail) > is exported (leaked) to the caller. > To be blunt, when a caller requests memory of certain size, and its > request is > satisfied, then it is not its business to learn details beyond that (and > they > should not be offered as well). > The API should be sanitized, in kernel and user space. > Otherwise, all kind of charlatans will try to play hair-raising games with > it. > If the caller wants to track the *requested size* programmatically, it is > its > business to do it and it can be done very easily. > There is a difference between applications peeking into implementation details that should be hidden, and providing instead limited and specific information through a well defined API. In general (not in the specific code I am handling and not something I personally need), what the caller might want to do is optimize its requests according to how system behaves, and it cannot do that without some help from the below. I have seen the following types of comments in this thread: - "you should get it right the first time and never realloc" Maybe, but then the offending api is realloc() not ksize() - "build your own allocator" Yes i do it when it makes sense, but sometimes it is either overkill or a bad idea (as it loses opportunities for global optimizations, duplicates code, takes memory in subsystem-specific freelists...) - "what if ksize()/malloc_usable_size() lies ?" Well, that would be a bug in the allocator: if it says the memory is usable, it must be usable, period. - "rather than ksize() i'll give you a fix for one use case" (the NO_REALLOC flag to realloc()). This i think would be a mistake -- it acknowledges the need for exposing some information but then only provides a specific fix for one use case. I'll just restate that there are multiple situations where an application might use some information on actual allocation sizes: - when it needs to extend memory and has a choice between a cheap realloc() (if extra space is available), chaining blocks (when the memcpy would be too expensive), give up and live with whatever space is available. - when it has freedom in picking the block size and so it wants to optimize its requests basing on what the underlying allocator does. As an example, long ago FreeBSD was really suboptimal when you allocated blocks whose size was a power of 2, because the metadata was inline. These days, there is a different issue: powers of 2 are ok but blocks 2049 bytes and above seem to be padded to a multiple of 2048, leading to a huge overhead in some cases. cheers luigi