From owner-freebsd-arch@FreeBSD.ORG Sun Mar 26 11:11:20 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6636B16A400; Sun, 26 Mar 2006 11:11:20 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B53F43D45; Sun, 26 Mar 2006 11:11:19 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id F190346C7B; Sun, 26 Mar 2006 06:11:18 -0500 (EST) Date: Sun, 26 Mar 2006 11:11:18 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Jason Evans In-Reply-To: <442595E2.5080804@FreeBSD.org> Message-ID: <20060326110929.V35431@fledge.watson.org> References: <44247DF1.8000002@FreeBSD.org> <200603250806.k2P86YJU011861@apollo.backplane.com> <4424FDE9.3080707@FreeBSD.org> <20060325185612.GC7001@funkthat.com> <442595E2.5080804@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: John-Mark Gurney , freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Mar 2006 11:11:20 -0000 On Sat, 25 Mar 2006, Jason Evans wrote: >> Ok, so what you are saying is that the function returns the size of the >> bucket (if any) that the memory was allocated from... But even though this >> function may return a larger value, the program is not allowed to use extra >> space, and it's only useful for further allocations of the same size? > > I'm saying that malloc_size_np() returns the size of the allocation, to the > best of the allocator's knowledge. If you malloc(17), and malloc_size_np() > returns 32 for that allocation, then you can treat it as a 32-byte > allocation. However, the malloc implementation could conceivably return any > value >=17. I wonder if the intuitive objection people are raising is actually with the name. Since malloc_size() is defined on at least one platform to return the requested size, maybe a name like malloc_allocated_size() (or something a bit more compact) would help avoid that confusion, and make it clear that the consumer is getting back a commitment and not a hint for future realloc(), etc. Robert N M Watson From owner-freebsd-arch@FreeBSD.ORG Sun Mar 26 18:04:19 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EA77616A401; Sun, 26 Mar 2006 18:04:19 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9602843D45; Sun, 26 Mar 2006 18:04:19 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id 68AD35E4911; Sun, 26 Mar 2006 10:04:19 -0800 (PST) Received: from [192.168.168.201] (moscow-cuda-gen2-68-64-60-20.losaca.adelphia.net [68.64.60.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id E461C5E48D8; Sun, 26 Mar 2006 10:04:17 -0800 (PST) Message-ID: <4426D7A0.4040007@FreeBSD.org> Date: Sun, 26 Mar 2006 10:04:16 -0800 From: Jason Evans User-Agent: Mozilla Thunderbird 1.0.7-1.4.1 (X11/20050929) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: <44247DF1.8000002@FreeBSD.org> <200603250806.k2P86YJU011861@apollo.backplane.com> <4424FDE9.3080707@FreeBSD.org> <20060325185612.GC7001@funkthat.com> <442595E2.5080804@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> In-Reply-To: <20060326110929.V35431@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.5 Cc: John-Mark Gurney , freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Mar 2006 18:04:20 -0000 Robert Watson wrote: > I wonder if the intuitive objection people are raising is actually with > the name. Since malloc_size() is defined on at least one platform to > return the requested size, maybe a name like malloc_allocated_size() (or > something a bit more compact) would help avoid that confusion, and make > it clear that the consumer is getting back a commitment and not a hint > for future realloc(), etc. Maybe you're right. We could just call it malloc_usable_size() and be compatible with Linux. Jason From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 01:17:13 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9DEA616A400 for ; Mon, 27 Mar 2006 01:17:13 +0000 (UTC) (envelope-from andrew@areilly.bpc-users.org) Received: from omta01ps.mx.bigpond.com (omta01ps.mx.bigpond.com [144.140.82.153]) by mx1.FreeBSD.org (Postfix) with ESMTP id 036CF43D46 for ; Mon, 27 Mar 2006 01:17:08 +0000 (GMT) (envelope-from andrew@areilly.bpc-users.org) Received: from areilly.bpc-users.org ([141.168.4.160]) by omta01ps.mx.bigpond.com with ESMTP id <20060327011706.ORHR19070.omta01ps.mx.bigpond.com@areilly.bpc-users.org> for ; Mon, 27 Mar 2006 01:17:06 +0000 Received: (qmail 66247 invoked by uid 501); 26 Mar 2006 21:03:41 -0000 Date: Mon, 27 Mar 2006 08:03:41 +1100 From: Andrew Reilly To: Poul-Henning Kamp Message-ID: <20060326210341.GA66065@gurney.reilly.home> References: <20060325081037.GC703@turion.vk2pj.dyndns.org> <62921.1143274763@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <62921.1143274763@critter.freebsd.dk> User-Agent: Mutt/1.4.2.1i Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 01:17:13 -0000 On Sat, Mar 25, 2006 at 09:19:23AM +0100, Poul-Henning Kamp wrote: > In message <20060325081037.GC703@turion.vk2pj.dyndns.org>, Peter Jeremy writes: > >That restricts its use to malloc() wrappers. I can also see the > >benefit in a function trying to validate the size of an arbitrary > >piece of memory that has been passed to it. > > No you can't, because you cannot be certain that the pointer passed > to you is actually an unadultered malloc pointer. Or even that it was 'alloc'd at all. Could be a pointer (in)to a static or global... -- Andrew From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 16:09:47 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 75D9316A424; Mon, 27 Mar 2006 16:09:47 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id BE87643D46; Mon, 27 Mar 2006 16:09:46 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from localhost (john@localhost [127.0.0.1]) by server.baldwin.cx (8.13.4/8.13.4) with ESMTP id k2RG9X4e061929; Mon, 27 Mar 2006 11:09:38 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-arch@freebsd.org Date: Mon, 27 Mar 2006 11:10:00 -0500 User-Agent: KMail/1.9.1 References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> In-Reply-To: <4426D7A0.4040007@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200603271110.02917.jhb@freebsd.org> X-Virus-Scanned: ClamAV 0.87.1/1357/Sat Mar 25 16:37:38 2006 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-3.8 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on server.baldwin.cx Cc: John-Mark Gurney , Robert Watson , Jason Evans Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 16:09:47 -0000 On Sunday 26 March 2006 13:04, Jason Evans wrote: > Robert Watson wrote: > > I wonder if the intuitive objection people are raising is actually with > > the name. Since malloc_size() is defined on at least one platform to > > return the requested size, maybe a name like malloc_allocated_size() (or > > something a bit more compact) would help avoid that confusion, and make > > it clear that the consumer is getting back a commitment and not a hint > > for future realloc(), etc. > > Maybe you're right. We could just call it malloc_usable_size() and be > compatible with Linux. It would help to know why such a function would be useful. :) Do you have a specific use-case? If the purpose is for a program to see that it really as Y >= X bytes when it did malloc(X) so that the program can use Y bytes, that would seem to only be a source of bugs and complexity. If the program wants Y bytes, it should malloc(Y). Many folks in the thread seem to think that this function would be used for a poor-man's realloc() wrapper or something, and I think such uses would be short-sighted at best. If there are other uses such as for having a debug malloc wrap the real one, then that might justify the API, but it is unclear what a useful use of this API would be. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 16:34:53 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0DFD516A400; Mon, 27 Mar 2006 16:34:53 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id B55AC43D64; Mon, 27 Mar 2006 16:34:44 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id 89D255E4906; Mon, 27 Mar 2006 08:34:44 -0800 (PST) Received: from [192.168.168.201] (moscow-cuda-gen2-68-64-60-20.losaca.adelphia.net [68.64.60.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id E673F5E48C1; Mon, 27 Mar 2006 08:34:42 -0800 (PST) Message-ID: <44281421.3060401@FreeBSD.org> Date: Mon, 27 Mar 2006 08:34:41 -0800 From: Jason Evans User-Agent: Mozilla Thunderbird 1.0.7-1.4.1 (X11/20050929) X-Accept-Language: en-us, en MIME-Version: 1.0 To: John Baldwin References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> In-Reply-To: <200603271110.02917.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.5 Cc: Vasil Dimov , Robert Watson , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 16:34:53 -0000 John Baldwin wrote: > On Sunday 26 March 2006 13:04, Jason Evans wrote: > >>Robert Watson wrote: >> >>>I wonder if the intuitive objection people are raising is actually with >>>the name. Since malloc_size() is defined on at least one platform to >>>return the requested size, maybe a name like malloc_allocated_size() (or >>>something a bit more compact) would help avoid that confusion, and make >>>it clear that the consumer is getting back a commitment and not a hint >>>for future realloc(), etc. >> >>Maybe you're right. We could just call it malloc_usable_size() and be >>compatible with Linux. > > It would help to know why such a function would be useful. :) Do you have > a specific use-case? If the purpose is for a program to see that it really > as Y >= X bytes when it did malloc(X) so that the program can use Y bytes, > that would seem to only be a source of bugs and complexity. If the program > wants Y bytes, it should malloc(Y). Many folks in the thread seem to think > that this function would be used for a poor-man's realloc() wrapper or > something, and I think such uses would be short-sighted at best. If there > are other uses such as for having a debug malloc wrap the real one, then > that might justify the API, but it is unclear what a useful use of this API > would be. I can think of a few straightforward uses: 1) Suppose that we are about to write to a string that we know is malloc()ed, and we want to be sure that we aren't overflowing the allocated buffer. We can add an assertion that the buffer is indeed large enough to contain what is about to be written to it. 2) Suppose that we are using some sort of dynamically scalable data structure, such as a hash table that we double in size every time fullness crosses some threshold. In order to do that, we need to know how large the table is. We can store that information, or we could just query the size. (In this example, performance constraints might dictate that we actually store the size of the table, but there are certainly similar examples that wouldn't be so concerned with performance.) 3) This is what I want malloc_usable_size() for: garbage collection. In order for a garbage collector to know when it should run, there needs to be some way of tracking how much memory is in use. By using dlsym(RTLD_NEXT, ...), I can intercept all malloc/calloc/realloc/free calls and track current memory usage. However, unless there is a way of getting the size of each allocation, I don't know how much to subtract from the count for realloc/free calls. 4) Porting code from Linux. Case in point: Xara Xtreme, currently being ported by Vasil Dimov. At the moment, he has to use dlmalloc. Jason Following is what I've written for the malloc(3) man page: ---- The malloc_usable_size() function returns the usable size of the allocation pointed to by ptr. The return value may be larger than the size that was requested during allocation. malloc_usable_size() is not intended as a mechanism for in-place realloc(), though it can be abused that way; rather it is primarily provided as a tool for introspection purposes. Any discrepancy between the requested allocation size and the size reported by malloc_usable_size() should not be depended on, since such behavior is entirely implementation-dependent. ---- From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 17:49:05 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CC52816A401; Mon, 27 Mar 2006 17:49:05 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 74F3943D46; Mon, 27 Mar 2006 17:49:05 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [192.168.2.4]) ([10.251.60.37]) by a50.ironport.com with ESMTP; 27 Mar 2006 09:49:06 -0800 Message-ID: <44282590.9070304@elischer.org> Date: Mon, 27 Mar 2006 09:49:04 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.11) Gecko/20050727 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jason Evans References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> In-Reply-To: <44281421.3060401@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Vasil Dimov , Robert Watson , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 17:49:05 -0000 Jason Evans wrote: > John Baldwin wrote: > >> On Sunday 26 March 2006 13:04, Jason Evans wrote: >> >>> Robert Watson wrote: >>> >>>> I wonder if the intuitive objection people are raising is actually >>>> with the name. Since malloc_size() is defined on at least one >>>> platform to return the requested size, maybe a name like >>>> malloc_allocated_size() (or something a bit more compact) would >>>> help avoid that confusion, and make it clear that the consumer is >>>> getting back a commitment and not a hint for future realloc(), etc. >>> >>> >>> Maybe you're right. We could just call it malloc_usable_size() and >>> be compatible with Linux. >> >> >> It would help to know why such a function would be useful. :) Do you >> have >> a specific use-case? If the purpose is for a program to see that it >> really >> as Y >= X bytes when it did malloc(X) so that the program can use Y >> bytes, >> that would seem to only be a source of bugs and complexity. If the >> program >> wants Y bytes, it should malloc(Y). Many folks in the thread seem to >> think >> that this function would be used for a poor-man's realloc() wrapper or >> something, and I think such uses would be short-sighted at best. If >> there >> are other uses such as for having a debug malloc wrap the real one, then >> that might justify the API, but it is unclear what a useful use of >> this API >> would be. > > > I can think of a few straightforward uses: > > 1) Suppose that we are about to write to a string that we know is > malloc()ed, and we want to be sure that we aren't overflowing the > allocated buffer. We can add an assertion that the buffer is indeed > large enough to contain what is about to be written to it. Iff it retunrs a numbre greater than teh alocated size, then this is a programmer error.. he should have malloc'd a bigger buffer. On the other hand malloc_allocated_size (which returns the allocated size) might be usefull if a library which is given a buffer wants to know how much room there is in a buffer. Of course there is nothing saying that we gave the library a pointer to the BEGINNING of the buffer so it should report "distance from the pointer to the end". Doesn't work for strings off the stack (alloca()) or static, etc > > 2) Suppose that we are using some sort of dynamically scalable data > structure, such as a hash table that we double in size every time > fullness crosses some threshold. In order to do that, we need to know > how large the table is. We can store that information, or we could > just query the size. (In this example, performance constraints might > dictate that we actually store the size of the table, but there are > certainly similar examples that wouldn't be so concerned with > performance.) As you say.. just store it... though a library wouldn't have access to that. > > 3) This is what I want malloc_usable_size() for: garbage collection. > In order for a garbage collector to know when it should run, there > needs to be some way of tracking how much memory is in use. By using > dlsym(RTLD_NEXT, ...), I can intercept all malloc/calloc/realloc/free > calls and track current memory usage. However, unless there is a way > of getting the size of each allocation, I don't know how much to > subtract from the count for realloc/free calls. > > 4) Porting code from Linux. Case in point: Xara Xtreme, currently > being ported by Vasil Dimov. At the moment, he has to use dlmalloc. > > Jason > > Following is what I've written for the malloc(3) man page: > ---- > The malloc_usable_size() function returns the usable size of the > allocation pointed to by ptr. The return value may be larger than the > size that was requested during allocation. malloc_usable_size() is > not intended as a mechanism for in-place realloc(), though it can be > abused that way; rather it is primarily provided as a tool for > introspection purposes. Any discrepancy between the requested > allocation size and the size reported by malloc_usable_size() should > not be depended on, since such behavior is entirely > implementation-dependent. > ---- > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 18:50:18 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE4A816A426; Mon, 27 Mar 2006 18:50:18 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id 561F543D46; Mon, 27 Mar 2006 18:50:18 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gvs8kjs2lk8l0apa@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.4/8.13.3) with ESMTP id k2RIoHPp004926; Mon, 27 Mar 2006 10:50:17 -0800 (PST) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.4/8.13.3/Submit) id k2RIoHqS004925; Mon, 27 Mar 2006 10:50:17 -0800 (PST) (envelope-from jmg) Date: Mon, 27 Mar 2006 10:50:17 -0800 From: John-Mark Gurney To: Jason Evans Message-ID: <20060327185017.GF7001@funkthat.com> Mail-Followup-To: Jason Evans , John Baldwin , Vasil Dimov , Robert Watson , freebsd-arch@freebsd.org References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <44281421.3060401@FreeBSD.org> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: Vasil Dimov , Robert Watson , John Baldwin , freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 18:50:19 -0000 Jason Evans wrote this message on Mon, Mar 27, 2006 at 08:34 -0800: > John Baldwin wrote: > >On Sunday 26 March 2006 13:04, Jason Evans wrote: > > > >>Robert Watson wrote: > >> > >>>I wonder if the intuitive objection people are raising is actually with > >>>the name. Since malloc_size() is defined on at least one platform to > >>>return the requested size, maybe a name like malloc_allocated_size() (or > >>>something a bit more compact) would help avoid that confusion, and make > >>>it clear that the consumer is getting back a commitment and not a hint > >>>for future realloc(), etc. > >> > >>Maybe you're right. We could just call it malloc_usable_size() and be > >>compatible with Linux. > > > >It would help to know why such a function would be useful. :) Do you have > >a specific use-case? If the purpose is for a program to see that it really > >as Y >= X bytes when it did malloc(X) so that the program can use Y bytes, > >that would seem to only be a source of bugs and complexity. If the program > >wants Y bytes, it should malloc(Y). Many folks in the thread seem to think > >that this function would be used for a poor-man's realloc() wrapper or > >something, and I think such uses would be short-sighted at best. If there > >are other uses such as for having a debug malloc wrap the real one, then > >that might justify the API, but it is unclear what a useful use of this API > >would be. > > I can think of a few straightforward uses: [...] > 2) Suppose that we are using some sort of dynamically scalable data > structure, such as a hash table that we double in size every time > fullness crosses some threshold. In order to do that, we need to know > how large the table is. We can store that information, or we could just > query the size. (In this example, performance constraints might dictate > that we actually store the size of the table, but there are certainly > similar examples that wouldn't be so concerned with performance.) This use case is incorrect... a) you already have the size, since you have your hashing value around.. and b) you could end up with part of your has table uninitalized, say, alloc and initalize 24 bytes, the function returns 32, so you think, oh, well, bytes 25 through 32 are initalized, so I don't have to do anything.. This use case is correct if you used it as a poor man's realloc, where you allocated a block, then asked the size, but then, you may very well end up with a very bad hash value.. and in this case, you might as well realloc to this side.. > 4) Porting code from Linux. Case in point: Xara Xtreme, currently being > ported by Vasil Dimov. At the moment, he has to use dlmalloc. sad to say, but we seem to always inherit Linux's warts in the name of portability.. :( I'd say adopt a method similar to the ports tree, once we have 10+ ports depending upon it, then we can consider this.. but importing this function for a single program seems a bit excessive.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 20:19:32 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1A36616A401; Mon, 27 Mar 2006 20:19:32 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7DF5643D48; Mon, 27 Mar 2006 20:19:31 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from localhost (john@localhost [127.0.0.1]) by server.baldwin.cx (8.13.4/8.13.4) with ESMTP id k2RKJUUm063473; Mon, 27 Mar 2006 15:19:30 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: Jason Evans Date: Mon, 27 Mar 2006 15:20:08 -0500 User-Agent: KMail/1.9.1 References: <44247DF1.8000002@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> In-Reply-To: <44281421.3060401@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200603271520.11381.jhb@freebsd.org> X-Virus-Scanned: ClamAV 0.87.1/1358/Mon Mar 27 11:12:27 2006 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-3.8 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on server.baldwin.cx Cc: Vasil Dimov , Robert Watson , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 20:19:32 -0000 On Monday 27 March 2006 11:34, Jason Evans wrote: > John Baldwin wrote: > > On Sunday 26 March 2006 13:04, Jason Evans wrote: > > > >>Robert Watson wrote: > >> > >>>I wonder if the intuitive objection people are raising is actually with > >>>the name. Since malloc_size() is defined on at least one platform to > >>>return the requested size, maybe a name like malloc_allocated_size() (or > >>>something a bit more compact) would help avoid that confusion, and make > >>>it clear that the consumer is getting back a commitment and not a hint > >>>for future realloc(), etc. > >> > >>Maybe you're right. We could just call it malloc_usable_size() and be > >>compatible with Linux. > > > > It would help to know why such a function would be useful. :) Do you have > > a specific use-case? If the purpose is for a program to see that it really > > as Y >= X bytes when it did malloc(X) so that the program can use Y bytes, > > that would seem to only be a source of bugs and complexity. If the program > > wants Y bytes, it should malloc(Y). Many folks in the thread seem to think > > that this function would be used for a poor-man's realloc() wrapper or > > something, and I think such uses would be short-sighted at best. If there > > are other uses such as for having a debug malloc wrap the real one, then > > that might justify the API, but it is unclear what a useful use of this API > > would be. > > I can think of a few straightforward uses: > > 1) Suppose that we are about to write to a string that we know is > malloc()ed, and we want to be sure that we aren't overflowing the > allocated buffer. We can add an assertion that the buffer is indeed > large enough to contain what is about to be written to it. In that case though you really want to know the size you allocated, not necessarily the backing-store size, esp. if you are in a library since your caller may (correctly) assume that if it allocates X bytes for a string the library will only write up to X bytes. So, I think in that case you want the requested size, not the backing store size. > 2) Suppose that we are using some sort of dynamically scalable data > structure, such as a hash table that we double in size every time > fullness crosses some threshold. In order to do that, we need to know > how large the table is. We can store that information, or we could just > query the size. (In this example, performance constraints might dictate > that we actually store the size of the table, but there are certainly > similar examples that wouldn't be so concerned with performance.) I think this devolves into the poor-mans realloc. :) Instead, the code should just realloc() when necessary and rely on the system malloc to know when it is safe for the realloc() to be done in place. > 3) This is what I want malloc_usable_size() for: garbage collection. In > order for a garbage collector to know when it should run, there needs to > be some way of tracking how much memory is in use. By using > dlsym(RTLD_NEXT, ...), I can intercept all malloc/calloc/realloc/free > calls and track current memory usage. However, unless there is a way of > getting the size of each allocation, I don't know how much to subtract > from the count for realloc/free calls. This is a good reason, and is what I hoped you had in mind when you proposed it. :) I think that other than this type of code though (malloc reimplementations or wrappers), nothing else should use this function. :) > 4) Porting code from Linux. Case in point: Xara Xtreme, currently being > ported by Vasil Dimov. At the moment, he has to use dlmalloc. Does it contain a G/C of its own or some such? > Jason > > Following is what I've written for the malloc(3) man page: > ---- > The malloc_usable_size() function returns the usable size of the > allocation pointed to by ptr. The return value may be larger than the > size that was requested during allocation. malloc_usable_size() is not > intended as a mechanism for in-place realloc(), though it can be abused > that way; rather it is primarily provided as a tool for introspection > purposes. Any discrepancy between the requested allocation size and the > size reported by malloc_usable_size() should not be depended on, since > such behavior is entirely implementation-dependent. > ---- I would word it stronger: "malloc_usable_size() should not be used as a mechanism for in-place realloc(). It is provided solely as a tool for introspection purposes." :) -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 20:51:32 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6733816A401; Mon, 27 Mar 2006 20:51:32 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id F1F5F43D48; Mon, 27 Mar 2006 20:51:31 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id D26925E4906; Mon, 27 Mar 2006 12:51:31 -0800 (PST) Received: from [129.101.167.145] (unknown [129.101.167.145]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id 2A8925E48D7; Mon, 27 Mar 2006 12:51:30 -0800 (PST) In-Reply-To: <20060327185017.GF7001@funkthat.com> References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <20060327185017.GF7001@funkthat.com> Mime-Version: 1.0 (Apple Message framework v746.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <352D1430-CC72-47EE-9E50-B1C4404ACA95@FreeBSD.org> Content-Transfer-Encoding: 7bit From: Jason Evans Date: Mon, 27 Mar 2006 12:52:17 -0800 To: John-Mark Gurney X-Mailer: Apple Mail (2.746.2) X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=failed version=3.0.5 Cc: Robert Watson , Poul-Henning Kamp , John Baldwin , freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 20:51:32 -0000 On Mar 27, 2006, at 10:50 AM, John-Mark Gurney wrote: > Jason Evans wrote this message on Mon, Mar 27, 2006 at 08:34 -0800: >> >> I can think of a few straightforward uses: > > [...] > >> 2) [...] > > This use case is incorrect... Fair enough. That leaves three use cases. >> 4) Porting code from Linux. Case in point: Xara Xtreme, currently >> being >> ported by Vasil Dimov. At the moment, he has to use dlmalloc. > > sad to say, but we seem to always inherit Linux's warts in the name of > portability.. :( I'd say adopt a method similar to the ports tree, > once > we have 10+ ports depending upon it, then we can consider this.. but > importing this function for a single program seems a bit excessive.. malloc_usable_size() is *not* a Linux wart. Instead, it mostly[1] addresses a fundamental flaw in the malloc(3) API. We must go back much farther than Linux if we really want to start disparaging malloc (3). Consider that the allocator must know how large allocations are so that it can do the right thing for realloc() and free(). Consider also that the application must know how large its objects are so that it can stay in bounds. That means that such information has to be stored in two places. Following is a much more reasonable API that doesn't burden the allocator with information that the application should be able to provide: void * malloc(size_t size); void * calloc(size_t size); void * realloc(void *ptr, size_t size, size_t old_size); void free(void *ptr, size_t size); That's not what we have though, and we're certainly not going to completely redesign malloc(3) at this point. So, instead, we're faced with trying to patch things up as well as is feasible without breaking compatibility. Let's re-visit Poul-Henning's suggestion, which involves a parallel API that exposes the internal sizes to the application. The following is not identical to what he proposed, but it has the same semantics: void * malloc_np(size_t *size); void * calloc_np(size_t *size); void * realloc_np(void *ptr, size_t *size, size_t *old_size); void * memalign_np(size_t alignment, size_t *size); void * free_np(void *ptr, size_t *size); This serves much the same purpose as the idealized malloc(3) API I outlined above but with the advantage that it can co-exist with the current malloc(3) API. From an engineering perspective, this is probably actually a better solution than malloc_usable_size(), but 1) it's a much broader API, 2) it's nothing like what any other OSes have done, and 3) in practice it really doesn't gain us anything over malloc_usable_size() (in fact it doesn't solve the problem of porting Linux and OS X software). For my own personal needs, I really don't care whether we add malloc_usable_size() or the broader API described above. However, I do feel like the latter is an over-engineered solution to the problem. To me, this isn't about reinventing a perfect wheel -- it's about filing off a rough edge on the stone wheel we're stuck with. Jason [1] In a slightly more ideal world, we would add malloc_size() instead of malloc_usable_size(), which would return the size that was passed to malloc()/calloc()/realloc(). However, that would impose implementation constraints on the allocator that would have far- reaching implications. I don't want to get into the details here, but suffice it to say that requiring support for malloc_size() would require a lot of extra space overhead for many allocator designs, including for phkmalloc and jemalloc. From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 20:57:02 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 203FF16A401; Mon, 27 Mar 2006 20:57:02 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id D965843D69; Mon, 27 Mar 2006 20:57:00 +0000 (GMT) (envelope-from jasone@freebsd.org) Received: by lh.synack.net (Postfix, from userid 100) id C27065E48D7; Mon, 27 Mar 2006 12:57:00 -0800 (PST) Received: from [129.101.167.145] (unknown [129.101.167.145]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id 1DADE5E48D7; Mon, 27 Mar 2006 12:57:00 -0800 (PST) In-Reply-To: <200603271520.11381.jhb@freebsd.org> References: <44247DF1.8000002@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <200603271520.11381.jhb@freebsd.org> Mime-Version: 1.0 (Apple Message framework v746.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Jason Evans Date: Mon, 27 Mar 2006 12:57:51 -0800 To: John Baldwin X-Mailer: Apple Mail (2.746.2) X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=failed version=3.0.5 Cc: Vasil Dimov , Robert Watson , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 20:57:02 -0000 On Mar 27, 2006, at 12:20 PM, John Baldwin wrote: > On Monday 27 March 2006 11:34, Jason Evans wrote: >> I can think of a few straightforward uses: >> >> 1) Suppose that we are about to write to a string that we know is >> malloc()ed, and we want to be sure that we aren't overflowing the >> allocated buffer. We can add an assertion that the buffer is indeed >> large enough to contain what is about to be written to it. > > In that case though you really want to know the size you allocated, > not necessarily the backing-store size, esp. if you are in a library > since your caller may (correctly) assume that if it allocates X > bytes for a string the library will only write up to X bytes. So, I > think in that case you want the requested size, not the backing store > size. Yes, you're right that we actually want the requested size rather than the backing-store size. This is only a "better than nothing" use case. >> 4) Porting code from Linux. Case in point: Xara Xtreme, currently >> being >> ported by Vasil Dimov. At the moment, he has to use dlmalloc. > > Does it contain a G/C of its own or some such? I don't know the answer to that. >> Following is what I've written for the malloc(3) man page: >> ---- >> The malloc_usable_size() function returns the usable size of the >> allocation pointed to by ptr. The return value may be larger than >> the >> size that was requested during allocation. malloc_usable_size() >> is not >> intended as a mechanism for in-place realloc(), though it can be >> abused >> that way; rather it is primarily provided as a tool for introspection >> purposes. Any discrepancy between the requested allocation size >> and the >> size reported by malloc_usable_size() should not be depended on, >> since >> such behavior is entirely implementation-dependent. >> ---- > > I would word it stronger: "malloc_usable_size() should not be used > as a > mechanism for in-place realloc(). It is provided solely as a tool for > introspection purposes." Okay, sounds fine to me. Thanks, Jason From owner-freebsd-arch@FreeBSD.ORG Mon Mar 27 21:20:54 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 70BA516A422; Mon, 27 Mar 2006 21:20:54 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id D510343D46; Mon, 27 Mar 2006 21:20:53 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (t8jxfldi4nygv0jp@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.4/8.13.3) with ESMTP id k2RLKrbj008982; Mon, 27 Mar 2006 13:20:53 -0800 (PST) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.4/8.13.3/Submit) id k2RLKrv1008981; Mon, 27 Mar 2006 13:20:53 -0800 (PST) (envelope-from jmg) Date: Mon, 27 Mar 2006 13:20:52 -0800 From: John-Mark Gurney To: John Baldwin Message-ID: <20060327212052.GK7001@funkthat.com> Mail-Followup-To: John Baldwin , Jason Evans , Vasil Dimov , Robert Watson , freebsd-arch@freebsd.org References: <44247DF1.8000002@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <200603271520.11381.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200603271520.11381.jhb@freebsd.org> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: Vasil Dimov , Robert Watson , Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2006 21:20:54 -0000 John Baldwin wrote this message on Mon, Mar 27, 2006 at 15:20 -0500: > On Monday 27 March 2006 11:34, Jason Evans wrote: > > Following is what I've written for the malloc(3) man page: > > ---- > > The malloc_usable_size() function returns the usable size of the > > allocation pointed to by ptr. The return value may be larger than the > > size that was requested during allocation. malloc_usable_size() is not > > intended as a mechanism for in-place realloc(), though it can be abused > > that way; rather it is primarily provided as a tool for introspection > > purposes. Any discrepancy between the requested allocation size and the > > size reported by malloc_usable_size() should not be depended on, since > > such behavior is entirely implementation-dependent. > > ---- > > I would word it stronger: "malloc_usable_size() should not be used as a ^^^^^^ shall > mechanism for in-place realloc(). It is provided solely as a tool for > introspection purposes." should is just a suggestion, it isn't strong enough.. I would also hope that the appropriate test cases are added so that any uses of memory beyond the allocated sizes (and returned by usable_size), are ensured not to have been touched, and that we abort() in cases that someone used memory outside their allocation size... As long as it returns a value that shall not extend the allocation size beyond what was allocated by realloc/malloc/calloc, then it's fine... If it requires a realloc in order to use the additional space, then I'm fine with it... I wouldn't mind an implied behavior that you can do a realloc to malloc_usable_size and it shall not incure a copy.. it'd be useful temp buffers that you grow so you can use the max of the allocated space, instead of waste a bit of unalloced memory... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 05:24:23 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 840E716A41F; Tue, 28 Mar 2006 05:24:23 +0000 (UTC) (envelope-from vd@datamax.bg) Received: from jengal.datamax.bg (jengal.datamax.bg [82.103.104.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0BDFE43D48; Tue, 28 Mar 2006 05:24:22 +0000 (GMT) (envelope-from vd@datamax.bg) Received: from qlovarnika.bg.datamax (qlovarnika.bg.datamax [192.168.10.2]) by jengal.datamax.bg (Postfix) with SMTP id 3D023B857; Tue, 28 Mar 2006 08:24:21 +0300 (EEST) Received: (nullmailer pid 97374 invoked by uid 1002); Tue, 28 Mar 2006 05:24:21 -0000 Date: Tue, 28 Mar 2006 08:24:21 +0300 From: Vasil Dimov To: Jason Evans Message-ID: <20060328052421.GA97222@qlovarnika.bg.datamax> References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="xHFwDpU9dbj6ez1V" Content-Disposition: inline In-Reply-To: <44281421.3060401@FreeBSD.org> X-OS: FreeBSD 6.0-STABLE User-Agent: Mutt/1.5.11 Cc: Robert Watson , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: vd@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 05:24:23 -0000 --xHFwDpU9dbj6ez1V Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 27, 2006 at 08:34:41AM -0800, Jason Evans wrote: [...] > Following is what I've written for the malloc(3) man page: > ---- > The malloc_usable_size() function returns the usable size of the=20 > allocation pointed to by ptr. The return value may be larger than the=20 > size that was requested during allocation. malloc_usable_size() is not= =20 > intended as a mechanism for in-place realloc(), though it can be abused= =20 > that way; rather it is primarily provided as a tool for introspection=20 > purposes. Any discrepancy between the requested allocation size and the= =20 > size reported by malloc_usable_size() should not be depended on, since=20 > such behavior is entirely implementation-dependent. > ---- Although this is obvious it should mention that the pointer passed must have been returned by a malloc() or realloc() call or unpredictable things will happen, right? --=20 Vasil Dimov gro.DSBeerF@dv Testing can show the presence of bugs, but not their absence. -- Edsger W. Dijkstra --xHFwDpU9dbj6ez1V Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- iD8DBQFEKMiFFw6SP/bBpCARAkXJAJ4rJIhi37k9Q/to78mu25VxYsyxOgCfVZZb VFYpZp3QmgfHlaPLc5Hqn9o= =0Y2J -----END PGP SIGNATURE----- --xHFwDpU9dbj6ez1V-- From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 06:07:16 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A420016A41F; Tue, 28 Mar 2006 06:07:16 +0000 (UTC) (envelope-from vd@datamax.bg) Received: from jengal.datamax.bg (jengal.datamax.bg [82.103.104.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2A0D043D48; Tue, 28 Mar 2006 06:07:16 +0000 (GMT) (envelope-from vd@datamax.bg) Received: from qlovarnika.bg.datamax (qlovarnika.bg.datamax [192.168.10.2]) by jengal.datamax.bg (Postfix) with SMTP id E685BB857; Tue, 28 Mar 2006 09:07:14 +0300 (EEST) Received: (nullmailer pid 97643 invoked by uid 1002); Tue, 28 Mar 2006 06:07:14 -0000 Date: Tue, 28 Mar 2006 09:07:14 +0300 From: Vasil Dimov To: John Baldwin Message-ID: <20060328060714.GB97222@qlovarnika.bg.datamax> References: <44247DF1.8000002@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <200603271520.11381.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GID0FwUMdk1T2AWN" Content-Disposition: inline In-Reply-To: <200603271520.11381.jhb@freebsd.org> X-OS: FreeBSD 6.0-STABLE User-Agent: Mutt/1.5.11 Cc: Robert Watson , Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: vd@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 06:07:16 -0000 --GID0FwUMdk1T2AWN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 27, 2006 at 03:20:08PM -0500, John Baldwin wrote: > On Monday 27 March 2006 11:34, Jason Evans wrote: [...] > > 4) Porting code from Linux. Case in point: Xara Xtreme, currently bein= g=20 > > ported by Vasil Dimov. At the moment, he has to use dlmalloc. >=20 > Does it contain a G/C of its own or some such? >=20 Here is what malloc_usable_size() is used for in Xara Xtreme: * They keep track of how much (heap) memory is available. Maybe they support an option like "do not use more that N MiB". This is achieved by a wrappers to realloc() and free() which manipulate a global? variable AvailableRAM - the realloc() wrapper decreases AvailableRAM with (newsize - oldsize) where oldsize is retrieved with malloc_usable_size(). The free() wrapper increases AvailableRAM with what is returned by malloc_usable_size(). * Debugging purposes like keeping track of how much memory they allocated, the filename and linenumber of the allocation * They use a function similar to realloc(), but which receives the number of bytes they need to increment the buffer with, rather than the new size, so they call realloc(p,increment+malloc_usable_size(p)) * A lot of other miscellaneous purposes where they should really keep track of how many bytes they requested from malloc() rather than calling malloc_usable_size(). --=20 Vasil Dimov gro.DSBeerF@dv Testing can show the presence of bugs, but not their absence. -- Edsger W. Dijkstra --GID0FwUMdk1T2AWN Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- iD8DBQFEKNKSFw6SP/bBpCARApOSAJ91DRTph0Jb5ynCUDFnT8Yx+e8gogCeNcuS nQCPJsUyC/K/Mi6EbEkvyH0= =VfHa -----END PGP SIGNATURE----- --GID0FwUMdk1T2AWN-- From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 07:03:50 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4F9CE16A400 for ; Tue, 28 Mar 2006 07:03:50 +0000 (UTC) (envelope-from rink@rink.nu) Received: from mx0.rink.nu (morion.rink.nu [80.112.228.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id A3EF543D46 for ; Tue, 28 Mar 2006 07:03:49 +0000 (GMT) (envelope-from rink@rink.nu) Received: from localhost (localhost [127.0.0.1]) by mx0.rink.nu (Postfix) with ESMTP id 5B02F7B for ; Tue, 28 Mar 2006 09:03:40 +0200 (CEST) Received: from mx0.rink.nu ([127.0.0.1]) by localhost (morion.rink.nu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 58015-09 for ; Tue, 28 Mar 2006 09:03:33 +0200 (CEST) Received: by mx0.rink.nu (Postfix, from userid 1678) id 0B6BC7A; Tue, 28 Mar 2006 09:03:32 +0200 (CEST) Date: Tue, 28 Mar 2006 09:03:32 +0200 From: Rink Springer To: arch@FreeBSD.org Message-ID: <20060328070332.GA53789@rink.nu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AhhlLboLdkugWU4S" Content-Disposition: inline User-Agent: Mutt/1.5.11 X-Virus-Scanned: amavisd-new at rink.nu Cc: Subject: [RFC] Syslogd service patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 07:03:50 -0000 --AhhlLboLdkugWU4S Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi everyone, I've devised a patch for syslogd, which allows you to specify a service name when logging remotely. This is all embedded in the current syntax. For logging remote under a different service, you can use @1.2.3.4:service [1]; for binding to a different service, you can use 'syslogd -b 1.2.3.4:service'. No breakage is introduced as the default service remains 'syslog'. The patch is at http://rink.nu/tmp/syslogd.diff, which I'd like to commit. Does anyone have questions or comments? [1] OpenBSD also has this functionality --=20 Rink P.W. Springer - http://rink.nu "Richter: Tribute? You steal men's souls, and make them your slaves! Dracula: Perhaps the same could be said of all religions." - Castlevania: Symphony of the Night --AhhlLboLdkugWU4S Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (FreeBSD) iD8DBQFEKN/Eb3O60uztv/8RAj6pAJ4iuxTLSL+HGWz58/aK5vUFwjmTUQCgkV4j cpSt4TCdGWIum6DPHbQS7Zc= =QzJd -----END PGP SIGNATURE----- --AhhlLboLdkugWU4S-- From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 09:11:59 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A9ABE16A401; Tue, 28 Mar 2006 09:11:59 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail22.syd.optusnet.com.au (mail22.syd.optusnet.com.au [211.29.133.160]) by mx1.FreeBSD.org (Postfix) with ESMTP id EBAAD43D48; Tue, 28 Mar 2006 09:11:58 +0000 (GMT) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-19-236.belrs4.nsw.optusnet.com.au [220.239.19.236]) by mail22.syd.optusnet.com.au (8.12.11/8.12.11) with ESMTP id k2S9BrwK000747 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Tue, 28 Mar 2006 20:11:56 +1100 Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.13.4/8.13.4) with ESMTP id k2S9Br7W002164; Tue, 28 Mar 2006 20:11:53 +1100 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.13.4/8.13.4/Submit) id k2S9Brg2002163; Tue, 28 Mar 2006 20:11:53 +1100 (EST) (envelope-from peter) Date: Tue, 28 Mar 2006 20:11:53 +1100 From: Peter Jeremy To: Jason Evans Message-ID: <20060328091153.GC961@turion.vk2pj.dyndns.org> References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <20060327185017.GF7001@funkthat.com> <352D1430-CC72-47EE-9E50-B1C4404ACA95@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <352D1430-CC72-47EE-9E50-B1C4404ACA95@FreeBSD.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.11 Cc: freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 09:11:59 -0000 On Mon, 2006-Mar-27 12:52:17 -0800, Jason Evans wrote: >void * >malloc(size_t size); > >void * >calloc(size_t size); > >void * >realloc(void *ptr, size_t size, size_t old_size); > >void >free(void *ptr, size_t size); ISTR AmigaDOS did something like this (my autodocs are packed away somewhere so I can't quickly check). The downside is that you have to keep the malloc() size stashed away so you can pass it to free(). This could be problematic where a function returns malloc'd memory (eg asprintf(), strdup()). It also fails to solve the leak from: ptr = realloc(ptr, NEW_SIZE); where realloc() returns NULL. In an ideal world, a pointer would be an [address, size] pair (or even [size, address, type] tuple) so that any the bounds (and type) can be verified by anything that wants to. (The iAPX432 tried this). I think we've got even less chance of re-designing C than re-designing malloc (and C++ can't do this because you can't override builtin types). >breaking compatibility. Let's re-visit Poul-Henning's suggestion, >which involves a parallel API that exposes the internal sizes to the >application. The following is not identical to what he proposed, but >it has the same semantics: > >void * >malloc_np(size_t *size); > >void * >calloc_np(size_t *size); > >void * >realloc_np(void *ptr, size_t *size, size_t *old_size); > >void * >memalign_np(size_t alignment, size_t *size); > >void * >free_np(void *ptr, size_t *size); A try_realloc() function would be nice - where you can determine whether the realloc succeeded or not (or maybe how much memory it could obtain) without accidently leaking the old region if the realloc() failed. >[1] In a slightly more ideal world, we would add malloc_size() >instead of malloc_usable_size(), which would return the size that was >passed to malloc()/calloc()/realloc(). However, that would impose >implementation constraints on the allocator that would have far- >reaching implications. I don't want to get into the details here, >but suffice it to say that requiring support for malloc_size() would >require a lot of extra space overhead for many allocator designs, >including for phkmalloc and jemalloc. If you really want to detect off-by-one errors, realloc() and free() need to know exactly how many bytes were asked for so they can verify that the immediately following byte still has the magic value that malloc/realloc wrote there. (Admittedly, this is for debugging only). -- Peter Jeremy From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 09:55:05 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6FB8916A420; Tue, 28 Mar 2006 09:55:05 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 29E5443D46; Tue, 28 Mar 2006 09:55:04 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id E3B9C1EC352; Tue, 28 Mar 2006 11:55:02 +0200 (CEST) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.4/8.13.4) with ESMTP id k2S9t0Ed061461; Tue, 28 Mar 2006 11:55:02 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Peter Jeremy From: "Poul-Henning Kamp" In-Reply-To: Your message of "Tue, 28 Mar 2006 20:11:53 +1100." <20060328091153.GC961@turion.vk2pj.dyndns.org> Date: Tue, 28 Mar 2006 11:55:00 +0200 Message-ID: <61460.1143539700@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 09:55:05 -0000 In message <20060328091153.GC961@turion.vk2pj.dyndns.org>, Peter Jeremy writes: >In an ideal world, a pointer would be an [address, size] pair (or even >[size, address, type] tuple) so that any the bounds (and type) can be >verified by anything that wants to. (The iAPX432 tried this). The first computer to actually _do_ this was Linn's "Rekursiv" computer, which had the microcode manage in-memory objects, including swapping to/from backing store. http://www.brouhaha.com/~eric/retrocomputing/rekursiv/ -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 10:06:28 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A8B3816A400 for ; Tue, 28 Mar 2006 10:06:28 +0000 (UTC) (envelope-from andrew@areilly.bpc-users.org) Received: from omta05sl.mx.bigpond.com (omta05sl.mx.bigpond.com [144.140.93.195]) by mx1.FreeBSD.org (Postfix) with ESMTP id 08FE643D46 for ; Tue, 28 Mar 2006 10:06:27 +0000 (GMT) (envelope-from andrew@areilly.bpc-users.org) Received: from areilly.bpc-users.org ([141.168.4.160]) by omta05sl.mx.bigpond.com with ESMTP id <20060328100626.SLSH25409.omta05sl.mx.bigpond.com@areilly.bpc-users.org> for ; Tue, 28 Mar 2006 10:06:26 +0000 Received: (qmail 88510 invoked by uid 501); 28 Mar 2006 10:07:35 -0000 Date: Tue, 28 Mar 2006 21:07:35 +1100 From: Andrew Reilly To: Poul-Henning Kamp Message-ID: <20060328100735.GA87799@gurney.reilly.home> References: <20060328091153.GC961@turion.vk2pj.dyndns.org> <61460.1143539700@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <61460.1143539700@critter.freebsd.dk> User-Agent: Mutt/1.4.2.1i Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 10:06:28 -0000 On Tue, Mar 28, 2006 at 11:55:00AM +0200, Poul-Henning Kamp wrote: > In message <20060328091153.GC961@turion.vk2pj.dyndns.org>, Peter Jeremy writes: > > > >In an ideal world, a pointer would be an [address, size] pair (or even > >[size, address, type] tuple) so that any the bounds (and type) can be > >verified by anything that wants to. (The iAPX432 tried this). > > The first computer to actually _do_ this was Linn's "Rekursiv" computer, > which had the microcode manage in-memory objects, including swapping > to/from backing store. > > http://www.brouhaha.com/~eric/retrocomputing/rekursiv/ Are you sure that was the first? I'm pretty sure that IBM's "Future System" (FS), and System/36 and AS/400 that came from it probably got there sooner. I think that Burroughs A-series mainframes did much the same thing too. They might not have had type tags, but they were constrained and had sizes. And don't forget, the Intel 286 learned it's segment tricks from iAPX432. Generally pretty unpleasant environments for C, IMO, but good for the other languages of the day. Lisp and smalltalk (Xerox) machines probably count, too. Dunno where the hardware versions come with respect to the rekursiv in time, though. Today, the Java and .NET runtimes have all that you could want, from this sort of functionality. It seems kind of broken to be trying to retro-fit it into C. I've just had a big argument in c.l.c/c.a.e about the issue, and I consider that I lost it, so I'm a bit sensitive about it... -- Andrew From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 10:19:55 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 85ED516A400 for ; Tue, 28 Mar 2006 10:19:55 +0000 (UTC) (envelope-from andrew@areilly.bpc-users.org) Received: from omta03sl.mx.bigpond.com (omta03sl.mx.bigpond.com [144.140.92.155]) by mx1.FreeBSD.org (Postfix) with ESMTP id CF2CD43D45 for ; Tue, 28 Mar 2006 10:19:54 +0000 (GMT) (envelope-from andrew@areilly.bpc-users.org) Received: from areilly.bpc-users.org ([141.168.4.160]) by omta03sl.mx.bigpond.com with ESMTP id <20060328101953.KYZN1358.omta03sl.mx.bigpond.com@areilly.bpc-users.org> for ; Tue, 28 Mar 2006 10:19:53 +0000 Received: (qmail 88617 invoked by uid 501); 28 Mar 2006 10:20:59 -0000 Date: Tue, 28 Mar 2006 21:20:59 +1100 From: Andrew Reilly To: Jason Evans Message-ID: <20060328102059.GB87799@gurney.reilly.home> References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <20060327185017.GF7001@funkthat.com> <352D1430-CC72-47EE-9E50-B1C4404ACA95@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <352D1430-CC72-47EE-9E50-B1C4404ACA95@FreeBSD.org> User-Agent: Mutt/1.4.2.1i Cc: John-Mark Gurney , Robert Watson , Poul-Henning Kamp , John Baldwin , freebsd-arch@FreeBSD.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 10:19:55 -0000 On Mon, Mar 27, 2006 at 12:52:17PM -0800, Jason Evans wrote: > [1] In a slightly more ideal world, we would add malloc_size() > instead of malloc_usable_size(), which would return the size that was > passed to malloc()/calloc()/realloc(). However, that would impose > implementation constraints on the allocator that would have far- > reaching implications. You could do this in your malloc wrapper, if you wanted: just have it maintain a (growable) hash table of sizes, keyed by allocation address. You'd return the real (asked-for) size as the malloc_useable_size(), but any client code of that function would have to behave sensibly in any case: it can't know that it isn't running against something like a traditional first-fit allocator or the like that doesn't round up. > I don't want to get into the details here, > but suffice it to say that requiring support for malloc_size() would > require a lot of extra space overhead for many allocator designs, > including for phkmalloc and jemalloc. I've just written my own allocator for an embedded project (extra constraints are often better served by changing API, imo.) Anyway, it's very simple, and malloc_useable_size would be quite slow; malloc_size would be impossible without something extra, like the hash-table that I mentioned. Incidentally, what should malloc_useable_size() do if passed a pointer that wasn't [cmr]alloc()'d? The same sort of warning/trap as free()? -- Andrew From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 10:31:52 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D924E16A401 for ; Tue, 28 Mar 2006 10:31:52 +0000 (UTC) (envelope-from andrew@areilly.bpc-users.org) Received: from omta01sl.mx.bigpond.com (omta01sl.mx.bigpond.com [144.140.92.153]) by mx1.FreeBSD.org (Postfix) with ESMTP id 57E4743D6B for ; Tue, 28 Mar 2006 10:31:51 +0000 (GMT) (envelope-from andrew@areilly.bpc-users.org) Received: from areilly.bpc-users.org ([141.168.4.160]) by omta01sl.mx.bigpond.com with ESMTP id <20060328103150.IODJ25377.omta01sl.mx.bigpond.com@areilly.bpc-users.org> for ; Tue, 28 Mar 2006 10:31:50 +0000 Received: (qmail 88747 invoked by uid 501); 28 Mar 2006 10:32:51 -0000 Date: Tue, 28 Mar 2006 21:32:51 +1100 From: Andrew Reilly To: Vasil Dimov Message-ID: <20060328103251.GC87799@gurney.reilly.home> References: <44247DF1.8000002@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <200603271520.11381.jhb@freebsd.org> <20060328060714.GB97222@qlovarnika.bg.datamax> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060328060714.GB97222@qlovarnika.bg.datamax> User-Agent: Mutt/1.4.2.1i Cc: Robert Watson , Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 10:31:52 -0000 On Tue, Mar 28, 2006 at 09:07:14AM +0300, Vasil Dimov wrote: > On Mon, Mar 27, 2006 at 03:20:08PM -0500, John Baldwin wrote: > > On Monday 27 March 2006 11:34, Jason Evans wrote: > [...] > > > 4) Porting code from Linux. Case in point: Xara Xtreme, currently being > > > ported by Vasil Dimov. At the moment, he has to use dlmalloc. > > > > Does it contain a G/C of its own or some such? > > > > Here is what malloc_usable_size() is used for in Xara Xtreme: > * They keep track of how much (heap) memory is available. Maybe they > support an option like "do not use more that N MiB". This is achieved > by a wrappers to realloc() and free() which manipulate a global? > variable AvailableRAM - the realloc() wrapper decreases AvailableRAM > with (newsize - oldsize) where oldsize is retrieved with > malloc_usable_size(). The free() wrapper increases AvailableRAM with > what is returned by malloc_usable_size(). oh, yuck! How do they deal with malloc_usable_size() returning different answers for different calls, depending on the intervening history of allocator activity? What if malloc_usable_size() is not the same as the amount of memory that would be freed on free()? (This would certainly be the case if a traditional first-fit or best-fit algorithm was sitting underneath.) Or is it the definition supposed to encompas that situation (so that malloc_usable_size is the same as the argument to malloc(), even if there happens to be free memory after the allocation, that a realloc would use without copy? What if, instead, it was linked against a garbage colector, where free() might not actually do anything at all? > * Debugging purposes like keeping track of how much memory they > allocated, the filename and linenumber of the allocation > * They use a function similar to realloc(), but which receives the > number of bytes they need to increment the buffer with, rather than > the new size, so they call realloc(p,increment+malloc_usable_size(p)) > * A lot of other miscellaneous purposes where they should really keep > track of how many bytes they requested from malloc() rather than > calling malloc_usable_size(). It certainly sounds as though the correct way to deal with this application is to link it against it's preferred allocator, rather than the system one. It's obvious that it relies on a great deal of specific allocator behaviour. -- Andrew From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 10:39:11 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6D4AF16A449; Tue, 28 Mar 2006 10:39:11 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8FB7B43D4C; Tue, 28 Mar 2006 10:39:10 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id 8A12C1EC30C; Tue, 28 Mar 2006 12:39:08 +0200 (CEST) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.4/8.13.4) with ESMTP id k2SAd6Nn061636; Tue, 28 Mar 2006 12:39:08 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Andrew Reilly From: "Poul-Henning Kamp" In-Reply-To: Your message of "Tue, 28 Mar 2006 21:07:35 +1100." <20060328100735.GA87799@gurney.reilly.home> Date: Tue, 28 Mar 2006 12:39:06 +0200 Message-ID: <61635.1143542346@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 10:39:11 -0000 In message <20060328100735.GA87799@gurney.reilly.home>, Andrew Reilly writes: >On Tue, Mar 28, 2006 at 11:55:00AM +0200, Poul-Henning Kamp wrote: >> In message <20060328091153.GC961@turion.vk2pj.dyndns.org>, Peter Jeremy writes: >> >In an ideal world, a pointer would be an [address, size] pair (or even >> >[size, address, type] tuple) so that any the bounds (and type) can be >> >verified by anything that wants to. (The iAPX432 tried this). >> >> The first computer to actually _do_ this was Linn's "Rekursiv" computer, >> which had the microcode manage in-memory objects, including swapping >> to/from backing store. >> >> http://www.brouhaha.com/~eric/retrocomputing/rekursiv/ > >Are you sure that was the first? > >I'm pretty sure that IBM's "Future System" (FS), and System/36 and >AS/400 that came from it probably got there sooner. Nope, not even close. What Recursiv did was novel in that the objects lived in hardware and microcode, there were no way to overrun an array, because you simply couldn't index it outside it's bounds. Not even in the operating system. The fact that all objects were referenced by their object ID also meant that they could be put or moved anywhere convenient, with all references being by ID, it would be found when needed. It obviously comes with a significant performance penalty, not unlike page table lookups. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 10:57:25 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0651D16A401 for ; Tue, 28 Mar 2006 10:57:25 +0000 (UTC) (envelope-from andrew@areilly.bpc-users.org) Received: from omta01sl.mx.bigpond.com (omta01sl.mx.bigpond.com [144.140.92.153]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3F22343D46 for ; Tue, 28 Mar 2006 10:57:24 +0000 (GMT) (envelope-from andrew@areilly.bpc-users.org) Received: from areilly.bpc-users.org ([141.168.4.160]) by omta01sl.mx.bigpond.com with ESMTP id <20060328105723.JKOM25377.omta01sl.mx.bigpond.com@areilly.bpc-users.org> for ; Tue, 28 Mar 2006 10:57:23 +0000 Received: (qmail 89091 invoked by uid 501); 28 Mar 2006 10:58:33 -0000 Date: Tue, 28 Mar 2006 21:58:33 +1100 From: Andrew Reilly To: Poul-Henning Kamp Message-ID: <20060328105833.GE87799@gurney.reilly.home> References: <20060328100735.GA87799@gurney.reilly.home> <61635.1143542346@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <61635.1143542346@critter.freebsd.dk> User-Agent: Mutt/1.4.2.1i Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 10:57:25 -0000 On Tue, Mar 28, 2006 at 12:39:06PM +0200, Poul-Henning Kamp wrote: > Nope, not even close. > > What Recursiv did was novel in that the objects lived in hardware > and microcode, there were no way to overrun an array, because you > simply couldn't index it outside it's bounds. Not even in the > operating system. How is that different from all of the preceding segmented machines like (to quote your .sig) the Z8001+MMU? (Or, later, the i286). (Besides associating a type tag with the segment, which is something that lisp and smalltalk machines did.) > The fact that all objects were referenced by their object ID also > meant that they could be put or moved anywhere convenient, with > all references being by ID, it would be found when needed. A segment ID doesn't seem to be too radically different from an object ID. You can't access outside the bounds of the segment on most segmented architectures, and you can put them or page them wherever you like, because it's the segment descriptor that tells you where in physical memory the segment is, not the segment ID. Application code can't get to see that. > It obviously comes with a significant performance penalty, not > unlike page table lookups. Amusingly, just as for SOAR, modern efforts covering most of the same ground (JVM, .NET) are starting to show that you can do better in software and simple hardware than you can with complicated hardware. I'm not sure why that is still true in this day where processor architects have more transistors than they know what to do with... -- Andrew From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 11:12:05 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1879B16A420; Tue, 28 Mar 2006 11:12:05 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id AE55943D46; Tue, 28 Mar 2006 11:12:04 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id 02C711EC387; Tue, 28 Mar 2006 13:11:58 +0200 (CEST) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.4/8.13.4) with ESMTP id k2SBBuGa061880; Tue, 28 Mar 2006 13:11:58 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Andrew Reilly From: "Poul-Henning Kamp" In-Reply-To: Your message of "Tue, 28 Mar 2006 21:58:33 +1100." <20060328105833.GE87799@gurney.reilly.home> Date: Tue, 28 Mar 2006 13:11:56 +0200 Message-ID: <61879.1143544316@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 11:12:05 -0000 In message <20060328105833.GE87799@gurney.reilly.home>, Andrew Reilly writes: >On Tue, Mar 28, 2006 at 12:39:06PM +0200, Poul-Henning Kamp wrote: >> Nope, not even close. >> >> What Recursiv did was novel in that the objects lived in hardware >> and microcode, there were no way to overrun an array, because you >> simply couldn't index it outside it's bounds. Not even in the >> operating system. > >How is that different from all of the preceding segmented >machines like (to quote your .sig) the Z8001+MMU? On segmented architectures, all the way back, you put multiple objects in one segment. The traditional UNIX way had two or three segments, all of the code in one and all of the data (+/- the stack) in another. What Rekursiv did was to put each single variable or object in its own segment. int i; int j; would take two objects (or segments if you like). This is not merely segmentation, this is object oriented memory. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 12:40:50 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 79C4416A424 for ; Tue, 28 Mar 2006 12:40:50 +0000 (UTC) (envelope-from andrew@areilly.bpc-users.org) Received: from omta01sl.mx.bigpond.com (omta01sl.mx.bigpond.com [144.140.92.153]) by mx1.FreeBSD.org (Postfix) with ESMTP id 72C6143D6A for ; Tue, 28 Mar 2006 12:40:49 +0000 (GMT) (envelope-from andrew@areilly.bpc-users.org) Received: from areilly.bpc-users.org ([141.168.4.160]) by omta01sl.mx.bigpond.com with ESMTP id <20060328124047.LXJB25377.omta01sl.mx.bigpond.com@areilly.bpc-users.org> for ; Tue, 28 Mar 2006 12:40:47 +0000 Received: (qmail 90440 invoked by uid 501); 28 Mar 2006 12:42:00 -0000 Date: Tue, 28 Mar 2006 23:42:00 +1100 From: Andrew Reilly To: Poul-Henning Kamp Message-ID: <20060328124200.GB90303@gurney.reilly.home> References: <20060328105833.GE87799@gurney.reilly.home> <61879.1143544316@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <61879.1143544316@critter.freebsd.dk> User-Agent: Mutt/1.4.2.1i Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 12:40:50 -0000 On Tue, Mar 28, 2006 at 01:11:56PM +0200, Poul-Henning Kamp wrote: > The traditional UNIX way had two or three segments, all of the code > in one and all of the data (+/- the stack) in another. > > What Rekursiv did was to put each single variable or object > in its own segment. > > int i; > int j; > > would take two objects (or segments if you like). Aah. OK. Ick. I'm pretty sure that the segmentation practiced by AS/400 and Burroughs A-series is almost this flavour, though. Every allocated chunk of memory gets it's own segment. That probably means that individual machine words don't, but arrays and structs would, including stack frames (activation records) most of the time. > This is not merely segmentation, this is object oriented memory. I really think that this is something better handled in software, as is done by JVM and CLI, these days. Hardware can't strength-reduce the necessary descriptor loads and checks, whereas software can. Obviously that probably has it's own attractions to the paranoid/banking/military crowd, but not to the go fast at low cost and low power crowd, which I count myself among... Cheers, -- Andrew From owner-freebsd-arch@FreeBSD.ORG Tue Mar 28 12:52:39 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B8C7816A400; Tue, 28 Mar 2006 12:52:39 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from pasmtp.tele.dk (pasmtp.tele.dk [193.162.159.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5988A43D48; Tue, 28 Mar 2006 12:52:39 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by pasmtp.tele.dk (Postfix) with ESMTP id 5B5E11EC36B; Tue, 28 Mar 2006 14:52:35 +0200 (CEST) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.4/8.13.4) with ESMTP id k2SCqIwI062504; Tue, 28 Mar 2006 14:52:19 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Andrew Reilly From: "Poul-Henning Kamp" In-Reply-To: Your message of "Tue, 28 Mar 2006 23:42:00 +1100." <20060328124200.GB90303@gurney.reilly.home> Date: Tue, 28 Mar 2006 14:52:18 +0200 Message-ID: <62503.1143550338@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2006 12:52:39 -0000 In message <20060328124200.GB90303@gurney.reilly.home>, Andrew Reilly writes: >> This is not merely segmentation, this is object oriented memory. > >I really think that this is something better handled in >software, as is done by JVM and CLI, these days. Well, if hardware could do the "full job" in an efficient manner, that would be the ideal way to do it, because even software bugs in the operating system would not put the system at risk. Unfortunately hardware can't, so we're left with software trying to fend as best it can... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Wed Mar 29 01:56:32 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0EC5916A400 for ; Wed, 29 Mar 2006 01:56:32 +0000 (UTC) (envelope-from trhodes@FreeBSD.org) Received: from pittgoth.com (ns1.pittgoth.com [216.38.206.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id B31E543D45 for ; Wed, 29 Mar 2006 01:56:31 +0000 (GMT) (envelope-from trhodes@FreeBSD.org) Received: from localhost (net-ix.gw.ai.net [205.134.160.6] (may be forged)) (authenticated bits=0) by pittgoth.com (8.13.4/8.13.4) with ESMTP id k2T2rT8J082042 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 28 Mar 2006 21:53:30 -0500 (EST) (envelope-from trhodes@FreeBSD.org) Date: Tue, 28 Mar 2006 20:56:25 -0500 From: Tom Rhodes To: Rink Springer Message-Id: <20060328205625.658396e8.trhodes@FreeBSD.org> In-Reply-To: <20060328070332.GA53789@rink.nu> References: <20060328070332.GA53789@rink.nu> X-Mailer: Sylpheed version 1.0.5 (GTK+ 1.2.10; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: arch@FreeBSD.org Subject: Re: [RFC] Syslogd service patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2006 01:56:32 -0000 On Tue, 28 Mar 2006 09:03:32 +0200 Rink Springer wrote: > Hi everyone, > > I've devised a patch for syslogd, which allows you to specify a service > name when logging remotely. > > This is all embedded in the current syntax. For logging remote under a > different service, you can use @1.2.3.4:service [1]; for binding to a > different service, you can use 'syslogd -b 1.2.3.4:service'. No breakage > is introduced as the default service remains 'syslog'. > > The patch is at http://rink.nu/tmp/syslogd.diff, which I'd like to > commit. Does anyone have questions or comments? > > [1] OpenBSD also has this functionality diff -rubB /usr/src/usr.sbin/syslogd/syslog.conf.5 syslogd/syslog.conf.5 --- /usr/src/usr.sbin/syslogd/syslog.conf.5 Thu Aug 18 17:01:27 2005 +++ syslogd/syslog.conf.5 Sat Mar 25 17:40:56 2006 @@ -336,7 +336,9 @@ sign). Selected messages are forwarded to the .Xr syslogd 8 -program on the named host. +program on the named host. host:service may be Need a new line here, perhaps reword to avoid starting a sentence with lower cased words. +used to specify a host with a different service (default is +syslog). ^^^^^^^ Add a period. .It A comma separated list of users. Selected messages are written to those users diff -rubB /usr/src/usr.sbin/syslogd/syslogd.8 syslogd/syslogd.8 --- /usr/src/usr.sbin/syslogd/syslogd.8 Wed Apr 13 05:19:41 2005 +++ syslogd/syslogd.8 Sat Mar 25 17:37:28 2006 @@ -38,7 +38,7 @@ .Nm .Op Fl 46Acdknosuv .Op Fl a Ar allowed_peer -.Op Fl b Ar bind_address +.Op Fl b Ar bind_address Oo Ar :service Oc .Op Fl f Ar config_file .Op Fl l Oo Ar mode : Oc Ns Ar path .Op Fl m Ar mark_interval @@ -127,7 +127,8 @@ will be substituted by 128. .It Xo .Sm off -.Ar domainname Op : Ar service +.Ar domainname +.Op : Ar service .Sm on .Xc Accept datagrams where the reverse address lookup yields @@ -153,10 +154,11 @@ options are ignored if the .Fl s option is also specified. -.It Fl b Ar bind_address +.It Fl b Ar bind_address Op :service Specify one specific IP address or hostname to bind to. If a hostname is specified, the IPv4 or IPv6 address which corresponds to it is used. +An optional service can be supplied as well. .It Fl c Disable the compression of repeated instances of the same line into a single line of the form diff -rubB /usr/src/usr.sbin/syslogd/syslogd.c syslogd/syslogd.c --- /usr/src/usr.sbin/syslogd/syslogd.c Sun Jan 15 18:50:37 2006 +++ syslogd/syslogd.c Sat Mar 25 17:28:49 2006 @@ -1715,7 +1715,7 @@ struct addrinfo hints, *res; int error, i, pri, syncfile; const char *p, *q; - char *bp; + char *bp, *cp; char buf[MAXLINE], ebuf[100]; dprintf("cfline(\"%s\", f, \"%s\", \"%s\")\n", line, prog, host); @@ -1869,8 +1869,13 @@ switch (*p) { case '@': + cp = strchr (p, ':'); + if (cp != NULL) + *cp++ = 0; ?? *cp++ = 0? Perhaps I'm just lost. + (void)strlcpy(f->f_un.f_forw.f_hname, ++p, sizeof(f->f_un.f_forw.f_hname)); + Is this new line on purpose? memset(&hints, 0, sizeof(hints)); hints.ai_family = family; hints.ai_socktype = SOCK_DGRAM; @@ -1874,7 +1879,8 @@ memset(&hints, 0, sizeof(hints)); hints.ai_family = family; hints.ai_socktype = SOCK_DGRAM; - error = getaddrinfo(f->f_un.f_forw.f_hname, "syslog", &hints, + error = getaddrinfo(f->f_un.f_forw.f_hname, + (cp == NULL) ? "syslog" : cp, &hints, &res); if (error) { logerror(gai_strerror(error)); @@ -2530,12 +2536,29 @@ { struct addrinfo hints, *res, *r; int error, maxs, *s, *socks; + char* cp, *tmpname; + + if (bindhostname != NULL) { + tmpname = strdup (bindhostname); + if (tmpname == NULL) { + logerror(strerror(errno)); + errno = 0; + die(0); + } + + cp = strchr(tmpname, ':'); + if (cp != NULL) + *cp++ = 0; + } memset(&hints, 0, sizeof(hints)); hints.ai_flags = AI_PASSIVE; hints.ai_family = af; hints.ai_socktype = SOCK_DGRAM; - error = getaddrinfo(bindhostname, "syslog", &hints, &res); + error = getaddrinfo(tmpname, (cp == NULL) ? "syslog" : cp, + &hints, &res); + if (tmpname != NULL) + free (tmpname); if (error) { logerror(gai_strerror(error)); errno = 0; -- Tom Rhodes From owner-freebsd-arch@FreeBSD.ORG Wed Mar 29 08:48:36 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC24C16A420 for ; Wed, 29 Mar 2006 08:48:36 +0000 (UTC) (envelope-from gni@gecko.de) Received: from kirk.baltic.net (kirk.baltic.net [193.189.247.10]) by mx1.FreeBSD.org (Postfix) with SMTP id 8CBFB43D46 for ; Wed, 29 Mar 2006 08:48:34 +0000 (GMT) (envelope-from gni@gecko.de) Received: (qmail 28899 invoked from network); 29 Mar 2006 10:16:16 -0000 Received: from waldorf.gecko.de (HELO asterix.int.gecko.de) (193.189.247.200) by kirk.baltic.net with SMTP; 29 Mar 2006 10:16:16 -0000 Received: from kermit.int.gecko.de [192.168.120.252] by asterix.int.gecko.de; Wed, 29 Mar 2006 10:48:45 +0200 Received: from lorien.int.gecko.de (lorien [192.168.120.159]) by kermit.int.gecko.de (8.12.10+Sun/8.12.10) with ESMTP id k2T8mHI1000016; Wed, 29 Mar 2006 10:48:17 +0200 (CEST) Received: from lorien.int.gecko.de (localhost [127.0.0.1]) by lorien.int.gecko.de (8.12.9/8.12.9) with ESMTP id k2T8mqBu065724; Wed, 29 Mar 2006 10:48:52 +0200 (MEST) (envelope-from munk@lorien.int.gecko.de) Received: (from munk@localhost) by lorien.int.gecko.de (8.12.9/8.12.9/Submit) id k2T8mkIK065723; Wed, 29 Mar 2006 10:48:46 +0200 (MEST) Date: Wed, 29 Mar 2006 10:48:46 +0200 From: Gunther Nikl To: Peter Jeremy Message-ID: <20060329084846.GA65703@lorien.int.gecko.de> References: <44247DF1.8000002@FreeBSD.org> <20060326110929.V35431@fledge.watson.org> <4426D7A0.4040007@FreeBSD.org> <200603271110.02917.jhb@freebsd.org> <44281421.3060401@FreeBSD.org> <20060327185017.GF7001@funkthat.com> <352D1430-CC72-47EE-9E50-B1C4404ACA95@FreeBSD.org> <20060328091153.GC961@turion.vk2pj.dyndns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060328091153.GC961@turion.vk2pj.dyndns.org> User-Agent: Mutt/1.4.2.1i Cc: Jason Evans , freebsd-arch@freebsd.org Subject: Re: Proposed addition of malloc_size_np() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2006 08:48:36 -0000 On Tue, Mar 28, 2006 at 08:11:53PM +1100, Peter Jeremy wrote: > On Mon, 2006-Mar-27 12:52:17 -0800, Jason Evans wrote: > >void * > >malloc(size_t size); > > > >void * > >calloc(size_t size); > > > >void * > >realloc(void *ptr, size_t size, size_t old_size); > > > >void > >free(void *ptr, size_t size); > > ISTR AmigaDOS did something like this (my autodocs are packed away > somewhere so I can't quickly check). FYI, its AllocMem (which also has a flags argument) and FreeMem. Later AllocVec (caching the size) and FreeVec were added to the system. Since using these system functins will lead to fragmented memory new functions were added using pools. An anpplication allocated a pool and then allocated from that pool. Then it could free individual allocations or deallocate the pool which would release all memory from the pool at once. Gunther From owner-freebsd-arch@FreeBSD.ORG Wed Mar 29 10:07:46 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 76C4416A424 for ; Wed, 29 Mar 2006 10:07:46 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 08EAC43D4C for ; Wed, 29 Mar 2006 10:07:46 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id B07AE46C2D for ; Wed, 29 Mar 2006 05:07:44 -0500 (EST) Date: Wed, 29 Mar 2006 10:07:44 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: arch@FreeBSD.org In-Reply-To: <20060315004530.B5861@fledge.watson.org> Message-ID: <20060329100513.D19236@fledge.watson.org> References: <20060315004530.B5861@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Re: netatm: plan for removal unless an active maintainer is found X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2006 10:07:46 -0000 On Wed, 15 Mar 2006, Robert Watson wrote: > In order to begin to merge revised socket/pcb code, required to fix a number > of current races manifesting in the TCP code under load, and required for > breaking out the tcbinfo lock which is a significant bottleneck in high > performance TCP and multi-processor TCP scalability, I will disconnect > netatm and dependent components from the build on April 1, 2006. At that > point, I will merge updated socket and pcb reference counting. Reminder: April 1 approaches. I've merged changes to many non-netinet protocols in support of the approaching socket/pcb reference model changes, but have the netinet changes depend on completing socket layer changes that are believed not to work with netatm as they stand. I'll be posting the socket and netinet changes to arch@ today; I've posted them previously to other lists, such as current@. Robert N M Watson From owner-freebsd-arch@FreeBSD.ORG Wed Mar 29 10:38:34 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0150816A401; Wed, 29 Mar 2006 10:38:34 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from smtp-1.dlr.de (smtp-1.dlr.de [195.37.61.185]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6969B43D48; Wed, 29 Mar 2006 10:38:32 +0000 (GMT) (envelope-from Hartmut.Brandt@dlr.de) Received: from beagle.kn.op.dlr.de ([129.247.173.6]) by smtp-1.dlr.de over TLS secured channel with Microsoft SMTPSVC(6.0.3790.1830); Wed, 29 Mar 2006 12:38:30 +0200 Date: Wed, 29 Mar 2006 12:38:28 +0200 (CEST) From: Harti Brandt X-X-Sender: brandt_h@beagle.kn.op.dlr.de To: Robert Watson In-Reply-To: <20060329100513.D19236@fledge.watson.org> Message-ID: <20060329123238.B87509@beagle.kn.op.dlr.de> References: <20060315004530.B5861@fledge.watson.org> <20060329100513.D19236@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-OriginalArrivalTime: 29 Mar 2006 10:38:30.0960 (UTC) FILETIME=[EBF61300:01C6531C] Cc: arch@FreeBSD.org Subject: Re: netatm: plan for removal unless an active maintainer is found X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Harti Brandt List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2006 10:38:34 -0000 On Wed, 29 Mar 2006, Robert Watson wrote: RW>On Wed, 15 Mar 2006, Robert Watson wrote: RW> RW>> In order to begin to merge revised socket/pcb code, required to fix a RW>> number of current races manifesting in the TCP code under load, and RW>> required for breaking out the tcbinfo lock which is a significant RW>> bottleneck in high performance TCP and multi-processor TCP scalability, I RW>> will disconnect netatm and dependent components from the build on April 1, RW>> 2006. At that point, I will merge updated socket and pcb reference RW>> counting. RW> RW>Reminder: April 1 approaches. RW> RW>I've merged changes to many non-netinet protocols in support of the RW>approaching socket/pcb reference model changes, but have the netinet changes RW>depend on completing socket layer changes that are believed not to work with RW>netatm as they stand. I'll be posting the socket and netinet changes to RW>arch@ today; I've posted them previously to other lists, such as current@. Skip Ford expressed interest in netatm, but he said also that he would continue to work on HARP even when it is removed. So I guess it could be revived in the future (just in the case). I've also sent him my half -IDT driver and he said he will first work on this. When this is ready we have all the hardware supported in ngATM which HARP also does. harti From owner-freebsd-arch@FreeBSD.ORG Wed Mar 29 11:13:35 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9CAF716A400; Wed, 29 Mar 2006 11:13:35 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 470AE43D45; Wed, 29 Mar 2006 11:13:35 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 622A246BDB; Wed, 29 Mar 2006 06:13:34 -0500 (EST) Date: Wed, 29 Mar 2006 11:13:34 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Harti Brandt In-Reply-To: <20060329123238.B87509@beagle.kn.op.dlr.de> Message-ID: <20060329110421.M19236@fledge.watson.org> References: <20060315004530.B5861@fledge.watson.org> <20060329100513.D19236@fledge.watson.org> <20060329123238.B87509@beagle.kn.op.dlr.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@FreeBSD.org Subject: Re: netatm: plan for removal unless an active maintainer is found X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2006 11:13:35 -0000 On Wed, 29 Mar 2006, Harti Brandt wrote: > On Wed, 29 Mar 2006, Robert Watson wrote: > > RW>On Wed, 15 Mar 2006, Robert Watson wrote: > RW> > RW>> In order to begin to merge revised socket/pcb code, required to fix a > RW>> number of current races manifesting in the TCP code under load, and > RW>> required for breaking out the tcbinfo lock which is a significant > RW>> bottleneck in high performance TCP and multi-processor TCP scalability, I > RW>> will disconnect netatm and dependent components from the build on April 1, > RW>> 2006. At that point, I will merge updated socket and pcb reference > RW>> counting. > RW> > RW>Reminder: April 1 approaches. > RW> > RW>I've merged changes to many non-netinet protocols in support of the > RW>approaching socket/pcb reference model changes, but have the netinet changes > RW>depend on completing socket layer changes that are believed not to work with > RW>netatm as they stand. I'll be posting the socket and netinet changes to > RW>arch@ today; I've posted them previously to other lists, such as current@. > > Skip Ford expressed interest in netatm, but he said also that he would > continue to work on HARP even when it is removed. So I guess it could be > revived in the future (just in the case). I've also sent him my half -IDT > driver and he said he will first work on this. When this is ready we have > all the hardware supported in ngATM which HARP also does. I have patches, and plan to commit them, that keep netatm compilable. The problem is that I am unable to test netatm, and have limited time to try to figure it out (and it's significant enough that it requires figuring out). There are really two sticking points with it remaining in the tree right now: (1) It's not MPSAFE, and in absence of a maintainer is unlikely to become so. This means it is a direct obstacle to removing the non-MPSAFEty crutches in the network stack, which we're otherwise increasingly done with. The other components with similar problems will be facing similar eviction notices if they don't have maintainers. (2) Continuing work to improve SMP performance requires significant changes to the socket code (and other parts of the stack). Each change in itself is relatively minor, but requires non-trivial protocol adaptation and testing. In absense of a maintainer for netatm, this work will either break netatm, or be unable to proceed. I can keep hacking on netatm to keep it compiling, but I can't promise the result will work, and given the complexity of the socket and protocol code, it's highly likely that it won't work. Rather than break it further and further, I'd rather either find a maintainer, or mark it as deprecated and on the removal path. The April 1 date is simply the date where the fact that I cannot test netatm no longer becomes a blocking factor for my committing changes that may break it, since otherwise these changes, and the changes dependent on them, won't have time to settle out before we reach 7.0, effectively stalling all further work along these lines, which is undesirable :-). The April 1 date is also only for disabling the compile and marking it as being on the removal path. The plan is not to remove it until late in the summer, and only then if it hasn't been fixed to work in the new world order or and shows no signs of moving in that direction. If there is significant progress, plans are easy to change. So all that aside, I continue to plan to commit my changes on April 1, but welcome any work to keep netatm working. It sounds like bluetooth may also briefly break; I've exchanged e-mail with emax and he has plans to fix it shortly thereafter. Robert N M Watson From owner-freebsd-arch@FreeBSD.ORG Wed Mar 29 11:37:10 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3DECC16A400; Wed, 29 Mar 2006 11:37:10 +0000 (UTC) (envelope-from bms@spc.org) Received: from mindfull.spc.org (mindfull.spc.org [83.167.185.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4852043D55; Wed, 29 Mar 2006 11:37:08 +0000 (GMT) (envelope-from bms@spc.org) Received: from arginine.spc.org ([83.167.185.2]) by mindfull.spc.org with esmtps (TLSv1:AES256-SHA:256) (Exim 4.52) id 1FOYz2-0000ZI-Co; Wed, 29 Mar 2006 12:37:04 +0100 Received: from localhost (localhost [127.0.0.1]) by arginine.spc.org (Postfix) with ESMTP id 96A306564F; Wed, 29 Mar 2006 12:37:06 +0100 (BST) Received: from arginine.spc.org ([127.0.0.1]) by localhost (arginine.spc.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 62326-02; Wed, 29 Mar 2006 12:37:05 +0100 (BST) Received: by arginine.spc.org (Postfix, from userid 1078) id 24C8865499; Wed, 29 Mar 2006 12:37:05 +0100 (BST) Date: Wed, 29 Mar 2006 12:37:05 +0100 From: Bruce M Simpson To: Harti Brandt Message-ID: <20060329113705.GV80492@spc.org> References: <20060315004530.B5861@fledge.watson.org> <20060329100513.D19236@fledge.watson.org> <20060329123238.B87509@beagle.kn.op.dlr.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060329123238.B87509@beagle.kn.op.dlr.de> User-Agent: Mutt/1.4.1i Organization: Incunabulum X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - mindfull.spc.org X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - spc.org X-Source: X-Source-Args: X-Source-Dir: Cc: arch@FreeBSD.org, Robert Watson Subject: Re: netatm: plan for removal unless an active maintainer is found X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2006 11:37:10 -0000 On Wed, Mar 29, 2006 at 12:38:28PM +0200, Harti Brandt wrote: > Skip Ford expressed interest in netatm, but he said also that he would > continue to work on HARP even when it is removed. So I guess it could be > revived in the future (just in the case). I've also sent him my half -IDT > driver and he said he will first work on this. When this is ready we have > all the hardware supported in ngATM which HARP also does. I have shipped IDT and ENI hardware to Skip which arrived as of last week. BMS From owner-freebsd-arch@FreeBSD.ORG Wed Mar 29 18:07:28 2006 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D922716A422; Wed, 29 Mar 2006 18:07:28 +0000 (UTC) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 791A843D49; Wed, 29 Mar 2006 18:07:28 +0000 (GMT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.13.0/8.13.0) with ESMTP id k2TI7RUM014469; Wed, 29 Mar 2006 10:07:27 -0800 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.13.0/8.13.0/Submit) id k2TI7RlP014468; Wed, 29 Mar 2006 10:07:27 -0800 Date: Wed, 29 Mar 2006 10:07:27 -0800 From: Brooks Davis To: Robert Watson Message-ID: <20060329180727.GA10956@odin.ac.hmc.edu> References: <20060315004530.B5861@fledge.watson.org> <20060329100513.D19236@fledge.watson.org> <20060329123238.B87509@beagle.kn.op.dlr.de> <20060329110421.M19236@fledge.watson.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/9DWx/yDrRhgMJTb" Content-Disposition: inline In-Reply-To: <20060329110421.M19236@fledge.watson.org> User-Agent: Mutt/1.4.1i X-Virus-Scanned: by amavisd-new X-Spam-Status: No, hits=0.0 required=8.0 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on odin.ac.hmc.edu Cc: arch@freebsd.org, Harti Brandt Subject: Re: netatm: plan for removal unless an active maintainer is found X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Mar 2006 18:07:29 -0000 --/9DWx/yDrRhgMJTb Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 29, 2006 at 11:13:34AM +0000, Robert Watson wrote: >=20 > On Wed, 29 Mar 2006, Harti Brandt wrote: >=20 > >On Wed, 29 Mar 2006, Robert Watson wrote: > > > >RW>On Wed, 15 Mar 2006, Robert Watson wrote: > >RW> > >RW>> In order to begin to merge revised socket/pcb code, required to fix= a > >RW>> number of current races manifesting in the TCP code under load, and > >RW>> required for breaking out the tcbinfo lock which is a significant > >RW>> bottleneck in high performance TCP and multi-processor TCP=20 > >scalability, I > >RW>> will disconnect netatm and dependent components from the build on= =20 > >April 1, > >RW>> 2006. At that point, I will merge updated socket and pcb reference > >RW>> counting. > >RW> > >RW>Reminder: April 1 approaches. > >RW> > >RW>I've merged changes to many non-netinet protocols in support of the > >RW>approaching socket/pcb reference model changes, but have the netinet= =20 > >changes > >RW>depend on completing socket layer changes that are believed not to wo= rk=20 > >with > >RW>netatm as they stand. I'll be posting the socket and netinet changes= to > >RW>arch@ today; I've posted them previously to other lists, such as=20 > >current@. > > > >Skip Ford expressed interest in netatm, but he said also that he would= =20 > >continue to work on HARP even when it is removed. So I guess it could be= =20 > >revived in the future (just in the case). I've also sent him my half -ID= T=20 > >driver and he said he will first work on this. When this is ready we hav= e=20 > >all the hardware supported in ngATM which HARP also does. >=20 > I have patches, and plan to commit them, that keep netatm compilable. Th= e=20 > problem is that I am unable to test netatm, and have limited time to try = to=20 > figure it out (and it's significant enough that it requires figuring out)= .=20 I'd be moderatly suprised if it worked at all. None of the ATM code was fun to deal with when I move struct ifnet out of the softc, but IIRC netatm way by far the most confusing. -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --/9DWx/yDrRhgMJTb Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFEKszeXY6L6fI4GtQRAgdLAKCJmAFAFiVpdudClqVgoY9TjrnBgwCgtp6m 9kpy1GutM8ssq1XqfgK3qI0= =APPd -----END PGP SIGNATURE----- --/9DWx/yDrRhgMJTb-- From owner-freebsd-arch@FreeBSD.ORG Sat Apr 1 17:33:26 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0D13A16A422 for ; Sat, 1 Apr 2006 17:33:26 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id A191443D46 for ; Sat, 1 Apr 2006 17:33:25 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id D7E8746C37 for ; Sat, 1 Apr 2006 12:33:24 -0500 (EST) Date: Sat, 1 Apr 2006 17:33:24 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: arch@FreeBSD.org In-Reply-To: <20060315004530.B5861@fledge.watson.org> Message-ID: <20060401173217.N82503@fledge.watson.org> References: <20060315004530.B5861@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Re: netatm: plan for removal unless an active maintainer is found X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Apr 2006 17:33:26 -0000 On Wed, 15 Mar 2006, Robert Watson wrote: > In order to begin to merge revised socket/pcb code, required to fix a number > of current races manifesting in the TCP code under load, and required for > breaking out the tcbinfo lock which is a significant bottleneck in high > performance TCP and multi-processor TCP scalability, I will disconnect > netatm and dependent components from the build on April 1, 2006. At that > point, I will merge updated socket and pcb reference counting. Per this e-mail (and the associated thread), socket/pcb reference counting changes have been committed that likely render netatm further disfunctional. Robert N M Watson From owner-freebsd-arch@FreeBSD.ORG Sat Apr 1 20:48:38 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE98B16A41F for ; Sat, 1 Apr 2006 20:48:38 +0000 (UTC) (envelope-from artwork@cartoon-factory.com) Received: from ceres.aros.net (ceres.aros.net [66.219.192.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7BFC443D58 for ; Sat, 1 Apr 2006 20:48:38 +0000 (GMT) (envelope-from artwork@cartoon-factory.com) Received: from cartoon (42dbd96d.dsl.aros.net [66.219.217.109]) by ceres.aros.net (8.13.3/8.13.3) with ESMTP id k31Kmb8k097421 for ; Sat, 1 Apr 2006 13:48:37 -0700 (MST) (envelope-from artwork@cartoon-factory.com) Message-Id: <200604012048.k31Kmb8k097421@ceres.aros.net> From: "Cartoon Factory" To: Date: Sat, 1 Apr 2006 13:48:48 -0700 Organization: The Cartoon Factory Animation Art Gallery MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.6353 Thread-Index: AcZVzD3Y2L5xAebxRmOnbvPFfDx4DA== X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 X-Virus-Scanned: ClamAV version 0.88, clamav-milter version 0.87 on ceres.aros.net X-Virus-Status: Clean Cc: Subject: Stupid Question X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: artwork@cartoon-factory.com List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Apr 2006 20:48:38 -0000 Hi: I have a stupid question. I am looking at the processors supported by FreeBSD, and I see AMD64 (which I know is AMD) and ia64 (which I have no clue what it is). I am looking at getting a new server, with a XEON processor. My confusion is two-fold. One, they say the processor is 32/64. Can FreeBSD use the 64 aspect of this? And if so, is it ia64, or under i386? Thank You! ------------------- Dave Koch Gallery Director www.toon.com artwork@cartoon-factory.com 801.255.5771 fax 801.255-5772 ____________________________________________ From owner-freebsd-arch@FreeBSD.ORG Sat Apr 1 20:51:23 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E978716A41F for ; Sat, 1 Apr 2006 20:51:23 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.208.78.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id 66EBD43D6A for ; Sat, 1 Apr 2006 20:51:23 +0000 (GMT) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.13.4/8.13.4) with ESMTP id k31KpMLq081697; Sat, 1 Apr 2006 12:51:22 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.13.4/8.13.1/Submit) id k31KpM6h081696; Sat, 1 Apr 2006 12:51:22 -0800 (PST) (envelope-from sgk) Date: Sat, 1 Apr 2006 12:51:22 -0800 From: Steve Kargl To: Cartoon Factory Message-ID: <20060401205122.GA81680@troutmask.apl.washington.edu> References: <200604012048.k31Kmb8k097421@ceres.aros.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200604012048.k31Kmb8k097421@ceres.aros.net> User-Agent: Mutt/1.4.2.1i Cc: freebsd-arch@freebsd.org Subject: Re: Stupid Question X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Apr 2006 20:51:24 -0000 On Sat, Apr 01, 2006 at 01:48:48PM -0700, Cartoon Factory wrote: > > I have a stupid question. I am looking at the processors supported by > FreeBSD, and I see AMD64 (which I know is AMD) and ia64 (which I have no > clue what it is). ia64 == Intel Itanium > I am looking at getting a new server, with a XEON processor. > > My confusion is two-fold. One, they say the processor is 32/64. Can FreeBSD > use the 64 aspect of this? And if so, is it ia64, or under i386? > If the XEON is an EM64T processor, then you want to use FreeBSD-amd64 to harness the 64-bit nature of the processor. -- Steve From owner-freebsd-arch@FreeBSD.ORG Sat Apr 1 21:12:43 2006 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A8A8C16A41F for ; Sat, 1 Apr 2006 21:12:43 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail18.syd.optusnet.com.au (mail18.syd.optusnet.com.au [211.29.132.199]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B01743D49 for ; Sat, 1 Apr 2006 21:12:42 +0000 (GMT) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-19-236.belrs4.nsw.optusnet.com.au [220.239.19.236]) by mail18.syd.optusnet.com.au (8.12.11/8.12.11) with ESMTP id k31LCbpD010588 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Sun, 2 Apr 2006 07:12:38 +1000 Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.13.4/8.13.4) with ESMTP id k31LCbnp003124; Sun, 2 Apr 2006 07:12:37 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.13.4/8.13.4/Submit) id k31LCbuK003123; Sun, 2 Apr 2006 07:12:37 +1000 (EST) (envelope-from peter) Date: Sun, 2 Apr 2006 07:12:37 +1000 From: Peter Jeremy To: Cartoon Factory Message-ID: <20060401211237.GD684@turion.vk2pj.dyndns.org> References: <200604012048.k31Kmb8k097421@ceres.aros.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200604012048.k31Kmb8k097421@ceres.aros.net> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.11 Cc: freebsd-arch@freebsd.org Subject: Re: Stupid Question X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Apr 2006 21:12:43 -0000 On Sat, 2006-Apr-01 13:48:48 -0700, Cartoon Factory wrote: >I have a stupid question. No such thing as stupid questions but this does belong on -questions rather then -arch. > I am looking at the processors supported by >FreeBSD, and I see AMD64 (which I know is AMD) and ia64 (which I have no >clue what it is). iA64 was Intel's replacement for the iA32 processor family. It sports an impressive collection of buzzwords that were state-of-the-art when it was first announced. It spent so long as vapourware and has turned out to be such a flop that Intel wound up licensing the AMD64 (which it re-named EM64T). >My confusion is two-fold. One, they say the processor is 32/64. Can FreeBSD >use the 64 aspect of this? And if so, is it ia64, or under i386? In the "CPU" section of your dmesg, you should have a line "AMD Features" with the "LM" flag set - this means you have an amd64/em64t. To run in 64-bit mode, you need FreeBSD/amd64. -- Peter Jeremy