From owner-freebsd-arch@FreeBSD.ORG Sun Jun 8 12:02:25 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 16C5937B401; Sun, 8 Jun 2003 12:02:25 -0700 (PDT) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3D99143FBD; Sun, 8 Jun 2003 12:02:24 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.12.9/8.12.6) with ESMTP id h58J2OVI040940; Sun, 8 Jun 2003 12:02:24 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.9/8.12.6/Submit) id h58J2OGW040939; Sun, 8 Jun 2003 12:02:24 -0700 (PDT) Date: Sun, 8 Jun 2003 12:02:24 -0700 (PDT) From: Matthew Dillon Message-Id: <200306081902.h58J2OGW040939@apollo.backplane.com> To: Doug Barton References: <20030605235254.W5414@znfgre.qbhto.arg> <20030606024813.Y5414@znfgre.qbhto.arg> <20030606233358.Y15459@znfgre.qbhto.arg> <20030607150857.S81111@znfgre.qbhto.arg> cc: freebsd-arch@freebsd.org Subject: Re: Way forward with BIND 8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jun 2003 19:02:25 -0000 :Well, rndc can be configured for remote control too. Since by default it's :configured locally though, I decided that the easiest way to deal with it :would just be to copy the sample file. However, based on your feedback :here, I just added a pkg-message that gives some information about this :topic. : :> Additionally, the rndc-confgen program does not even appear to work, :> at least not on my system. If I run 'rndc-confgen -a' it just stays :> stuck in a select() somewhere and does nothing. : :http://people.freebsd.org/~dougb/randomness.html :) : :Thanks for the feedback, : :Doug Oh great, that makes /dev/random rather un-useful if manual intervention is required to get it going on 4.x. Another thing to add to the list, I guess. -Matt Matthew Dillon From owner-freebsd-arch@FreeBSD.ORG Sun Jun 8 16:38:43 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 92D6437B401 for ; Sun, 8 Jun 2003 16:38:43 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.attbi.com [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 15D0D43FE5 for ; Sun, 8 Jun 2003 16:38:43 -0700 (PDT) (envelope-from DougB@freebsd.org) Received: from master.dougb.net (12-234-22-23.client.attbi.com[12.234.22.23](untrusted sender)) by attbi.com (rwcrmhc11) with SMTP id <2003060823383701300alufqe>; Sun, 8 Jun 2003 23:38:37 +0000 Date: Sun, 8 Jun 2003 16:38:36 -0700 (PDT) From: Doug Barton To: Terry Lambert In-Reply-To: <3EDD7CFA.4795FB99@mindspring.com> Message-ID: <20030608160735.I5603@znfgre.qbhto.arg> References: <20030602171942.GA87863@roark.gnf.org> <3EDD7CFA.4795FB99@mindspring.com> Organization: http://www.FreeBSD.org/ X-message-flag: Outlook -- Not just for spreading viruses anymore! MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: Matthew Dillon Subject: Parallelizing rc (Was: Re: Making a dynamically-linked root) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jun 2003 23:38:43 -0000 On Tue, 3 Jun 2003, Terry Lambert wrote: > People try to pretend that the dependencies that exist are > between programs, but they're actually between service > providers and service consumers, and largely independent of > the programs providing the services. On top of that, the > dependencies tend to be both hard and soft, e.g. it's possible > to continue to offer a degraded service, rather than failing > outright, if some dependent services aren't there (e.g. you > can log by IP address if DNS isn't up to provide reverse > name mappings to look pretty in your logs, etc.). A lot of people have suggested that parallelizing rc ought to be an easy thing to do, especially with the new rc system in place. However, we've actually looked at doing that, and in practice it turns out to be much more difficult than it sounds. Terry's point here is well taken, and shouldn't be trivialized. The rc team has regular conversations about improving the dependency ordering, and they generally go 'round in circles and land in the same location, since what we have now works, and historically changes have led to the dreaded "unpredictable results" for edge cases. All that said however, if someone really wants to take up this banner, please feel free. I'd appreciate it if you'd coordinate with the folks at freebsd-rc@yahoogroups.com, but it's not a requirement. Before committing to the tree however, please post your work for review, and make sure it gets lots of testing with lots of different situations. Doug -- This .signature sanitized for your protection From owner-freebsd-arch@FreeBSD.ORG Sun Jun 8 16:40:48 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6014237B401 for ; Sun, 8 Jun 2003 16:40:48 -0700 (PDT) Received: from sccrmhc11.attbi.com (sccrmhc11.attbi.com [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id AFA7643FA3 for ; Sun, 8 Jun 2003 16:40:47 -0700 (PDT) (envelope-from DougB@freebsd.org) Received: from master.dougb.net (12-234-22-23.client.attbi.com[12.234.22.23](untrusted sender)) by attbi.com (sccrmhc11) with SMTP id <2003060823404601100bkrfte>; Sun, 8 Jun 2003 23:40:46 +0000 Date: Sun, 8 Jun 2003 16:40:45 -0700 (PDT) From: Doug Barton To: Marcel Moolenaar In-Reply-To: <20030602224734.GC1345@dhcp01.pn.xcllnt.net> Message-ID: <20030608164005.X5603@znfgre.qbhto.arg> References: <20030602171942.GA87863@roark.gnf.org> <20030602214956.GG87863@roark.gnf.org> <20030602224734.GC1345@dhcp01.pn.xcllnt.net> Organization: http://www.FreeBSD.org/ X-message-flag: Outlook -- Not just for spreading viruses anymore! MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org Subject: Re: Making a dynamically-linked root X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jun 2003 23:40:48 -0000 On Mon, 2 Jun 2003, Marcel Moolenaar wrote: > I suggest we get the functionality in without actually changing the > default. We can change the default anytime after that when we are > confident that we covered everything and have understanding of the > overall impact of switching... I strongly support this position, for whatever that's worth. Doug -- This .signature sanitized for your protection From owner-freebsd-arch@FreeBSD.ORG Mon Jun 9 03:57:41 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4830337B401 for ; Mon, 9 Jun 2003 03:57:41 -0700 (PDT) Received: from mta02ps.bigpond.com (mta02ps.bigpond.com [144.135.25.134]) by mx1.FreeBSD.org (Postfix) with ESMTP id CCAF143FAF for ; Mon, 9 Jun 2003 03:57:39 -0700 (PDT) (envelope-from areilly@bigpond.net.au) Received: from areilly.bpc-users.org ([144.135.25.69]) by mta02ps.bigpond.com (Netscape Messaging Server 4.15 mta02ps Jul 16 2002 22:47:55) with SMTP id HG7NRX00.BNJ for ; Mon, 9 Jun 2003 20:57:33 +1000 Received: from cpe-144-132-191-61.nsw.bigpond.net.au ([144.132.191.61]) by psmam01bpa.bigpond.com(MAM $Name: REL_3_3_2b $ 71/6787786); 09 Jun 2003 20:57:33 Received: (qmail 8073 invoked from network); 9 Jun 2003 10:57:34 -0000 Received: from localhost (HELO ?127.0.0.1?) (andrew@127.0.0.1) by localhost with SMTP; 9 Jun 2003 10:57:34 -0000 From: Andrew Reilly To: Terry Lambert In-Reply-To: <3EDD7CFA.4795FB99@mindspring.com> References: <20030602171942.GA87863@roark.gnf.org> <20030603080456.GA57773@cirb503493.alcatel.com.au> <3EDD7CFA.4795FB99@mindspring.com> Content-Type: text/plain Organization: Message-Id: <1055156254.1799.10.camel@gurney.reilly.home> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.2.4 Date: 09 Jun 2003 20:57:34 +1000 Content-Transfer-Encoding: 7bit cc: Matthew Dillon cc: arch@freebsd.org Subject: Re: Making a dynamically-linked root X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 10:57:41 -0000 On Wed, 2003-06-04 at 15:00, Terry Lambert wrote: > The main problem we ran into with doing this on the InterJet > was thatsome services started later would finish starting > before earlier services on which they were dependent. > [examples] > > People try to pretend that the dependencies that exist are > between programs, but they're actually between service > providers and service consumers, and largely independent of > the programs providing the services. On top of that, the > dependencies tend to be both hard and soft, e.g. it's possible > to continue to offer a degraded service, rather than failing > outright, if some dependent services aren't there (e.g. you > can log by IP address if DNS isn't up to provide reverse > name mappings to look pretty in your logs, etc.). That there are dependencies between services is unarguable, but I wonder how much of that really needs to be codified in startup orders, and how much can be sorted out dynamically, by making the implementations of these services robust in the absence of their dependencies. Maybe that path is more trouble than just codifying a dependency graph. Which was what was so good about the traditional, linear, rc script, of course. I haven't analysed the startup in any great detail myself, but as far as I can tell, my "djb land" does indeed all start up in parallel, and the various pieces just cope until the others are up. That's qmail, djbdns and tinydns. There's certainly no dependency or causality encoded explicitly in the /service directory. I've thought about moving other services that currently run from /usr/local/etc/rc.d (i.e., things that the system doesn't care about) to run from /service in the same way, but lack of circular tuits has prevented me, thus far. Cheers, -- Andrew Reilly From owner-freebsd-arch@FreeBSD.ORG Mon Jun 9 04:12:48 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7862A37B401 for ; Mon, 9 Jun 2003 04:12:48 -0700 (PDT) Received: from mta07bw.bigpond.com (mta07bw.bigpond.com [144.135.24.134]) by mx1.FreeBSD.org (Postfix) with ESMTP id AF62743F75 for ; Mon, 9 Jun 2003 04:12:47 -0700 (PDT) (envelope-from areilly@bigpond.net.au) Received: from areilly.bpc-users.org ([144.135.24.84]) by mta07bw.email.bigpond.com (iPlanet Messaging Server 5.2 HotFix 1.14 (built Mar 18 2003)) with SMTP id <0HG7005GNOGF9D@mta07bw.email.bigpond.com> for arch@freebsd.org; Mon, 09 Jun 2003 21:12:15 +1000 (EST) Received: from cpe-144-132-191-61.nsw.bigpond.net.au ([144.132.191.61]) by bwmam06bpa.bigpond.com(MailRouter V3.2g 53/12102630); Mon, 09 Jun 2003 21:12:15 +0000 Received: (qmail 8310 invoked from network); Mon, 09 Jun 2003 11:12:16 +0000 Received: from localhost (HELO ?127.0.0.1?) (andrew@127.0.0.1) by localhost with SMTP; Mon, 09 Jun 2003 11:12:16 +0000 Date: Mon, 09 Jun 2003 21:12:16 +1000 From: Andrew Reilly In-reply-to: <20030606072909.GA26354@over-yonder.net> To: "Matthew D. Fuller" Message-id: <1055157135.1799.19.camel@gurney.reilly.home> Organization: MIME-version: 1.0 X-Mailer: Ximian Evolution 1.2.4 Content-type: text/plain Content-transfer-encoding: 7BIT References: <20030605221114.GB51432@over-yonder.net> <20030606063105.D3B442A8C1@canning.wemm.org> <20030606072909.GA26354@over-yonder.net> cc: arch@freebsd.org Subject: Re: Making a dynamically-linked root X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 11:12:48 -0000 On Fri, 2003-06-06 at 17:29, Matthew D. Fuller wrote: > And all else being > equal, I'm fully of the belief that the increase in potential minor > calamities (which some manual /rescue/* intervention can recover) is a > small price to pay for some of the gains that a dynamic / gives. Is static/dynamic necessarily a dichotomy? As an example of an intermediate state, couldn't a nominally dynamic /bin be built, but with libc partially linked-in, in each case? How much of /bin would then have no further shared dependencies? -- Andrew Reilly From owner-freebsd-arch@FreeBSD.ORG Tue Jun 10 02:40:24 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A269537B401; Tue, 10 Jun 2003 02:40:24 -0700 (PDT) Received: from relay1.cris.net (relay1.cris.net [212.110.128.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6E80E43F3F; Tue, 10 Jun 2003 02:40:18 -0700 (PDT) (envelope-from phantom@phantom.cris.net) Received: from phantom.cris.net (root@phantom.cris.net [212.110.130.74]) by relay1.cris.net (8.12.6/8.12.6) with ESMTP id h5ACeNnp078799; Tue, 10 Jun 2003 12:40:24 GMT Received: (from phantom@localhost) by phantom.cris.net (8.12.6/8.12.2) id h5A9llAW007649; Tue, 10 Jun 2003 12:47:47 +0300 (EEST) (envelope-from phantom) Date: Tue, 10 Jun 2003 12:47:47 +0300 From: Alexey Zelkin To: Doug Barton Message-ID: <20030610124747.A7560@phantom.cris.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i X-Operating-System: FreeBSD 4.7-STABLE i386 cc: =?koi8-r?B?4c7E0sXKIP7F0s7P1w==?= cc: arch@freebsd.org Subject: removing stale files (was: Re: cvs commit: src/etc Makefile locale.alias locale.deprecated nls.alias) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 09:40:24 -0000 [moved to -arch] Well. Then I have to rehash $subj issue again. There's important point with removing old (currently unsupported, or correctly to say -- partly supported) locales. They should be removed at installworld stage. We have to define and implement infrastructure to remove old files at installworld stage. But I think there're already someone who has it implemented. Otherwise I'll spend some time, write and commit it. On Tue, Jun 10, 2003 at 02:01:05AM -0700, Doug Barton wrote: > On Tue, 10 Jun 2003, Alexey Zelkin wrote: > > > Hmm... I like this idea. Will send patch to mergemaster mainatiner > > in few minutes. > > I can save you some work... mergemaster doesn't remove files, and > shouldn't. If you need users to remove files, you need to educate them > about the need to do so. We can't educate and force everybody to do right things. And since we can take care people from making mistakes - we should to do it, IMHO. From owner-freebsd-arch@FreeBSD.ORG Tue Jun 10 02:53:11 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8717C37B401 for ; Tue, 10 Jun 2003 02:53:11 -0700 (PDT) Received: from mail-relay1.yahoo.com (mail-relay1.yahoo.com [216.145.48.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1648343FDF for ; Tue, 10 Jun 2003 02:53:11 -0700 (PDT) (envelope-from DougB@FreeBSD.org) Received: from master.dougb.net (12-234-22-23.client.attbi.com [12.234.22.23]) (using TLSv1 with cipher EDH-RSA-DES-CBC3-SHA (168/168 bits)) (No client certificate requested) by mail-relay1.yahoo.com (Postfix) with ESMTP id BB23B8B5E4; Tue, 10 Jun 2003 02:53:10 -0700 (PDT) Date: Tue, 10 Jun 2003 02:53:09 -0700 (PDT) From: Doug Barton To: Alexey Zelkin In-Reply-To: <20030610124747.A7560@phantom.cris.net> Message-ID: <20030610024524.D23396@znfgre.qbhto.arg> References: <20030610124747.A7560@phantom.cris.net> Organization: http://www.FreeBSD.org/ X-message-flag: Outlook -- Not just for spreading viruses anymore! MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: =?koi8-r?B?4c7E0sXKIP7F0s7P1w==?= cc: arch@freebsd.org Subject: Re: removing stale files (was: Re: cvs commit: src/etc Makefile locale.alias locale.deprecated nls.alias) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 09:53:11 -0000 On Tue, 10 Jun 2003, Alexey Zelkin wrote: > [moved to -arch] > > Well. Then I have to rehash $subj issue again. There's important > point with removing old (currently unsupported, or correctly to say > -- partly supported) locales. They should be removed at installworld > stage. I think that's a better way to do it. This same topic of removing stale files at installworld time has been discussed before, and it seems to be the least evil solution. I hate to be so rigid about not having mergemaster do it, but it was a principle at the beginning, and users are now justifiably reliant on things being consistent. If we're going to change our historical behavior of not deleting files then it should be something entirely new. Doug -- This .signature sanitized for your protection From owner-freebsd-arch@FreeBSD.ORG Tue Jun 10 16:23:32 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 12DD437B401 for ; Tue, 10 Jun 2003 16:23:32 -0700 (PDT) Received: from blackhelicopters.org (geburah.blackhelicopters.org [209.69.178.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 539B043F75 for ; Tue, 10 Jun 2003 16:23:31 -0700 (PDT) (envelope-from mwlucas@blackhelicopters.org) Received: from blackhelicopters.org (mwlucas@localhost [127.0.0.1]) by blackhelicopters.org (8.12.8/8.12.8) with ESMTP id h5ANNUSW015886; Tue, 10 Jun 2003 19:23:30 -0400 (EDT) (envelope-from mwlucas@blackhelicopters.org) Received: (from mwlucas@localhost) by blackhelicopters.org (8.12.8/8.12.8/Submit) id h5ANNT68015885; Tue, 10 Jun 2003 19:23:29 -0400 (EDT) Date: Tue, 10 Jun 2003 19:23:29 -0400 From: "Michael W . Lucas" To: Alexey Zelkin Message-ID: <20030610192329.A15847@blackhelicopters.org> References: <20030610124747.A7560@phantom.cris.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20030610124747.A7560@phantom.cris.net>; from phantom@FreeBSD.org.ua on Tue, Jun 10, 2003 at 12:47:47PM +0300 cc: arch@FreeBSD.org Subject: Re: removing stale files (was: Re: cvs commit: src/etc Makefile locale.alias locale.deprecated nls.alias) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 23:23:32 -0000 [cc trimmed] On Tue, Jun 10, 2003 at 12:47:47PM +0300, Alexey Zelkin wrote: > But I think there're already someone who > has it implemented. Otherwise I'll spend some time, write and commit > it. NetBSD's etcupdate has this functionality, and a bunch of other stuff. Of course, etcupdate started life as our mergemaster... ==ml -- Michael Lucas mwlucas@FreeBSD.org, mwlucas@BlackHelicopters.org http://www.BlackHelicopters.org/~mwlucas/ Absolute OpenBSD: http://www.AbsoluteOpenBSD.com/ From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 00:57:56 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 07F2237B401; Wed, 11 Jun 2003 00:57:56 -0700 (PDT) Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by mx1.FreeBSD.org (Postfix) with ESMTP id 352C143F3F; Wed, 11 Jun 2003 00:57:54 -0700 (PDT) (envelope-from grog@lemis.com) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id F329C527A8; Wed, 11 Jun 2003 17:27:50 +0930 (CST) Date: Wed, 11 Jun 2003 17:27:50 +0930 From: Greg 'groggy' Lehey To: Doug Barton Message-ID: <20030611075750.GB57496@wantadilla.lemis.com> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DKU6Jbt7q3WqK7+M" Content-Disposition: inline In-Reply-To: <20030610024524.D23396@znfgre.qbhto.arg> User-Agent: Mutt/1.4i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 cc: ache@nagual.pp.ru cc: Alexey Zelkin cc: arch@freebsd.org Subject: Re: removing stale files (was: Re: cvs commit: src/etc Makefile locale.alias locale.deprecated nls.alias) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 07:57:56 -0000 --DKU6Jbt7q3WqK7+M Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tuesday, 10 June 2003 at 2:53:09 -0700, Doug Barton wrote: > On Tue, 10 Jun 2003, Alexey Zelkin wrote: > >> [moved to -arch] >> >> Well. Then I have to rehash $subj issue again. There's important >> point with removing old (currently unsupported, or correctly to say >> -- partly supported) locales. They should be removed at installworld >> stage. > > I think that's a better way to do it. This same topic of removing stale > files at installworld time has been discussed before, and it seems to be > the least evil solution. Agreed. I think we should work towards a policy where we say "these directories belong to the system. Don't install your own things there until you know exactly what you are doing.". That would greatly simplify the job of keeping things up to date. Greg -- See complete headers for address and phone numbers --DKU6Jbt7q3WqK7+M Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (FreeBSD) iD8DBQE+5uD+IubykFB6QiMRAumnAJ9Di9FSURHiVaZBqfyM1To6alFwrACgq1AA bSBRPj91AupwpWBMVhZaGio= =lfSi -----END PGP SIGNATURE----- --DKU6Jbt7q3WqK7+M-- From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 07:46:46 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8340537B401; Wed, 11 Jun 2003 07:46:46 -0700 (PDT) Received: from burka.carrier.kiev.ua (burka.carrier.kiev.ua [193.193.193.107]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7F0B343FD7; Wed, 11 Jun 2003 07:46:44 -0700 (PDT) (envelope-from netch@lucky.net) Received: from netch@localhost [127.0.0.1] (netch@localhost [127.0.0.1]) by burka.carrier.kiev.ua with ESMTP id h5BEkcY2070142; Wed, 11 Jun 2003 17:46:39 +0300 (EEST) (envelope-from netch@burka.carrier.kiev.ua) Received: (from netch@localhost) by burka.carrier.kiev.ua (8.12.8p1/8.12.8/Submit) id h5BEkbjY070139; Wed, 11 Jun 2003 17:46:37 +0300 (EEST) (envelope-from netch) Date: Wed, 11 Jun 2003 17:46:37 +0300 From: Valentin Nechayev To: "Greg 'groggy' Lehey" Message-ID: <20030611144637.GO83663@lucky.net> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030611075750.GB57496@wantadilla.lemis.com> X-42: On X-Verify-Sender: verified cc: arch@freebsd.org Subject: Re: removing stale files (was: Re: cvs commit: src/etc Makefile locale.alias locale.deprecated nls.alias) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: netch@lucky.net List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 14:46:46 -0000 Wed, Jun 11, 2003 at 17:27:50, grog wrote about "Re: removing stale files (was: Re: cvs commit: src/etc Makefile locale.alias locale.deprecated nls.alias)": > Agreed. > I think we should work towards a policy where we say "these > directories belong to the system. Don't install your own things there > until you know exactly what you are doing.". That would greatly > simplify the job of keeping things up to date. But such tool should respect a config which says don't touch some files. E.g. admin uses BIND from port and says NO_BIND=yes in make.conf. Tool shouldn't delete files only because they aren't listed from last buildworld. OTOH, one can say make DESTDIR=/somewhere installworld and then move directories to their usual place. -netch- From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 08:33:44 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7A7E037B401; Wed, 11 Jun 2003 08:33:44 -0700 (PDT) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 30D1F43FB1; Wed, 11 Jun 2003 08:33:39 -0700 (PDT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.8/8.12.3) with ESMTP id h5BFXWkA021865; Wed, 11 Jun 2003 09:33:33 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 11 Jun 2003 09:32:03 -0600 (MDT) Message-Id: <20030611.093203.26943899.imp@bsdimp.com> To: grog@freebsd.org From: "M. Warner Losh" In-Reply-To: <20030611075750.GB57496@wantadilla.lemis.com> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> X-Mailer: Mew version 2.1 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: ache@nagual.pp.ru cc: DougB@freebsd.org cc: phantom@FreeBSD.org.ua cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 15:33:44 -0000 In message: <20030611075750.GB57496@wantadilla.lemis.com> "Greg 'groggy' Lehey" writes: : On Tuesday, 10 June 2003 at 2:53:09 -0700, Doug Barton wrote: : > On Tue, 10 Jun 2003, Alexey Zelkin wrote: : > : >> [moved to -arch] : >> : >> Well. Then I have to rehash $subj issue again. There's important : >> point with removing old (currently unsupported, or correctly to say : >> -- partly supported) locales. They should be removed at installworld : >> stage. : > : > I think that's a better way to do it. This same topic of removing stale : > files at installworld time has been discussed before, and it seems to be : > the least evil solution. : : Agreed. : : I think we should work towards a policy where we say "these : directories belong to the system. Don't install your own things there : until you know exactly what you are doing.". That would greatly : simplify the job of keeping things up to date. In the past, it has been suggested that there be a 'make rmobs' target or something similar. I'd strongly oppose a non-turn-off-able doing this as part of installworld. That's too dangerous. I've experimented with some automatically delete obsolete files schemes in the past, and there is *MUCH* potential for foot shooting and *** ******* to make it be on by default. NetBSD's approach in etcupdate is based on a master list of files that have gone away. Warner From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 11:36:59 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7ED9637B401; Wed, 11 Jun 2003 11:36:59 -0700 (PDT) Received: from smtp3.server.rpi.edu (smtp3.server.rpi.edu [128.113.2.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 859FE43FB1; Wed, 11 Jun 2003 11:36:58 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp3.server.rpi.edu (8.12.9/8.12.9) with ESMTP id h5BIaoiJ012794; Wed, 11 Jun 2003 14:36:53 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <20030611.093203.26943899.imp@bsdimp.com> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> Date: Wed, 11 Jun 2003 14:36:49 -0400 To: "M. Warner Losh" , grog@freebsd.org From: Garance A Drosihn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.28 cc: ache@nagual.pp.ru cc: DougB@freebsd.org cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 18:36:59 -0000 At 9:32 AM -0600 6/11/03, M. Warner Losh wrote: >In message: <20030611075750.GB57496@wantadilla.lemis.com> > "Greg 'groggy' Lehey" writes: >: I think we should work towards a policy where we say "these >: directories belong to the system. Don't install your own >: things there until you know exactly what you are doing.". >: That would greatly simplify the job of keeping things up >: to date. > >In the past, it has been suggested that there be a 'make rmobs' rmobs? ... remove obsolete? >target or something similar. I'd strongly oppose a >non-turn-off-able doing this as part of installworld. That's >too dangerous. I've experimented with some automatically >delete obsolete files schemes in the past, and there is >*MUCH* potential for foot shooting and *** ******* to make >it be on by default. I have also tried to implement a few of my own ideas, and often ended up with something that was just too fragile for me to trust. I mean, I knew I could get it to work for *my* system and what *I* need (after some trial-and-error), but I would never trust it enough to recommend that "everyone" could just blindly run it. >NetBSD's approach in etcupdate is based on a master list of >files that have gone away. I think that something small and simple like this is what we should do at first, because it's something we can do in a short amount of time, and still have reasonable confidence that we aren't doing any harm. That's what we are doing with the "oldRC" files, and I think we could easily list some other specific files where we know the user should not still have those files after the upgrade. Out-of-date locale files could be another example. No, a short list won't be a perfect solution, but we can at least look at all the files in that list and say that each file in that list *will* cause trouble if it is not removed. Does the netbsd approach include any "meta-information" in the master list of files which have gone away? Something like "delete this file if __FreeBSD_version > 500113" ? -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 12:06:30 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 038F237B401; Wed, 11 Jun 2003 12:06:30 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id F025443F3F; Wed, 11 Jun 2003 12:06:28 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5BJ6FhS021394; Wed, 11 Jun 2003 12:06:15 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5BJ6Fiw015761; Wed, 11 Jun 2003 12:06:15 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5BJ6Fnd015760; Wed, 11 Jun 2003 12:06:15 -0700 (PDT) Date: Wed, 11 Jun 2003 12:06:15 -0700 From: Marcel Moolenaar To: Garance A Drosihn Message-ID: <20030611190615.GA15695@dhcp01.pn.xcllnt.net> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: grog@freebsd.org cc: ache@nagual.pp.ru cc: DougB@freebsd.org cc: "M. Warner Losh" cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 19:06:30 -0000 On Wed, Jun 11, 2003 at 02:36:49PM -0400, Garance A Drosihn wrote: > > >NetBSD's approach in etcupdate is based on a master list of > >files that have gone away. > > I think that something small and simple like this is what we > should do at first, because it's something we can do in a > short amount of time, and still have reasonable confidence > that we aren't doing any harm. I'm not sure people realize that things are being mixed-up. The removal of, say, stale header files has nothing to do with /etc or configuration. The removal of a header file should be done at installworld time when the new bits that define FreeBSD without said header is being put in place. The same applies to locale files, libraries and binaries. Do not add to the problem space those bits that are handled (or supposed to be handled) by mergemaster. That we don't touch. As for the argument that removing a library, binary or header may break some tools, scripts or sources that depend on it: If that happens, we failed to maintain backward compatibility. The bug is not in the removal of a stale file, but in the fact that the file has become stale. Keeping stale files around only hides the bug. If breaking the backward compatibility was intentional, then removal of the file is mandatory. FWIW, -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 13:08:47 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1B27B37B401; Wed, 11 Jun 2003 13:08:47 -0700 (PDT) Received: from smtp3.server.rpi.edu (smtp3.server.rpi.edu [128.113.2.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0AD3743FDF; Wed, 11 Jun 2003 13:08:44 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp3.server.rpi.edu (8.12.9/8.12.9) with ESMTP id h5BK8eiJ026295; Wed, 11 Jun 2003 16:08:40 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <20030611190615.GA15695@dhcp01.pn.xcllnt.net> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> Date: Wed, 11 Jun 2003 16:08:39 -0400 To: Marcel Moolenaar From: Garance A Drosihn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.28 cc: grog@freebsd.org cc: ache@nagual.pp.ru cc: DougB@freebsd.org cc: "M. Warner Losh" cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 20:08:47 -0000 At 12:06 PM -0700 6/11/03, Marcel Moolenaar wrote: >As for the argument that removing a library, binary or header >may break some tools, scripts or sources that depend on it: >If that happens, we failed to maintain backward compatibility. >The bug is not in the removal of a stale file, but in the >fact that the file has become stale. Keeping stale files >around only hides the bug. That isn't quite what I'm concerned about. I just want to be sure that we are *only* removing files when we *know* what they are. Many suggestions amount to removing anything "which looks old", because "if it is old, it must be bad". That means you're removing files when you don't really know what they contain, or why they're there. Or you remove files which *you* don't need, but someone else will need because they are running freebsd with different build options. Once you know that a file is indeed a stale file from a previous freebsd install, then it's pretty clear that it should be deleted. If there are bugs with backwards- compatibility, then we fix the bugs. I am not concerned with those bugs, I am concerned that the utility might remove files that we had no business removing. I started some work on what seemed like an obviously-workable solution to me, and found that it rapidly gets complicated if you want it to be something that *every* freebsd user could run without doing damage. Real damage, not "We goofed with backward-compatibility" bugs. I know what I would want to do for my next attempt at this, but I also know that I'll have no time for it anytime soon. So, I don't want to discourage anyone else from working on it. At this point, I'd almost be happy to see it done wrong, just to see something real that people could get experience with. Once we have some starting point, then we're more likely to make some progress with it. I'm just saying that a utility with a list of specific files is much less likely to blow off some poor user's foot when they run it. Certainly it makes sense to look at what netbsd has, and see if that gives us something workable. I'd like to see something that at least will "do no damage", even if might not solve all the problems we can think of with stale files. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 14:06:57 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B6C5437B401; Wed, 11 Jun 2003 14:06:57 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id C062A43F85; Wed, 11 Jun 2003 14:06:53 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from athlon.pn.xcllnt.net (athlon.pn.xcllnt.net [192.168.4.3]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5BL6XhS021956; Wed, 11 Jun 2003 14:06:33 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from athlon.pn.xcllnt.net (localhost [127.0.0.1]) by athlon.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5BL6XCJ000784; Wed, 11 Jun 2003 14:06:33 -0700 (PDT) (envelope-from marcel@athlon.pn.xcllnt.net) Received: (from marcel@localhost) by athlon.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5BL6WsQ000783; Wed, 11 Jun 2003 14:06:32 -0700 (PDT) Date: Wed, 11 Jun 2003 14:06:32 -0700 From: Marcel Moolenaar To: Garance A Drosihn Message-ID: <20030611210632.GA695@athlon.pn.xcllnt.net> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: grog@freebsd.org cc: ache@nagual.pp.ru cc: DougB@freebsd.org cc: "M. Warner Losh" cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 21:06:58 -0000 On Wed, Jun 11, 2003 at 04:08:39PM -0400, Garance A Drosihn wrote: > At 12:06 PM -0700 6/11/03, Marcel Moolenaar wrote: > >As for the argument that removing a library, binary or header > >may break some tools, scripts or sources that depend on it: > >If that happens, we failed to maintain backward compatibility. > >The bug is not in the removal of a stale file, but in the > >fact that the file has become stale. Keeping stale files > >around only hides the bug. > > That isn't quite what I'm concerned about. I just want to > be sure that we are *only* removing files when we *know* > what they are. We always know, because we own them. > Many suggestions amount to removing anything > "which looks old", because "if it is old, it must be bad". That's not a good way to approach it. > That means you're removing files when you don't really know > what they contain, or why they're there. Which is bad, because it means you don't know your own source tree. > Or you remove files which *you* don't need, but someone else > will need because they are running freebsd with different > build options. Staleness is not a property that is affected by build options. Hence, the removal of stale files should not be made conditional upon build options. To help out in those cases where parts of our tree aren't built and installed and the user builds and installs off-tree substitutes (which means that ownership of files is messed with), you suppress the wholesale removal of stale files. Those users can be expected to manually remove stale files... > Once you know that a file is indeed a stale file from a > previous freebsd install, then it's pretty clear that it > should be deleted. Precisely. That's what this discussion is (should be) all about. We should never delete anything that isn't or wasn't ours. The past tense here is to plug the philosophical hole that if a file is stale, it's not ours anymore and removing it violates the premise that we should not remove files we don't install. :-) > If there are bugs with backwards- > compatibility, then we fix the bugs. I am not concerned > with those bugs, I am concerned that the utility might > remove files that we had no business removing. That's why files should be removed as part of installworld and not by some random tool or target. The tricky part is time: if I cause a file to become obsolete or stale, then removing it right away is the safest. The longer I wait, the bigger the chance that the file (name) is reused for other purposes and thus the lower the certainty that the file I'm removing is in fact the one that I've made stale. Thus: my commit that causes a file to become stale should include the logic to remove the stale file. On top of that we should make sure that we keep track of those build rules so that we can remove them after some next release. You don't want to keep the logic to remove some stale file forever... > I started some work on what seemed like an obviously-workable > solution to me, and found that it rapidly gets complicated > if you want it to be something that *every* freebsd user > could run without doing damage. Real damage, not "We goofed > with backward-compatibility" bugs. This very much surprises me, because I don't see how you can mess up if you stick to the fundamentals. Do I have a huge blind spot? > I'm just saying that a utility with a list of specific files is > much less likely to blow off some poor user's foot when they > run it. True. The easiest utility is a a simple for loop as the last action of installworld, like: Index: Makefile.inc1 =================================================================== RCS file: /home/ncvs/src/Makefile.inc1,v retrieving revision 1.365 diff -u -r1.365 Makefile.inc1 --- Makefile.inc1 8 Jun 2003 04:15:05 -0000 1.365 +++ Makefile.inc1 11 Jun 2003 21:04:41 -0000 @@ -409,6 +409,12 @@ done cd ${.CURDIR}; ${IMAKE} re${.TARGET:S/world$//} rm -rf ${INSTALLTMP} +.for F in ${STALE_FILES} + rm -f $F +.endfor +.for D in ${STALE_DIRS} + rmdir $D || true +.endfor # # reinstall -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 14:33:23 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6940337B401; Wed, 11 Jun 2003 14:33:23 -0700 (PDT) Received: from smtp4.server.rpi.edu (smtp4.server.rpi.edu [128.113.2.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 756F043FA3; Wed, 11 Jun 2003 14:33:22 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp4.server.rpi.edu (8.12.9/8.12.9) with ESMTP id h5BLXFPx028015; Wed, 11 Jun 2003 17:33:16 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <20030611210632.GA695@athlon.pn.xcllnt.net> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> <20030611210632.GA695@athlon.pn.xcllnt.net> Date: Wed, 11 Jun 2003 17:33:14 -0400 To: Marcel Moolenaar From: Garance A Drosihn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.28 cc: grog@freebsd.org cc: ache@nagual.pp.ru cc: DougB@freebsd.org cc: "M. Warner Losh" cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 21:33:23 -0000 At 2:06 PM -0700 6/11/03, Marcel Moolenaar wrote: >On Wed, Jun 11, 2003, Garance A Drosihn wrote: > > Many suggestions amount to removing anything >> "which looks old", because "if it is old, it must be bad". > >That's not a good way to approach it. Right. That's really the main thing I want to avoid. > > That means you're removing files when you don't really know >> what they contain, or why they're there. > >Which is bad, because it means you don't know your own source >tree. No, it means that "you" (for some value of "you") do not know "my" source tree (for every value of "my", == all freebsd users). > > I'm just saying that a utility with a list of specific files is >> much less likely to blow off some poor user's foot when they >> run it. > >True. The easiest utility is a a simple for loop as the last >action of installworld, like: > >Index: Makefile.inc1 >=================================================================== >RCS file: /home/ncvs/src/Makefile.inc1,v >retrieving revision 1.365 >diff -u -r1.365 Makefile.inc1 >--- Makefile.inc1 8 Jun 2003 04:15:05 -0000 1.365 >+++ Makefile.inc1 11 Jun 2003 21:04:41 -0000 >@@ -409,6 +409,12 @@ > done > cd ${.CURDIR}; ${IMAKE} re${.TARGET:S/world$//} > rm -rf ${INSTALLTMP} >+.for F in ${STALE_FILES} >+ rm -f $F >+.endfor >+.for D in ${STALE_DIRS} >+ rmdir $D || true >+.endfor > > # > # reinstall Perhaps true, although you'll be surprised at how long that list of STALE_FILES will get... If you think this is workable, then try it. I think at this point what we need to do is to get some actual implementations going, and see how they work in practice. I'll see if I can dig up the various lists-of-files that I came up with in my earlier attempt. Well, actually, maybe I shouldn't. This does not need to be an exhaustive list. It would be fine to start with just the files we *know* will cause trouble, and go on from there. I keep trying for the perfect all-encompassing solution, and that's where I get bogged down... -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 14:49:04 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A677F37B404; Wed, 11 Jun 2003 14:49:04 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.attbi.com [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60D1143F93; Wed, 11 Jun 2003 14:49:03 -0700 (PDT) (envelope-from DougB@freebsd.org) Received: from 12-234-22-23.client.attbi.com ([12.234.22.23]) by attbi.com (rwcrmhc11) with SMTP id <2003061121490301300anddse>; Wed, 11 Jun 2003 21:49:03 +0000 Date: Wed, 11 Jun 2003 14:49:02 -0700 (PDT) From: Doug Barton To: Marcel Moolenaar In-Reply-To: <20030611210632.GA695@athlon.pn.xcllnt.net> Message-ID: <20030611144115.E26376@12-234-22-23.pyvrag.nggov.pbz> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> <20030611210632.GA695@athlon.pn.xcllnt.net> Organization: http://www.FreeBSD.org/ X-message-flag: Outlook -- Not just for spreading viruses anymore! MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: grog@freebsd.org cc: ache@nagual.pp.ru cc: "M. Warner Losh" cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 21:49:05 -0000 FWIW, I want to express strong support for Garance's caution here, and Warner's request that this feature NOT be made automatic. More specific comments inline below. On Wed, 11 Jun 2003, Marcel Moolenaar wrote: > On Wed, Jun 11, 2003 at 04:08:39PM -0400, Garance A Drosihn wrote: > > At 12:06 PM -0700 6/11/03, Marcel Moolenaar wrote: > > >As for the argument that removing a library, binary or header > > >may break some tools, scripts or sources that depend on it: > > >If that happens, we failed to maintain backward compatibility. > > >The bug is not in the removal of a stale file, but in the > > >fact that the file has become stale. Keeping stale files > > >around only hides the bug. > > > > That isn't quite what I'm concerned about. I just want to > > be sure that we are *only* removing files when we *know* > > what they are. > > We always know, because we own them. The problem is, we don't know if the user has modified them or not. This is one of the reasons mergemaster exists for /etc stuff. > That's why files should be removed as part of installworld > and not by some random tool or target. The tricky part is > time: if I cause a file to become obsolete or stale, then > removing it right away is the safest. The longer I wait, > the bigger the chance that the file (name) is reused for > other purposes and thus the lower the certainty that the > file I'm removing is in fact the one that I've made stale. I don't agree with the idea of removing files via installworld, but your point about the dangers of file name reuse is well taken. > > I started some work on what seemed like an obviously-workable > > solution to me, and found that it rapidly gets complicated > > if you want it to be something that *every* freebsd user > > could run without doing damage. Real damage, not "We goofed > > with backward-compatibility" bugs. > > This very much surprises me, because I don't see how you can > mess up if you stick to the fundamentals. Do I have a huge > blind spot? Not a blind spot necessarily, but perhaps you haven't had a lot of experience with the infinite "creativity" of our users. I would say that at bare minimum, IF the community consensus is to do this in installworld, that there be a knob to turn it off. I'd still rather see this functionality in a seperate utility though. Doug -- This .signature sanitized for your protection From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 14:57:29 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DB78D37B401; Wed, 11 Jun 2003 14:57:29 -0700 (PDT) Received: from sccrmhc11.attbi.com (sccrmhc11.attbi.com [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1E4BF43F93; Wed, 11 Jun 2003 14:57:29 -0700 (PDT) (envelope-from DougB@freebsd.org) Received: from 12-234-22-23.client.attbi.com ([12.234.22.23]) by attbi.com (sccrmhc11) with SMTP id <2003061121572101100ravqpe>; Wed, 11 Jun 2003 21:57:21 +0000 Date: Wed, 11 Jun 2003 14:57:20 -0700 (PDT) From: Doug Barton To: Greg 'groggy' Lehey In-Reply-To: <20030611075750.GB57496@wantadilla.lemis.com> Message-ID: <20030611145534.N26376@12-234-22-23.pyvrag.nggov.pbz> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> Organization: http://www.FreeBSD.org/ X-message-flag: Outlook -- Not just for spreading viruses anymore! MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: ache@nagual.pp.ru cc: Alexey Zelkin cc: arch@freebsd.org Subject: Re: removing stale files (was: Re: cvs commit: src/etc Makefile locale.alias locale.deprecated nls.alias) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 21:57:30 -0000 On Wed, 11 Jun 2003, Greg 'groggy' Lehey wrote: > I think we should work towards a policy where we say "these > directories belong to the system. Don't install your own things there > until you know exactly what you are doing.". That would greatly > simplify the job of keeping things up to date. I don't disagree with that in concept, but we need to keep the issue of using ports to replace base system elements in scope with this discussion. I'd be hesitant to move forward with any approach that increases the monolithic conglomeration view of the base. Doug -- This .signature sanitized for your protection From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 15:04:17 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 970CD37B401; Wed, 11 Jun 2003 15:04:17 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 84A8C43FB1; Wed, 11 Jun 2003 15:04:16 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from athlon.pn.xcllnt.net (athlon.pn.xcllnt.net [192.168.4.3]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5BM47hS022306; Wed, 11 Jun 2003 15:04:07 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from athlon.pn.xcllnt.net (localhost [127.0.0.1]) by athlon.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5BM47CJ000928; Wed, 11 Jun 2003 15:04:07 -0700 (PDT) (envelope-from marcel@athlon.pn.xcllnt.net) Received: (from marcel@localhost) by athlon.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5BM47LG000927; Wed, 11 Jun 2003 15:04:07 -0700 (PDT) Date: Wed, 11 Jun 2003 15:04:07 -0700 From: Marcel Moolenaar To: Garance A Drosihn Message-ID: <20030611220407.GA870@athlon.pn.xcllnt.net> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> <20030611210632.GA695@athlon.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: grog@freebsd.org cc: ache@nagual.pp.ru cc: DougB@freebsd.org cc: "M. Warner Losh" cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 22:04:17 -0000 On Wed, Jun 11, 2003 at 05:33:14PM -0400, Garance A Drosihn wrote: > > > > That means you're removing files when you don't really know > >> what they contain, or why they're there. > > > >Which is bad, because it means you don't know your own source > >tree. > > No, it means that "you" (for some value of "you") do not know > "my" source tree (for every value of "my", == all freebsd users). I (for some definition of I :-) don't have to know. I only have to care about our (=FreeBSD) tree and I only have to come up with solutions that do not gratuitously prohibit using our (=FreeBSD) tree for unspecified purposes. Having an open mind and listen to people that use our tree in unspecified ways is generally good, but it's too much to ask of FreeBSD developers to worry about those cases. Let us worry about the common case. I'm positive we get the feedback if there's a solution that doesn't break some unknown class of environments. IMO of course. > I think at this point what we need to do is to get some > actual implementations going, and see how they work in practice. Agreed. Any KISS-able mechanism is a good start. It will get complicated all by itself :-) > I'll see if I can dig up the various lists-of-files that I came > up with in my earlier attempt. Well, actually, maybe I shouldn't. > This does not need to be an exhaustive list. It would be fine to > start with just the files we *know* will cause trouble, and go > on from there. I keep trying for the perfect all-encompassing > solution, and that's where I get bogged down... Yup. We can start with the locale files for example. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 15:22:13 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8754A37B401; Wed, 11 Jun 2003 15:22:13 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6E15F43F3F; Wed, 11 Jun 2003 15:22:12 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from athlon.pn.xcllnt.net (athlon.pn.xcllnt.net [192.168.4.3]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5BMLxhS022415; Wed, 11 Jun 2003 15:21:59 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from athlon.pn.xcllnt.net (localhost [127.0.0.1]) by athlon.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5BMLwCJ000962; Wed, 11 Jun 2003 15:21:58 -0700 (PDT) (envelope-from marcel@athlon.pn.xcllnt.net) Received: (from marcel@localhost) by athlon.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5BMLwGs000961; Wed, 11 Jun 2003 15:21:58 -0700 (PDT) Date: Wed, 11 Jun 2003 15:21:58 -0700 From: Marcel Moolenaar To: Doug Barton Message-ID: <20030611222158.GB870@athlon.pn.xcllnt.net> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> <20030611210632.GA695@athlon.pn.xcllnt.net> <20030611144115.E26376@12-234-22-23.pyvrag.nggov.pbz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030611144115.E26376@12-234-22-23.pyvrag.nggov.pbz> User-Agent: Mutt/1.5.4i cc: grog@freebsd.org cc: ache@nagual.pp.ru cc: "M. Warner Losh" cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 22:22:13 -0000 On Wed, Jun 11, 2003 at 02:49:02PM -0700, Doug Barton wrote: > > > > This very much surprises me, because I don't see how you can > > mess up if you stick to the fundamentals. Do I have a huge > > blind spot? > > Not a blind spot necessarily, but perhaps you haven't had a lot of > experience with the infinite "creativity" of our users. Yes, I have and that's why I don't make it my responsibility. No matter how hard we try to keep all the infinite creativity into account, there's always someone with a mind more perverted than we think would be possible. There's no way we can make it fool proof. So why bother? Keep it simple and fundamental and adjust according to feedback. The more perfect our initial implementation the better, but when dealing with a lot of unknowns and uncertainties, any start is a good one. Example: if we get a lot of negative feedback that removing stale files (unconditionally) causes more problems than is acceptable, you can opt to do a md5 checksum and only remove stale files if the checksum matches. We do that with ports and it must work well enough considering the number of problem reports... -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-arch@FreeBSD.ORG Wed Jun 11 20:46:46 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B703637B401; Wed, 11 Jun 2003 20:46:46 -0700 (PDT) Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by mx1.FreeBSD.org (Postfix) with ESMTP id C867E43FB1; Wed, 11 Jun 2003 20:46:43 -0700 (PDT) (envelope-from grog@lemis.com) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id 6A3E1527A7; Thu, 12 Jun 2003 13:16:40 +0930 (CST) Date: Thu, 12 Jun 2003 13:16:40 +0930 From: Greg 'groggy' Lehey To: Garance A Drosihn , Doug Barton Message-ID: <20030612034640.GM57496@wantadilla.lemis.com> References: <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/t6ASE28jIy1gGy9" Content-Disposition: inline In-Reply-To: <20030611144115.E26376@12-234-22-23.pyvrag.nggov.pbz> User-Agent: Mutt/1.4i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 cc: ache@nagual.pp.ru cc: Marcel Moolenaar cc: "M. Warner Losh" cc: arch@freebsd.org Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 03:46:47 -0000 --/t6ASE28jIy1gGy9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wednesday, 11 June 2003 at 14:36:49 -0400, Garance A Drosihn wrote: > At 9:32 AM -0600 6/11/03, M. Warner Losh wrote: >> In message: <20030611075750.GB57496@wantadilla.lemis.com> >> "Greg 'groggy' Lehey" writes: >>> I think we should work towards a policy where we say "these >>> directories belong to the system. Don't install your own >>> things there until you know exactly what you are doing.". >>> That would greatly simplify the job of keeping things up >>> to date. > > I have also tried to implement a few of my own ideas, and > often ended up with something that was just too fragile for > me to trust. I mean, I knew I could get it to work for *my* > system and what *I* need (after some trial-and-error), but > I would never trust it enough to recommend that "everyone" > could just blindly run it. Exactly. This is the biggest problem we have. We want to have the warm fuzzies that it will work, preferably without intervention. On Wednesday, 11 June 2003 at 16:08:39 -0400, Garance A Drosihn wrote: > At 12:06 PM -0700 6/11/03, Marcel Moolenaar wrote: >> As for the argument that removing a library, binary or header >> may break some tools, scripts or sources that depend on it: >> If that happens, we failed to maintain backward compatibility. >> The bug is not in the removal of a stale file, but in the >> fact that the file has become stale. Keeping stale files >> around only hides the bug. > > That isn't quite what I'm concerned about. I just want to > be sure that we are *only* removing files when we *know* > what they are. There are several ways to achieve this. One is the somewhat fascist method I suggest above: for a specific list of directories (/bin, /sbin, /usr/bin, /usr/sbin, /usr/include and so on), we say "this belongs to the system. Don't put anything there". That way we really can remove old stuff without worrying about it. On Wednesday, 11 June 2003 at 14:49:02 -0700, Doug Barton wrote: > On Wed, 11 Jun 2003, Marcel Moolenaar wrote: > >> On Wed, Jun 11, 2003 at 04:08:39PM -0400, Garance A Drosihn wrote: >>> At 12:06 PM -0700 6/11/03, Marcel Moolenaar wrote: >>>> As for the argument that removing a library, binary or header >>>> may break some tools, scripts or sources that depend on it: >>>> If that happens, we failed to maintain backward compatibility. >>>> The bug is not in the removal of a stale file, but in the >>>> fact that the file has become stale. Keeping stale files >>>> around only hides the bug. >>> >>> That isn't quite what I'm concerned about. I just want to >>> be sure that we are *only* removing files when we *know* >>> what they are. >> >> We always know, because we own them. > > The problem is, we don't know if the user has modified them or > not. This is one of the reasons mergemaster exists for /etc stuff. The problem with mergemaster in its current form is that it leaves too much work to the user. I've just installed a 5.0 CD-ROM, upgraded to latest -CURRENT, and run mergemaster -ia. It left me with 268 files to merge by hand. A big part of the problem is exactly the fact that mergemaster can't know whether it can change files like /etc/defaults/rc.conf. That's because we have no official policy about it. Sure, at the top it says: # This is rc.conf - a file full of useful variables that you can set # to change the default startup behavior of your system. You should # not edit this file! But it's not official: "should". /etc is a special case, of course. We can't just blow everything away. But we can extend the concept of /etc/defaults/rc.conf reading /etc/rc.conf to other files as well. With others (/etc/inetd.conf) we can't. But if we were to move all user-modifiable script-like files to a separate subdirectory, we could not only make the upgrade less painful, but also come closer to a read-only root file system. > I would say that at bare minimum, IF the community consensus is to > do this in installworld, that there be a knob to turn it off. Absolutely. > I'd still rather see this functionality in a seperate utility > though. I'd *really* like to get back to the good old days of upgrading with a single "make world". If it's a different target (make autoupgrade, for example), that's fine by me. Greg -- See complete headers for address and phone numbers --/t6ASE28jIy1gGy9 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (FreeBSD) iD8DBQE+5/egIubykFB6QiMRArHXAJ9YjlwoRu/DyAr9r2M3v8z+8KwF3ACfUiiR vWzL9E3o9j1siKcSHHEjy/U= =dTy6 -----END PGP SIGNATURE----- --/t6ASE28jIy1gGy9-- From owner-freebsd-arch@FreeBSD.ORG Thu Jun 12 00:31:25 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0284D37B401; Thu, 12 Jun 2003 00:31:25 -0700 (PDT) Received: from HAL9000.homeunix.com (ip114.bella-vista.sfo.interquest.net [66.199.86.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 28EE743FDD; Thu, 12 Jun 2003 00:31:24 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.9/8.12.9) with ESMTP id h5C7VLaK011743; Thu, 12 Jun 2003 00:31:21 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.9/8.12.9/Submit) id h5C7VJOX011742; Thu, 12 Jun 2003 00:31:19 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Date: Thu, 12 Jun 2003 00:31:19 -0700 From: David Schultz To: "Greg 'groggy' Lehey" Message-ID: <20030612073119.GA11366@HAL9000.homeunix.com> Mail-Followup-To: Greg 'groggy' Lehey , Doug Barton , ache@nagual.pp.ru, Alexey Zelkin , arch@freebsd.org References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030611075750.GB57496@wantadilla.lemis.com> cc: ache@nagual.pp.ru cc: Doug Barton cc: Alexey Zelkin cc: arch@FreeBSD.ORG Subject: Re: removing stale files (was: Re: cvs commit: src/etc Makefile locale.alias locale.deprecated nls.alias) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 07:31:25 -0000 On Wed, Jun 11, 2003, Greg 'groggy' Lehey wrote: > On Tuesday, 10 June 2003 at 2:53:09 -0700, Doug Barton wrote: > > On Tue, 10 Jun 2003, Alexey Zelkin wrote: > > > >> [moved to -arch] > >> > >> Well. Then I have to rehash $subj issue again. There's important > >> point with removing old (currently unsupported, or correctly to say > >> -- partly supported) locales. They should be removed at installworld > >> stage. > > > > I think that's a better way to do it. This same topic of removing stale > > files at installworld time has been discussed before, and it seems to be > > the least evil solution. > > Agreed. > > I think we should work towards a policy where we say "these > directories belong to the system. Don't install your own things there > until you know exactly what you are doing.". That would greatly > simplify the job of keeping things up to date. I like this idea, too, but unfortunately, it seems to be too contentious to move forward with as is. (This has been discussed before.) It also doesn't solve the problem that mergemaster often requires significantly more intervention than it should. Another idea is to develop a simple API that would make it easy for the folks developing new features to write ``upgrade scripts''. Think of this as a smart mergemaster on steriods. The appeal of this approach is that configuration files could be automatically patched, and specific crufty files could be automatically deleted. Such an API should support the following features: - Every upgrade operation can be undone until the administrator specifically requests that the backup files be destroyed. - Upgrades can be classified by ``risk level'', which is the estimated likelihood that the change is so nontrivial that the automatic upgrade will fail. - The upgrade process will clearly report exactly what changes have been made and associate a unique identifier with each ``transaction.'' Developers would be encouraged, but not required, to write these trivial specifications when they make changes that will affect the upgrade process, and there are only a few such changes per release. (Only upgrading between releases or security branches needs to be supported.) To give an example of what I mean: update uucp { desc = "Remove UUCP from the base system"; bsdversion = 500000; // or whatever it is risk = 4; // of 10 action { remove bin/uucp bin/uuname [...] ; } } update etc_debuglog { desc = "Introduce debug.log which gets debug.*" bsdversion = 501100; risk = 6; action { // We wouldn't use ed in the real thing, but the // specific choice would be up to the person // writing the spec...probably patch(1) in some cases update etc/syslog.conf { ed etc/syslog.conf < Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6A29637B401; Thu, 12 Jun 2003 01:29:57 -0700 (PDT) Received: from mx0.freebsd-services.com (survey.codeburst.net [195.149.39.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6DB6C43FCB; Thu, 12 Jun 2003 01:29:56 -0700 (PDT) (envelope-from paul@freebsd-services.com) Received: by mx0.freebsd-services.com (Postfix, from userid 1002) id 4B0341B29D; Thu, 12 Jun 2003 09:29:55 +0100 (BST) Date: Thu, 12 Jun 2003 09:29:55 +0100 From: Paul Richards To: Greg 'groggy' Lehey Message-ID: <20030612082954.GU26927@survey.codeburst.net> References: <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611075750.GB57496@wantadilla.lemis.com> <20030611.093203.26943899.imp@bsdimp.com> <20030612034640.GM57496@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030612034640.GM57496@wantadilla.lemis.com> User-Agent: Mutt/1.5.4i cc: arch@freebsd.org cc: Doug Barton cc: ache@nagual.pp.ru cc: "M. Warner Losh" cc: Marcel Moolenaar Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 08:29:57 -0000 On Thu, Jun 12, 2003 at 01:16:40PM +0930, Greg 'groggy' Lehey wrote: > The problem with mergemaster in its current form is that it leaves too > much work to the user. I've just installed a 5.0 CD-ROM, upgraded to > latest -CURRENT, and run mergemaster -ia. It left me with 268 files > to merge by hand. > > A big part of the problem is exactly the fact that mergemaster can't > know whether it can change files like /etc/defaults/rc.conf. That's > because we have no official policy about it. Sure, at the top it > says: Brian suggested something to me a while ago about improving mergemaster. Why not use the cvs id to diff the file against that revision. If it's the same then just update it, because it hasn't been changed, otherwise fall back to the usual method. Ok, you need a cvs repo to do this, but it's an extra feature for people who have one, which is probably a large percentage of the users who regularly do updates. It could be done in such a way that a remote repo is used for this. The src tree must have been updated from somewhere in the first place. There are some issues with making this universal, but even if it's initially only available to those with local repos it would still be a big improvement. -- Tis a wise thing to know what is wanted, wiser still to know when it has been achieved and wisest of all to know when it is unachievable for then striving is folly. [Magician] From owner-freebsd-arch@FreeBSD.ORG Thu Jun 12 01:36:08 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ED69137B401; Thu, 12 Jun 2003 01:36:06 -0700 (PDT) Received: from mx0.freebsd-services.com (survey.codeburst.net [195.149.39.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id 781D243FA3; Thu, 12 Jun 2003 01:36:05 -0700 (PDT) (envelope-from paul@freebsd-services.com) Received: by mx0.freebsd-services.com (Postfix, from userid 1002) id C9F401B29D; Thu, 12 Jun 2003 09:36:04 +0100 (BST) Date: Thu, 12 Jun 2003 09:36:04 +0100 From: Paul Richards To: Doug Barton Message-ID: <20030612083604.GV26927@survey.codeburst.net> References: <20030610124747.A7560@phantom.cris.net> <20030610024524.D23396@znfgre.qbhto.arg> <20030611.093203.26943899.imp@bsdimp.com> <20030611190615.GA15695@dhcp01.pn.xcllnt.net> <20030611210632.GA695@athlon.pn.xcllnt.net> <20030611144115.E26376@12-234-22-23.pyvrag.nggov.pbz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030611144115.E26376@12-234-22-23.pyvrag.nggov.pbz> User-Agent: Mutt/1.5.4i cc: grog@freebsd.org cc: ache@nagual.pp.ru cc: arch@freebsd.org cc: "M. Warner Losh" cc: Marcel Moolenaar Subject: Re: removing stale files X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 08:36:09 -0000 On Wed, Jun 11, 2003 at 02:49:02PM -0700, Doug Barton wrote: > FWIW, I want to express strong support for Garance's caution here, and > Warner's request that this feature NOT be made automatic. More specific > comments inline below. > > > On Wed, 11 Jun 2003, Marcel Moolenaar wrote: > > > On Wed, Jun 11, 2003 at 04:08:39PM -0400, Garance A Drosihn wrote: > > > At 12:06 PM -0700 6/11/03, Marcel Moolenaar wrote: > > > >As for the argument that removing a library, binary or header > > > >may break some tools, scripts or sources that depend on it: > > > >If that happens, we failed to maintain backward compatibility. > > > >The bug is not in the removal of a stale file, but in the > > > >fact that the file has become stale. Keeping stale files > > > >around only hides the bug. > > > > > > That isn't quite what I'm concerned about. I just want to > > > be sure that we are *only* removing files when we *know* > > > what they are. > > > > We always know, because we own them. > > The problem is, we don't know if the user has modified them or not. This > is one of the reasons mergemaster exists for /etc stuff. I've thought about this problem quite a lot recently. I got as far as thinking of a possible solution. Basically, we implement some code in install (which NetBSD already has) that registers files as they get installed. This gives you a registry of all files that are owned by the system. When you run the next make world it compares the new list with the old list and removes those that are now obsolete. If the file is a library then it gets moved to a compat dir. Additionally, a scan can be done of installed binaries and a report produced that lists which are still using deprecated libraries. The registry of known files can also be referenced by a tool that reports all files that are not meant to be there i.e. the local user has put them there, and with a suitable tool the user can register their own files in the registry if they want them to become supported by this mechanism. There's no reason this can't be used by ports as well. -- Tis a wise thing to know what is wanted, wiser still to know when it has been achieved and wisest of all to know when it is unachievable for then striving is folly. [Magician] From owner-freebsd-arch@FreeBSD.ORG Thu Jun 12 02:21:05 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 530A237B401 for ; Thu, 12 Jun 2003 02:21:05 -0700 (PDT) Received: from mail.chesapeake.net (chesapeake.net [208.142.252.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5565C43FAF for ; Thu, 12 Jun 2003 02:21:04 -0700 (PDT) (envelope-from jroberson@chesapeake.net) Received: from localhost (jroberson@localhost) by mail.chesapeake.net (8.11.6/8.11.6) with ESMTP id h5C9L3E64375 for ; Thu, 12 Jun 2003 05:21:03 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Thu, 12 Jun 2003 05:21:03 -0400 (EDT) From: Jeff Roberson To: arch@freebsd.org Message-ID: <20030612051553.J36168-100000@mail.chesapeake.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: HyperThreading and CPU topologies. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 09:21:05 -0000 While I was at USENIX I started implementing HTT support in ULE. To do this I needed a way for the machine dependant code to tell the MI code about the CPU topology. I decided to make this generic enough to handle architectures more complicated than a few HTT capable cores. What I came up with can be used to describe N tiers of CPU groups each containing a variable number of processors. For the Hyper threading case you make one level of groups that each have all of the logical cores for one physical cpu in that group. For something like NUMA you would have one group for each node and one group for each CPU in that node. This is different from HTT where you have one group for each physical cpu. Below I'll supply the patch for sys/smp.h. One thing to note is that I intend for the smp_topology to be optional. If it is not supplied strict SMP will be assumed. That way MD code for all architectures doesn't have to be updated. Just those who support HTT (Or some other SMT implementation) or NUMA (hah). Here's the structures: Index: smp.h =================================================================== RCS file: /home/ncvs/src/sys/sys/smp.h,v retrieving revision 1.72 diff -r1.72 smp.h 19a20,46 > > /* > * Topology of a NUMA or HTT system. > * > * The top level topology is an array of pointers to groups. Each group > * contains a bitmask of cpus in its group or subgroups. It may also > * contain a pointer to an array of child groups. > * > * The bitmasks at non leaf groups may be used by consumers who support > * a smaller depth than the hardware provides. > * > * The topology may be omitted by systems where all CPUs are equal. > */ > > struct cpu_group { > int cg_mask; /* Mask of cpus in this group. */ > int cg_count; /* Count of cpus in this group. */ > int cg_children; /* Number of children groups. */ > struct cpu_group **cg_child; /* Optional child group. */ > }; > > struct cpu_top { > int ct_count; /* Count of groups. */ > struct cpu_group **ct_group; /* Array of pointers to cpu groups. */ > }; > > extern struct cpu_top *smp_topology; Cheers, Jeff From owner-freebsd-arch@FreeBSD.ORG Thu Jun 12 04:46:00 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BACB537B401 for ; Thu, 12 Jun 2003 04:46:00 -0700 (PDT) Received: from critter.freebsd.dk (esplanaden.cybercity.dk [212.242.40.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8A57143FA3 for ; Thu, 12 Jun 2003 04:45:57 -0700 (PDT) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.9/8.12.9) with ESMTP id h5CBjquw001888; Thu, 12 Jun 2003 13:45:52 +0200 (CEST) (envelope-from phk@phk.freebsd.dk) To: Jeff Roberson From: "Poul-Henning Kamp" In-Reply-To: Your message of "Thu, 12 Jun 2003 05:21:03 EDT." <20030612051553.J36168-100000@mail.chesapeake.net> Date: Thu, 12 Jun 2003 13:45:52 +0200 Message-ID: <1887.1055418352@critter.freebsd.dk> cc: arch@freebsd.org Subject: Re: HyperThreading and CPU topologies. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 11:46:01 -0000 In message <20030612051553.J36168-100000@mail.chesapeake.net>, Jeff Roberson wr ites: >Index: smp.h >=================================================================== >RCS file: /home/ncvs/src/sys/sys/smp.h,v >retrieving revision 1.72 >diff -r1.72 smp.h >19a20,46 >> >> /* >> * Topology of a NUMA or HTT system. >> * >> * The top level topology is an array of pointers to groups. Each group >> * contains a bitmask of cpus in its group or subgroups. It may also >> * contain a pointer to an array of child groups. >> * >> * The bitmasks at non leaf groups may be used by consumers who support >> * a smaller depth than the hardware provides. >> * >> * The topology may be omitted by systems where all CPUs are equal. >> */ >> >> struct cpu_group { >> int cg_mask; /* Mask of cpus in this group. */ u_int ? uint32_t ? >> int cg_count; /* Count of cpus in this group. */ >> int cg_children; /* Number of children groups. */ >> struct cpu_group **cg_child; /* Optional child group. */ >> }; >> >> struct cpu_top { >> int ct_count; /* Count of groups. */ >> struct cpu_group **ct_group; /* Array of pointers to cpu >groups. */ >> }; >> >> extern struct cpu_top *smp_topology; -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Thu Jun 12 11:03:42 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3590F37B401 for ; Thu, 12 Jun 2003 11:03:42 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 564E543FBF for ; Thu, 12 Jun 2003 11:03:39 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5CI3ahS027894; Thu, 12 Jun 2003 11:03:36 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5CI3aiw036648; Thu, 12 Jun 2003 11:03:36 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5CI3ZTb036644; Thu, 12 Jun 2003 11:03:35 -0700 (PDT) Date: Thu, 12 Jun 2003 11:03:35 -0700 From: Marcel Moolenaar To: Poul-Henning Kamp Message-ID: <20030612180335.GA36606@dhcp01.pn.xcllnt.net> References: <20030612051553.J36168-100000@mail.chesapeake.net> <1887.1055418352@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1887.1055418352@critter.freebsd.dk> User-Agent: Mutt/1.5.4i cc: arch@freebsd.org Subject: Re: HyperThreading and CPU topologies. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 18:03:42 -0000 On Thu, Jun 12, 2003 at 01:45:52PM +0200, Poul-Henning Kamp wrote: > In message <20030612051553.J36168-100000@mail.chesapeake.net>, Jeff Roberson wr > ites: > >Index: smp.h > >=================================================================== > >RCS file: /home/ncvs/src/sys/sys/smp.h,v > >retrieving revision 1.72 > >diff -r1.72 smp.h > >19a20,46 > >> > >> /* > >> * Topology of a NUMA or HTT system. > >> * > >> * The top level topology is an array of pointers to groups. Each group > >> * contains a bitmask of cpus in its group or subgroups. It may also > >> * contain a pointer to an array of child groups. > >> * > >> * The bitmasks at non leaf groups may be used by consumers who support > >> * a smaller depth than the hardware provides. > >> * > >> * The topology may be omitted by systems where all CPUs are equal. > >> */ > >> > >> struct cpu_group { > >> int cg_mask; /* Mask of cpus in this group. */ > > u_int ? > > uint32_t ? It's best to make it MD depedendent. I prefer 64-bit masks in ia64. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-arch@FreeBSD.ORG Thu Jun 12 11:29:02 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7ED3337B401 for ; Thu, 12 Jun 2003 11:29:02 -0700 (PDT) Received: from mail.chesapeake.net (chesapeake.net [208.142.252.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id A6CE043FBF for ; Thu, 12 Jun 2003 11:29:01 -0700 (PDT) (envelope-from jroberson@chesapeake.net) Received: from localhost (jroberson@localhost) by mail.chesapeake.net (8.11.6/8.11.6) with ESMTP id h5CISrV78377; Thu, 12 Jun 2003 14:28:53 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Thu, 12 Jun 2003 14:28:52 -0400 (EDT) From: Jeff Roberson To: Marcel Moolenaar In-Reply-To: <20030612180335.GA36606@dhcp01.pn.xcllnt.net> Message-ID: <20030612142802.W36168-100000@mail.chesapeake.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: Poul-Henning Kamp Subject: Re: HyperThreading and CPU topologies. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 18:29:02 -0000 On Thu, 12 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 12, 2003 at 01:45:52PM +0200, Poul-Henning Kamp wrote: > > In message <20030612051553.J36168-100000@mail.chesapeake.net>, Jeff Roberson wr > > ites: > > >Index: smp.h > > >=================================================================== > > >RCS file: /home/ncvs/src/sys/sys/smp.h,v > > >retrieving revision 1.72 > > >diff -r1.72 smp.h > > >19a20,46 > > >> > > >> /* > > >> * Topology of a NUMA or HTT system. > > >> * > > >> * The top level topology is an array of pointers to groups. Each group > > >> * contains a bitmask of cpus in its group or subgroups. It may also > > >> * contain a pointer to an array of child groups. > > >> * > > >> * The bitmasks at non leaf groups may be used by consumers who support > > >> * a smaller depth than the hardware provides. > > >> * > > >> * The topology may be omitted by systems where all CPUs are equal. > > >> */ > > >> > > >> struct cpu_group { > > >> int cg_mask; /* Mask of cpus in this group. */ > > > > u_int ? > > > > uint32_t ? > > It's best to make it MD depedendent. I prefer 64-bit masks in ia64. > I thought I was being consistent with the rest of the file but it turns out that it uses u_int so I'm changing to that. If someone wants to make a pass through the smp code and make a cpu_bitmap_t or something they are welcome. Cheers, Jeff From owner-freebsd-arch@FreeBSD.ORG Thu Jun 12 11:54:44 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5958037B4B3 for ; Thu, 12 Jun 2003 11:54:43 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8E25843F75 for ; Thu, 12 Jun 2003 11:54:41 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from athlon.pn.xcllnt.net (athlon.pn.xcllnt.net [192.168.4.3]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5CIsdhS028213; Thu, 12 Jun 2003 11:54:39 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from athlon.pn.xcllnt.net (localhost [127.0.0.1]) by athlon.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5CIsdK4000564; Thu, 12 Jun 2003 11:54:39 -0700 (PDT) (envelope-from marcel@athlon.pn.xcllnt.net) Received: (from marcel@localhost) by athlon.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5CIsdBZ000563; Thu, 12 Jun 2003 11:54:39 -0700 (PDT) Date: Thu, 12 Jun 2003 11:54:39 -0700 From: Marcel Moolenaar To: Jeff Roberson Message-ID: <20030612185439.GA543@athlon.pn.xcllnt.net> References: <20030612180335.GA36606@dhcp01.pn.xcllnt.net> <20030612142802.W36168-100000@mail.chesapeake.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030612142802.W36168-100000@mail.chesapeake.net> User-Agent: Mutt/1.5.4i cc: arch@freebsd.org cc: Poul-Henning Kamp Subject: Re: HyperThreading and CPU topologies. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 18:54:44 -0000 On Thu, Jun 12, 2003 at 02:28:52PM -0400, Jeff Roberson wrote: > > On Thu, 12 Jun 2003, Marcel Moolenaar wrote: > > > On Thu, Jun 12, 2003 at 01:45:52PM +0200, Poul-Henning Kamp wrote: > > > In message <20030612051553.J36168-100000@mail.chesapeake.net>, Jeff Roberson wr > > > ites: > > > >Index: smp.h > > > >=================================================================== > > > >RCS file: /home/ncvs/src/sys/sys/smp.h,v > > > >retrieving revision 1.72 > > > >diff -r1.72 smp.h > > > >19a20,46 > > > >> > > > >> /* > > > >> * Topology of a NUMA or HTT system. > > > >> * > > > >> * The top level topology is an array of pointers to groups. Each group > > > >> * contains a bitmask of cpus in its group or subgroups. It may also > > > >> * contain a pointer to an array of child groups. > > > >> * > > > >> * The bitmasks at non leaf groups may be used by consumers who support > > > >> * a smaller depth than the hardware provides. > > > >> * > > > >> * The topology may be omitted by systems where all CPUs are equal. > > > >> */ > > > >> > > > >> struct cpu_group { > > > >> int cg_mask; /* Mask of cpus in this group. */ > > > > > > u_int ? > > > > > > uint32_t ? > > > > It's best to make it MD depedendent. I prefer 64-bit masks in ia64. > > > > I thought I was being consistent with the rest of the file but it turns > out that it uses u_int so I'm changing to that. If someone wants to make > a pass through the smp code and make a cpu_bitmap_t or something they are > welcome. I have this on my radar for a while. Mostly because of the PCPU masks. It's not urgent, so the sweep will not happen anytime soon. Hence, it's also not a problem if you favor consistency for now. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 08:27:07 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0AD4437B401 for ; Sat, 14 Jun 2003 08:27:07 -0700 (PDT) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7114643F93 for ; Sat, 14 Jun 2003 08:27:06 -0700 (PDT) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id 2F164530E; Sat, 14 Jun 2003 17:27:05 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: arch@freebsd.org From: Dag-Erling Smorgrav Date: Sat, 14 Jun 2003 17:27:04 +0200 Message-ID: User-Agent: Gnus/5.1001 (Gnus v5.10.1) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Subject: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 15:27:07 -0000 --=-=-= David's latest commit to cdefs.h breaks the build by causing lint to fail for every source file that directly or indirectly includes ; it will similarly cause non-GNU compilers to fail on the same files. This is entirely unnecessary as the patch was only meant to add alloca(3) support on compilers that support it. I'd like to commit the attached patch (after suitable testing of course). It removes all mention of alloca(3) from cdefs.h, and instead modifies the declaration in stdlib.h so that GNU compilers see alloca(sz) defined to __builtin_alloca(sz) while other compilers (and linters) see a regular prototype. I would also like to add (at some future date) a link-time warning for alloca(3) similar to what we already have for gets(), mktemp() etc. DES -- Dag-Erling Smorgrav - des@ofug.org --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=alloca.diff Index: sys/sys/cdefs.h =================================================================== RCS file: /home/ncvs/src/sys/sys/cdefs.h,v retrieving revision 1.70 diff -u -r1.70 cdefs.h --- sys/sys/cdefs.h 14 Jun 2003 06:01:35 -0000 1.70 +++ sys/sys/cdefs.h 14 Jun 2003 15:04:29 -0000 @@ -142,11 +142,6 @@ #define __section(x) __attribute__((__section__(x))) #endif #endif -#ifdef __GNUC__ -#define alloca(sz) __builtin_alloca(sz) -#else -#error FreeBSD alloca support needed for this compiler -#endif /* XXX: should use `#if __STDC_VERSION__ < 199901'. */ #if !(__GNUC__ == 2 && __GNUC_MINOR__ >= 7 || __GNUC__ >= 3) Index: include/stdlib.h =================================================================== RCS file: /home/ncvs/src/include/stdlib.h,v retrieving revision 1.48 diff -u -r1.48 stdlib.h --- include/stdlib.h 12 Mar 2003 20:29:58 -0000 1.48 +++ include/stdlib.h 14 Jun 2003 15:06:02 -0000 @@ -222,7 +222,12 @@ extern void (*_malloc_message)(const char *, const char *, const char *, const char *); -void *alloca(size_t); /* built-in for gcc */ +#ifdef __GNUC__ +#define alloca(sz) __builtin_alloca(sz) +#else +void *alloca(size_t); +#endif + __uint32_t arc4random(void); void arc4random_addrandom(unsigned char *dat, int datlen); --=-=-=-- From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 09:36:25 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 247A237B401 for ; Sat, 14 Jun 2003 09:36:25 -0700 (PDT) Received: from mailout01.sul.t-online.com (mailout01.sul.t-online.com [194.25.134.80]) by mx1.FreeBSD.org (Postfix) with ESMTP id DD66143F85 for ; Sat, 14 Jun 2003 09:36:23 -0700 (PDT) (envelope-from Alexander@Leidinger.net) Received: from fwd05.aul.t-online.de by mailout01.sul.t-online.com with smtp id 19RE0s-00056E-01; Sat, 14 Jun 2003 18:36:22 +0200 Received: from Andro-Beta.Leidinger.net (XVstRqZAoe4oH5vSwedm5xeFxeZB7-tXBd+lg4afDIuh-YdGZCk7QM@[217.83.22.5]) by fmrl05.sul.t-online.com with esmtp id 19RE0b-0DgavY0; Sat, 14 Jun 2003 18:36:05 +0200 Received: from Magelan.Leidinger.net (Magelan [192.168.1.1]) h5EGZjoM079802 for ; Sat, 14 Jun 2003 18:35:45 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from Magelan.Leidinger.net (netchild@localhost [127.0.0.1]) by Magelan.Leidinger.net (8.12.9/8.12.9) with SMTP id h5EGZi0x002318 for ; Sat, 14 Jun 2003 18:35:44 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Date: Sat, 14 Jun 2003 18:35:44 +0200 From: Alexander Leidinger To: freebsd-arch@freebsd.org Message-Id: <20030614183544.051c7a57.Alexander@Leidinger.net> In-Reply-To: References: X-Mailer: Sylpheed version 0.8.10claws (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Seen: false X-ID: XVstRqZAoe4oH5vSwedm5xeFxeZB7-tXBd+lg4afDIuh-YdGZCk7QM@t-dialin.net Subject: Re: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 16:36:25 -0000 On Sat, 14 Jun 2003 17:27:04 +0200 Dag-Erling Smorgrav wrote: > I'd like to commit the attached patch (after suitable testing of > course). It removes all mention of alloca(3) from cdefs.h, and > instead modifies the declaration in stdlib.h so that GNU compilers see > alloca(sz) defined to __builtin_alloca(sz) while other compilers (and > linters) see a regular prototype. Please also add a comment about the actual pitfalls (alloca function not possible to implement on AMD64, ... in libc and broken on IA32). > I would also like to add (at some future date) a link-time warning for > alloca(3) similar to what we already have for gets(), mktemp() etc. Wasn't there consensus to remove it entirely from libc? Bye, Alexander. -- Give a man a fish and you feed him for a day; teach him to use the Net and he won't bother you for weeks. http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7 From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 09:43:15 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 550DD37B401 for ; Sat, 14 Jun 2003 09:43:15 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8381B43FA3 for ; Sat, 14 Jun 2003 09:43:14 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.9/8.12.9) with ESMTP id h5EGh7M7044570 for ; Sat, 14 Jun 2003 09:43:11 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <200306141643.h5EGh7M7044570@gw.catspoiler.org> Date: Sat, 14 Jun 2003 09:43:07 -0700 (PDT) From: Don Lewis To: arch@FreeBSD.org MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Subject: vnode/buf locking deadlock between nfsiod and getblk() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 16:43:15 -0000 The one remaining vnode locking issue in the NFS client code that I'm aware of is that nfs_doio() does stuff with the vnode associated with the buf passed to it that requires the vnode lock to be held, but the vnode is not locked when nfs_nfsiod() calls nfs_doio(). I've made a couple of attempts to fix nfs_nfsiod() by locking the vnode, and I've always run into deadlocks like this: 43 c63f9b58 e4b3c000 0 0 0 0000204 [SLP]nfs 0xc687f794] nfsiod 0 570 c6728790 e6ddb000 1001 563 570 0004002 [SLP]getblk 0xd28980a4] ls mi_switch(c61c6000,50,c051cc8e,cd,0) at mi_switch+0x210 msleep(c687f794,c05dffc4,50,c05299ce,0) at msleep+0x484 acquire(e4b0ec6c,1000000,600,f1,c61c6000) at acquire+0x9e lockmgr(c687f794,1010002,c687f6d8,c61c6000,d2897fd8) at lockmgr+0x387 vop_sharedlock(e4b0ec9c,0,c0524105,360,e4b0ecb0) at vop_sharedlock+0x84 vn_lock(c687f6d8,20002,c61c6000,c0529f78,0) at vn_lock+0xe9 nfssvc_iod(c060d6e0,e4b0ed48,c051a213,30e,0) at nfssvc_iod+0x12a fork_exit(c03e3e10,c060d6e0,e4b0ed48) at fork_exit+0xc0 fork_trampoline() at fork_trampoline+0x1a mi_switch(c6729390,50,c0328ad0,c6729390,0) at mi_switch+0x210 msleep(d28980a4,c05e06ac,50,c0522430,c8) at msleep+0x484 acquire(e6dbc9e8,2000020,600,f1,c6729390) at acquire+0x9e lockmgr(d28980a4,2090022,c687f6d8,c6729390,c687f6d8) at lockmgr+0x387 BUF_TIMELOCK(d2897fd8,10022,c687f6d8,c0522430,0) at BUF_TIMELOCK+0x80 getblk(c687f6d8,1,0,1000,0) at getblk+0x141 nfs_getcacheblk(c687f6d8,1,0,1000,c6729390) at nfs_getcacheblk+0xc9 nfs_bioread(c687f6d8,e6dbccb4,0,c66f2d00,165) at nfs_bioread+0x87a nfs_readdir(e6dbcc34,c05158ca,c05b6c20,c687f6d8,e6dbccb4) at nfs_readdir+0xd4 VOP_READDIR(c687f6d8,e6dbccb4,c66f2d00,e6dbcc84,0) at VOP_READDIR+0x67 getdirentries(c6729390,e6dbcd10,c0537b17,3fd,4) at getdirentries+0x11d syscall(2f,2f,2f,80e2600,80d9040) at syscall+0x26e Xint0x80_syscall() at Xint0x80_syscall+0x1d In this case, 'ls' had a vnode locked and was trying to lock a buf, and 'nfsiod' was waiting to obtain a lock on the same vnode. I finally dug around in the code and discovered that the problem is fairly fundamental. If a thread calls VOP_STRATEGY() to for asynchronous I/O on an NFS mounted filesystem, or if it calls nfs_biord() which decides to do readahead, the request is handled by nfs_asyncio(), which uses BUF_KERNPROC() to transfer ownership of the buf lock to the system, and queues the buf on nmp->nm_bufq for nfsiod to handle later. Everything is fine if nfsiod is able to service the request before another thread requests the buf. The problem occurs when another thread attempts to do I/O on the file, grabs the vnode lock and then tries to grab the buf lock before nfsiod has gotten around servicing the request. The thread requesting the I/O can't proceed until it gets the buf lock, which won't happen until the the queue request has been serviced, and nfsiod can handle the I/O request in the buf because it can't obtain the vnode lock. The only reason that we don't see this failure is that nfsiod is not requesting the vnode lock and is allowing nfs_doio() to play with an unlocked vnode (or one locked by another thread). I came up with three possible ways of fixing this, none of which sound very appealing: Fix nfs_doio() so that it and the functions that it calls don't touch any vnode fields that require the vnode lock. When attempting to lock a buf whose current lockholder is LK_KERNPROC, back off by dropping the vnode lock and retrying. When attempting to lock a buf whose current lockholder is LK_KERNPROC, steal the buf back and do the requested I/O synchronously before proceeding if the previously requested I/O was not already in progress. Comments? Suggestions? From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 10:54:55 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 85AE237B401 for ; Sat, 14 Jun 2003 10:54:55 -0700 (PDT) Received: from storm.FreeBSD.org.uk (storm.FreeBSD.org.uk [194.242.157.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7B38E43FB1 for ; Sat, 14 Jun 2003 10:54:54 -0700 (PDT) (envelope-from mark@grondar.org) Received: from storm.FreeBSD.org.uk (Ugrondar@localhost [127.0.0.1]) by storm.FreeBSD.org.uk (8.12.9/8.12.9) with ESMTP id h5EHsr1f051990 for ; Sat, 14 Jun 2003 18:54:53 +0100 (BST) (envelope-from mark@grondar.org) Received: (from Ugrondar@localhost)h5EHsrGL051989 for arch@freebsd.org; Sat, 14 Jun 2003 18:54:53 +0100 (BST) X-Authentication-Warning: storm.FreeBSD.org.uk: Ugrondar set sender to mark@grondar.org using -f Received: from grondar.org (localhost [127.0.0.1]) by grimreaper.grondar.org (8.12.9/8.12.9) with ESMTP id h5EHpcHh027173 for ; Sat, 14 Jun 2003 18:51:38 +0100 (BST) (envelope-from mark@grondar.org) From: Mark Murray Message-Id: <200306141751.h5EHpcHh027173@grimreaper.grondar.org> To: arch@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0" Content-ID: <27170.1055613077.0@grondar.org> Date: Sat, 14 Jun 2003 18:51:38 +0100 Sender: mark@grondar.org X-Spam-Status: No, hits=1.7 required=5.0 tests=FROM_NO_LOWER,PATCH_UNIFIED_DIFF version=2.55 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 2.55 (1.174.2.19-2003-05-19-exp) Subject: change to config(8) to help with linting the kernel. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 17:54:55 -0000 ------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-ID: <27170.1055613077.1@grondar.org> Hiya The enclosed diff (a not very pretty hack) causes config(8) to emit targets to enable .ln files to be built for linting. It is not very pretty because 1) it can't use a "lint/nolint" flag in the conf/files.* flag because that would take a hectic redesign of config. 2) it therefore emits *.ln-building rules for _all_ targets. this is overkill for .s files, but it does not break things, as these files are never linted. Comments? M -- Mark Murray iumop ap!sdn w,I idlaH ------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-ID: <27170.1055613077.2@grondar.org> Content-Description: diff Index: mkmakefile.c =================================================================== RCS file: /home/ncvs/src/usr.sbin/config/mkmakefile.c,v retrieving revision 1.76 diff -u -d -r1.76 mkmakefile.c --- mkmakefile.c 24 Apr 2003 00:52:58 -0000 1.76 +++ mkmakefile.c 30 Apr 2003 13:38:16 -0000 @@ -707,12 +707,20 @@ tail(np), np); continue; } - if (ftp->f_depends) + if (ftp->f_depends) { + fprintf(f, "%sln: $S/%s%c %s\n", tail(np), + np, och, ftp->f_depends); + fprintf(f, "\t${NORMAL_LINT}\n\n"); fprintf(f, "%so: $S/%s%c %s\n", tail(np), np, och, ftp->f_depends); - else + } + else { + fprintf(f, "%sln: $S/%s%c\n", tail(np), + np, och); + fprintf(f, "\t${NORMAL_LINT}\n\n"); fprintf(f, "%so: $S/%s%c\n", tail(np), np, och); + } } tp = tail(np); compilewith = ftp->f_compilewith; ------- =_aaaaaaaaaa0-- From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 13:04:15 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3811037B401 for ; Sat, 14 Jun 2003 13:04:15 -0700 (PDT) Received: from HAL9000.homeunix.com (ip114.bella-vista.sfo.interquest.net [66.199.86.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7BD1743FAF for ; Sat, 14 Jun 2003 13:04:13 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.9/8.12.9) with ESMTP id h5EK496l033545; Sat, 14 Jun 2003 13:04:09 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.9/8.12.9/Submit) id h5EK47ZX033544; Sat, 14 Jun 2003 13:04:07 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Date: Sat, 14 Jun 2003 13:04:07 -0700 From: David Schultz To: Dag-Erling Smorgrav Message-ID: <20030614200407.GA31706@HAL9000.homeunix.com> Mail-Followup-To: Dag-Erling Smorgrav , arch@FreeBSD.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: cc: arch@FreeBSD.ORG Subject: Re: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 20:04:15 -0000 On Sat, Jun 14, 2003, Dag-Erling Smorgrav wrote: > David's latest commit to cdefs.h breaks the build by causing lint to > fail for every source file that directly or indirectly includes > ; it will similarly cause non-GNU compilers to fail on > the same files. This is entirely unnecessary as the patch was only > meant to add alloca(3) support on compilers that support it. > > I'd like to commit the attached patch (after suitable testing of > course). It removes all mention of alloca(3) from cdefs.h, and > instead modifies the declaration in stdlib.h so that GNU compilers see > alloca(sz) defined to __builtin_alloca(sz) while other compilers (and > linters) see a regular prototype. alloca() is, by necessity, a compiler feature. AFAIK, you shouldn't need to #define alloca __builtin_alloca, since any compiler should be providing it if it is supported at all. I think simply declaring it should be sufficient; at least this is the case for gcc. > I would also like to add (at some future date) a link-time warning for > alloca(3) similar to what we already have for gets(), mktemp() etc. I worry a little bit about overdoing those. The gets() and mktemp() warnings are there for security reasons; it's virtually impossible to use those functions without introducing a possibly security-relevant bug. An alloca() warning would be more of a portability warning, which is of no interest to the end users of the software as long as alloca() works, and redundant if it doesn't work. Security, on the other hand, *is* a concern to end users. Then again, there's virtually no software out there that uses alloca(), so if you have your heart set on it, I can hardly object to a warning that I never see. From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 13:29:09 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 62A3337B401 for ; Sat, 14 Jun 2003 13:29:09 -0700 (PDT) Received: from dragon.nuxi.com (trang.nuxi.com [66.93.134.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id C0D8C43FBD for ; Sat, 14 Jun 2003 13:29:08 -0700 (PDT) (envelope-from obrien@NUXI.com) Received: from dragon.nuxi.com (obrien@localhost [127.0.0.1]) by dragon.nuxi.com (8.12.9/8.12.9) with ESMTP id h5EKT3Oc096042; Sat, 14 Jun 2003 13:29:07 -0700 (PDT) (envelope-from obrien@dragon.nuxi.com) Received: (from obrien@localhost) by dragon.nuxi.com (8.12.9/8.12.9/Submit) id h5EKT3go096041; Sat, 14 Jun 2003 13:29:03 -0700 (PDT) Date: Sat, 14 Jun 2003 13:29:02 -0700 From: "David O'Brien" To: Dag-Erling Smorgrav , arch@FreeBSD.org Message-ID: <20030614202902.GA95990@dragon.nuxi.com> Mail-Followup-To: David O'Brien , Dag-Erling Smorgrav , arch@FreeBSD.org References: <20030614200407.GA31706@HAL9000.homeunix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030614200407.GA31706@HAL9000.homeunix.com> User-Agent: Mutt/1.4i X-Operating-System: FreeBSD 5.1-BETA Organization: The NUXI BSD Group X-Pgp-Rsa-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Rsa-Keyid: 1024/34F9F9D5 Subject: Re: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: obrien@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 20:29:09 -0000 On Sat, Jun 14, 2003 at 01:04:07PM -0700, David Schultz wrote: > Then again, there's virtually no software out there that uses > alloca(), so if you have your heart set on it, I can hardly object > to a warning that I never see. All the GNU tools -- gcc, binutils, gdb, grep assumes it exists. There are alot of other things in ports that uses it. I think your statement isn't fully substantiated. -- -- David (obrien@FreeBSD.org) From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 15:19:58 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D04B637B401 for ; Sat, 14 Jun 2003 15:19:58 -0700 (PDT) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3796F43FB1 for ; Sat, 14 Jun 2003 15:19:58 -0700 (PDT) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id ECD88530E; Sun, 15 Jun 2003 00:19:56 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: arch@FreeBSD.org References: <20030614200407.GA31706@HAL9000.homeunix.com> From: Dag-Erling Smorgrav Date: Sun, 15 Jun 2003 00:19:56 +0200 In-Reply-To: <20030614200407.GA31706@HAL9000.homeunix.com> (David Schultz's message of "Sat, 14 Jun 2003 13:04:07 -0700") Message-ID: User-Agent: Gnus/5.1001 (Gnus v5.10.1) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 22:19:59 -0000 David Schultz writes: > alloca() is, by necessity, a compiler feature. AFAIK, you > shouldn't need to #define alloca __builtin_alloca, since any > compiler should be providing it if it is supported at all. I > think simply declaring it should be sufficient; at least this is > the case for gcc. Not if you build with -fno-builtin or some other option that disables builtins. This is precisely why -CURRENT is currently broken. >> I would also like to add (at some future date) a link-time warning for >> alloca(3) similar to what we already have for gets(), mktemp() etc. > I worry a little bit about overdoing those. The gets() and > mktemp() warnings are there for security reasons; it's virtually > impossible to use those functions without introducing a possibly > security-relevant bug. An alloca() warning would be more of a > portability warning, which is of no interest to the end users of > the software as long as alloca() works, and redundant if it > doesn't work. Security, on the other hand, *is* a concern to end > users. I think you misunderstand. If we actually reference an alloca symbol in libc (which is what it would take to trigger the warning), the program is broken since that alloca is guaranteed not to work. If we're using the builtin alloca, the call will expand inline and the warning will never be triggered. DES -- Dag-Erling Smorgrav - des@ofug.org From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 15:44:55 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A7ADD37B401 for ; Sat, 14 Jun 2003 15:44:55 -0700 (PDT) Received: from HAL9000.homeunix.com (ip114.bella-vista.sfo.interquest.net [66.199.86.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7AF6243FBD for ; Sat, 14 Jun 2003 15:44:54 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.9/8.12.9) with ESMTP id h5EMioXQ001133; Sat, 14 Jun 2003 15:44:51 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.9/8.12.9/Submit) id h5EMimuQ001132; Sat, 14 Jun 2003 15:44:48 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Date: Sat, 14 Jun 2003 15:44:48 -0700 From: David Schultz To: Dag-Erling Smorgrav Message-ID: <20030614224448.GA1080@HAL9000.homeunix.com> Mail-Followup-To: Dag-Erling Smorgrav , arch@freebsd.org References: <20030614200407.GA31706@HAL9000.homeunix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: cc: arch@FreeBSD.ORG Subject: Re: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 22:44:56 -0000 On Sun, Jun 15, 2003, Dag-Erling Smorgrav wrote: > David Schultz writes: > > alloca() is, by necessity, a compiler feature. AFAIK, you > > shouldn't need to #define alloca __builtin_alloca, since any > > compiler should be providing it if it is supported at all. I > > think simply declaring it should be sufficient; at least this is > > the case for gcc. > > Not if you build with -fno-builtin or some other option that disables > builtins. This is precisely why -CURRENT is currently broken. Bleh, I apologize. I was reading: -#ifdef __GNUC__ -#define alloca(sz) __builtin_alloca(sz) -#else -#error FreeBSD alloca support needed for this compiler -#endif as: +#ifdef __GNUC__ +#define alloca(sz) __builtin_alloca(sz) +#else +#error FreeBSD alloca support needed for this compiler +#endif /me crawls back into his cave. From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 15:45:27 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ECCE737B401 for ; Sat, 14 Jun 2003 15:45:26 -0700 (PDT) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4C38E43FAF for ; Sat, 14 Jun 2003 15:45:26 -0700 (PDT) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id 7461A5310; Sun, 15 Jun 2003 00:45:24 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: Alexander Leidinger References: <20030614183544.051c7a57.Alexander@Leidinger.net> From: Dag-Erling Smorgrav Date: Sun, 15 Jun 2003 00:45:23 +0200 In-Reply-To: <20030614183544.051c7a57.Alexander@Leidinger.net> (Alexander Leidinger's message of "Sat, 14 Jun 2003 18:35:44 +0200") Message-ID: User-Agent: Gnus/5.1001 (Gnus v5.10.1) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-arch@freebsd.org Subject: Re: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 22:45:27 -0000 Alexander Leidinger writes: > Please also add a comment about the actual pitfalls (alloca function not > possible to implement on AMD64, ... in libc and broken on IA32). How's this? @@ -222,7 +222,23 @@ extern void (*_malloc_message)(const char *, const char *, const char *, const char *); -void *alloca(size_t); /* built-in for gcc */ +/* + * The alloca() function can't be implemented in C, and on some + * platforms it can't be implemented at all as a callable function. + * The GNU C compiler provides a built-in alloca() which we can use; + * in all other cases, provide a prototype, mainly to pacify various + * incarnations of lint. On platforms where alloca() is not in libc, + * programs which use it will fail to link when compiled with non-GNU + * compilers. + */ +#ifdef __GNUC__ +#ifndef alloca +#define alloca(sz) __builtin_alloca(sz) +#endif +#else +void *alloca(size_t); +#endif + __uint32_t arc4random(void); void arc4random_addrandom(unsigned char *dat, int datlen); DES -- Dag-Erling Smorgrav - des@ofug.org From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 16:08:33 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B9D0837B401 for ; Sat, 14 Jun 2003 16:08:33 -0700 (PDT) Received: from wall.polstra.com (wall-gw.polstra.com [206.213.73.130]) by mx1.FreeBSD.org (Postfix) with ESMTP id A0F5943FA3 for ; Sat, 14 Jun 2003 16:08:32 -0700 (PDT) (envelope-from jdp@polstra.com) Received: from strings.polstra.com (strings.polstra.com [206.213.73.20]) by wall.polstra.com (8.12.3p2/8.12.3) with ESMTP id h5EN8VPM032167 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Sat, 14 Jun 2003 16:08:31 -0700 (PDT) (envelope-from jdp@strings.polstra.com) Received: (from jdp@localhost) by strings.polstra.com (8.12.6/8.12.6/Submit) id h5EN8VEK015107; Sat, 14 Jun 2003 16:08:31 -0700 (PDT) (envelope-from jdp) Date: Sat, 14 Jun 2003 16:08:31 -0700 (PDT) Message-Id: <200306142308.h5EN8VEK015107@strings.polstra.com> To: arch@freebsd.org From: John Polstra In-Reply-To: References: Organization: Polstra & Co., Seattle, WA X-Bogosity: No, tests=bogofilter, spamicity=0.499759, version=0.11.2 cc: des@ofug.org Subject: Re: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 23:08:34 -0000 In article , Dag-Erling Smorgrav wrote: > --=-=-= > I'd like to commit the attached patch (after suitable testing of > course). It removes all mention of alloca(3) from cdefs.h, and > instead modifies the declaration in stdlib.h so that GNU compilers see > alloca(sz) defined to __builtin_alloca(sz) while other compilers (and > linters) see a regular prototype. I tried your patch, but it broke world in ranlib: ===> gnu/usr.bin/binutils/ranlib cc -O -pipe -march=pentiumpro -D_GNU_SOURCE -I. -I/a/src/gnu/usr.bin/binutils/ranlib -I/a/src/gnu/usr.bin/binutils/ranlib/../libbfd/i386 -I/a/src/gnu/usr.bin/binutils/ranlib/../../../../contrib/binutils/include -I/a/src/gnu/usr.bin/binutils/ranlib/../libbinutils -I/a/src/gnu/usr.bin/binutils/ranlib/../../../../contrib/binutils/binutils -I/a/src/gnu/usr.bin/binutils/ranlib/../../../../contrib/binutils/bfd -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -c /a/src/gnu/usr.bin/binutils/ranlib/../../../../contrib/binutils/binutils/ar.c In file included from /a/src/contrib/binutils/binutils/bucomm.h:64, from /a/src/contrib/binutils/binutils/ar.c:32: /usr/obj/a/src/i386/usr/include/stdlib.h:226:1: "alloca" redefined In file included from /a/src/contrib/binutils/binutils/ar.c:30: /a/src/contrib/binutils/include/libiberty.h:289:1: this is the location of the previous definition *** Error code 1 John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Two buttocks cannot avoid friction." -- Malawi saying From owner-freebsd-arch@FreeBSD.ORG Sat Jun 14 16:48:40 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0F58637B401 for ; Sat, 14 Jun 2003 16:48:40 -0700 (PDT) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id EC24F43FD7 for ; Sat, 14 Jun 2003 16:48:38 -0700 (PDT) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id 50B43530F; Sun, 15 Jun 2003 01:48:35 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: John Polstra References: <200306142308.h5EN8VEK015107@strings.polstra.com> From: Dag-Erling Smorgrav Date: Sun, 15 Jun 2003 01:48:34 +0200 In-Reply-To: <200306142308.h5EN8VEK015107@strings.polstra.com> (John Polstra's message of "Sat, 14 Jun 2003 16:08:31 -0700 (PDT)") Message-ID: User-Agent: Gnus/5.1001 (Gnus v5.10.1) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" cc: arch@freebsd.org Subject: Re: unbreaking alloca X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jun 2003 23:48:40 -0000 --=-=-= John Polstra writes: > I tried your patch, but it broke world in ranlib: I got that too, you need to wrap the #define in #ifndef alloca / #endif. See the attached updated patch. DES -- Dag-Erling Smorgrav - des@ofug.org --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=alloca.diff Index: sys/sys/cdefs.h =================================================================== RCS file: /home/ncvs/src/sys/sys/cdefs.h,v retrieving revision 1.70 diff -u -r1.70 cdefs.h --- sys/sys/cdefs.h 14 Jun 2003 06:01:35 -0000 1.70 +++ sys/sys/cdefs.h 14 Jun 2003 15:04:29 -0000 @@ -142,11 +142,6 @@ #define __section(x) __attribute__((__section__(x))) #endif #endif -#ifdef __GNUC__ -#define alloca(sz) __builtin_alloca(sz) -#else -#error FreeBSD alloca support needed for this compiler -#endif /* XXX: should use `#if __STDC_VERSION__ < 199901'. */ #if !(__GNUC__ == 2 && __GNUC_MINOR__ >= 7 || __GNUC__ >= 3) Index: include/stdlib.h =================================================================== RCS file: /home/ncvs/src/include/stdlib.h,v retrieving revision 1.48 diff -u -r1.48 stdlib.h --- include/stdlib.h 12 Mar 2003 20:29:58 -0000 1.48 +++ include/stdlib.h 14 Jun 2003 23:47:55 -0000 @@ -222,7 +222,23 @@ extern void (*_malloc_message)(const char *, const char *, const char *, const char *); -void *alloca(size_t); /* built-in for gcc */ +#ifndef alloca +/* + * The alloca() function can't be implemented in C, and on some + * platforms it can't be implemented at all as a callable function. + * The GNU C compiler provides a built-in alloca() which we can use; + * in all other cases, provide a prototype, mainly to pacify various + * incarnations of lint. On platforms where alloca() is not in libc, + * programs which use it will fail to link when compiled with non-GNU + * compilers. + */ +#ifdef __GNUC__ +#define alloca(sz) __builtin_alloca(sz) +#else +void *alloca(size_t); +#endif +#endif + __uint32_t arc4random(void); void arc4random_addrandom(unsigned char *dat, int datlen); --=-=-=--