From owner-freebsd-hackers Sat Sep 16 21:24: 8 2000 Delivered-To: freebsd-hackers@freebsd.org Received: from dt051n37.san.rr.com (dt051n37.san.rr.com [204.210.32.55]) by hub.freebsd.org (Postfix) with ESMTP id AAB1C37B422 for ; Sat, 16 Sep 2000 21:24:00 -0700 (PDT) Received: from gorean.org (doug@master [10.0.0.2]) by dt051n37.san.rr.com (8.9.3/8.9.3) with ESMTP id VAA52144 for ; Sat, 16 Sep 2000 21:23:55 -0700 (PDT) (envelope-from DougB@gorean.org) Message-ID: <39C4475A.DBDA6C5@gorean.org> Date: Sat, 16 Sep 2000 21:23:54 -0700 From: Doug Barton Organization: Triborough Bridge & Tunnel Authority X-Mailer: Mozilla 4.75 [en] (X11; U; FreeBSD 5.0-CURRENT-091 i386) X-Accept-Language: en MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: mergemaster RFC (long) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Greetings campers, Mergemaster first made its appearance as a port two years ago tomorrow, and became part of the FreeBSD source tree last October 20th. Lots of things have changed since the original version of the program (or script, whatever), and there are some proposals on the board for it that I'd really like to get some input on. For those that aren't aware, mm started its life kind of on a dare. My life before getting the full time job that I have now was mostly doing consulting work, so I did a lot of updates remotely. Making the system safe for remote upgrading has always been a major goal of mine. The proto-mergemaster was just a little script that I used because I got tired of forgetting the little bits and pieces. During a conversation on line about how the configuration system should be expanded, I mentioned that I had a little script that helped during upgrades, and a lot of people wanted a copy. Since I had to clean the code up anyway, and since certain people said it couldn't be done, I created what turned into the mergemaster we know and love today. I had some design goals when I started the project, most of which came from my background in doing remote upgrades. The first goal was to make mm as secure as possible. There are numerous precautions built in to avoid overwriting files inappropriately, avoid typical /tmp file overwriting exploits, etc. I also wanted it to be as independent as possible from having to know anything about specific files. This would allow it to continue to be useful no matter what got changed or upgraded. To accomplish this goal I mercilessly cribbed some ideas from Nik Clayton's make world page. The next goal was to make the program as simple as possible given the complexity of some of the stuff that it deals with. I tried to accomplish that by making it as unix like as possible, with tools, options, etc. that act as much like "regular" unix tools as my knowledge and ability allowed. Finally, one of the central design goals was to never take any action that the user didn't specifically request. ALL of the defaults in the program are to do nothing, because in almost all cases that's the safest thing to (not) do. That final goal has been compromised somewhat in the past year, as different people have requested various hacks to avoid having to look at certain files that they never want to upgrade. I'm uncomfortable with this for several reasons. First, because of the way mm works it's already possible (and in fact, desirable) to "hide" local modifications to files by keeping their cvs $FreeBSD ($Id) tags the same as the current version in source. In this way, you only have to deal with the file when there are actual updates, which I feel strongly is something you _should_ do anyway. However, over time I've come to accept the fact that mm now belongs more to the community than to me exclusively, so what *I* think you should do is less important than it was. The other struggle I'm having currently is that the thing is really becoming a beast. I wanted it to be small, simple and fast, and in the tradition of unix tools to do one thing and do it well. The thing already has WAY more options than I ever use, which isn't necessarily the ultimate barometer, but it is starting to give me the willies. The other reason I'm growing increasingly concerned is the number of posts in -questions, et al that go something like; "I see in UPDATING that I should be using mergemaster, but I tried it and I don't really understand what the hell it does, so I just didn't bother." I tried to make the man page as simple as possible, and include stuff that I wish was present in other man pages, but it's not uncommon for people to not even bother reading the man page, or read it, get overwhelmed by the options, give up, and go anyway. All of that is background to aid in understanding where I'm coming from when I talk about the following. There are a couple of categories of proposed changes. Some are things that I'm definitely going to do. Some are things that I plan to do, and am currently working on and/or have patches for. I'm including those two categories because if I don't someone will feel the need to suggest them. :) The final category are things that I would like comments about, hence the subject of this e-mail. First, the things I am definitely going to do. Christian "naddy" Weisgerber has taken on the task of porting mm to openbsd. He has made some very reasonable requests that will make his life easier and reduce gratuitous differences between versions. Also, several people have asked for the ability to specify DESTDIR, which is an easy fix and definitely an understandable request. Second, the things I am planning to do. Several people have asked for mergemaster to make use of cvs, going all the way back to freebsdcon. However, it's only been recently that anyone has given me actual concrete plans on things to integrate. The first in this category is an option to check whether the installed version of a file is different from the stock cvs version of that same file, and if it's not, just go ahead and install it. The second is to offer the ability to patch the installed version of the file with the cvs diff as an alternative to merging with sdiff. Both of these are excellent ideas, and although they will take a little more work, they are worth it, and will be done soon. Another option that I've decided to add is cribbed from the dhclient script. I'm going to add variables to the .mergemasterrc file for mergemaster-start and mergemaster-end scripts (names may change). These scripts will run after the temproot environment is built, and after mm is done with it's run. These scripts would be source'd from within mm so all the variables will be available to them. This should allow people to add in their own "options" that I don't want to add to the program. Last, the things I want comment on. Top of this list is the very (very!) frequently requested item of having mm automatically eliminate certain files from consideration. As stated, I am extremely uncomfortable with this idea. However, it comes up SO often that some method of dealing with it has to be agreed on. There are two alternatives that come to mind. The first is to just refer users to the mergemaster-start script idea above and let them nuke whatever they want. If the weight of opinion here is to add the feature, here's how it would look. This is cobbled together from various people's suggestions. We need a list of files somewhere (probably in .mergemasterrc) that people don't ever want to have to deal with via the "normal" mm process. If a file on that list has been updated (according to the cvs tag) then instead of presenting that file to the user, it will be ignored the first time through. Then, at the end of the script there will be a printout similar to what happens when the user leaves a file to do by hand later. Something like: *** The following files from your ignore list have newer versions than the ones you have installed, and will remain for you to deal with by hand: master.passwd crontab Then the user will have a choice of either deleting the temproot directory, or dealing with those files by hand. Overall, I prefer the idea of letting people do it on their own. Next on the list of things I'm open to suggestions on is a small script I wrote that compares the variables in /etc/rc.conf[.local] to the ones in /etc/defaults/rc.conf, and does the same for make.conf. This helps you catch those situations where variables you have and depend on may have changed. It's a short script, so I don't see it as really needing to be on its own, but the question is where in the mm process to put it. I generally run mm before I build my world, since if things blow up I have good backups. Other people do it after world & kernel but before reboot (which is sound advice for new users anyway). That's fine for the rc.conf variables, but doesn't really help you for the make.conf variables. I could add more switches (ack!) to just do the conf file comparisons seperately I suppose... This also brings up the question of which files in /etc are good to examine before a world build. In the past there have been things in make.conf, master.passwd and group that have all bit people and caused failed builds (and/or installs). I have kicked around the idea of having a "pre-build" mode for mergemaster that does the temproot build then checks out these 3 files (any other suggestions?) and does the make.conf comparison for you. Is this overkill? Too much handholding? There was a PR filed in regards to the current situation with the new sendmail moving the aliases file to /etc/mail, and mergemaster's "failure" to deal with that. In my opinion, I don't think that fixing this "problem" is in mergemaster's purview. If the site depends on their aliases file, the administrator should probably take note of the fact that mm is installing a new aliases file in a new location and take appropriate steps. There was also a suggestion to have mm create a symlink in /etc to /etc/mail/aliases, which I definitely do not think is a good idea. If the site needs this, the administrator should handle it. If it's not essential, mm shouldn't be messing with it. The analogy in the PR regarding /.cshrc and /.profile is not really valid, since that is a default part of the system which has no objectionable alternatives. My experience is that anything involving mail is likely to start a holy war, and I want to stay far from it. Additionally there is another alternative that I haven't seen discussed, namely putting both /etc/aliases and /etc/mail/aliases in sendmail.cf. This may be undesirable from the "cleanliness" perspective, but it would solve the problem. I have an idea related to getting users to actually READ the man page which entails checking for the presence of a $HOME/.mergemasterrc file, and if one is not present to print a helpful message. This could get annoying though for experienced users doing an update on a new machine. And the last of the ideas I want feedback on is related to the problem of determining permissions on the files to be installed. To my knowledge there is no existing C utility that will tell you the octal permissions on a file, and preferably also combine them with the umask to spit out what the permissions should be. My current solution for that is a couple of one line perl scripts, but I really want to eliminate that as my contribution to the "don't make anything in the base depend on perl" campaign. I have written a little tiny program in C to do this which could be included along with mergemaster, what do people think? It is possible to do the same thing in sh, but it's very cumbersome, unless someone can suggest some combination of existing utilities that I haven't considered yet. The main problem with doing it in sh (which I've done in the past) is that the umask effectively means "different" things if it's a binary/directory or if it's a regular file, so even though getting the permissions is easy (with ls, whatever) combining it with the umask is hard. Of course, I could always take that whole section out, and just install the files with the permissions created by /usr/src/etc/Makefile. I see less need now of custom permissions than I did ages ago, but what do others think? Ok... those are all of my ideas. Now comment away! If you have some additional suggestions of your own, don't hesitate to add those on as well. I finally have some time to work with this, and I'd really like to get something that's ready to MFC by the 9/22 deadline for the 4.1.1 release. Keep those cards and letters coming, Doug -- "The dead cannot be seduced." - Kai, "Lexx" Do YOU Yahoo!? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message