Date: Sat, 16 Sep 2000 21:23:54 -0700 From: Doug Barton <DougB@gorean.org> To: freebsd-hackers@freebsd.org Subject: mergemaster RFC (long) Message-ID: <39C4475A.DBDA6C5@gorean.org>
next in thread | raw e-mail | index | archive | help
Greetings campers,
Mergemaster first made its appearance as a port two years ago tomorrow,
and became part of the FreeBSD source tree last October 20th. Lots of
things have changed since the original version of the program (or
script, whatever), and there are some proposals on the board for it that
I'd really like to get some input on.
For those that aren't aware, mm started its life kind of on a dare. My
life before getting the full time job that I have now was mostly doing
consulting work, so I did a lot of updates remotely. Making the system
safe for remote upgrading has always been a major goal of mine. The
proto-mergemaster was just a little script that I used because I got
tired of forgetting the little bits and pieces. During a conversation on
line about how the configuration system should be expanded, I mentioned
that I had a little script that helped during upgrades, and a lot of
people wanted a copy. Since I had to clean the code up anyway, and since
certain people said it couldn't be done, I created what turned into the
mergemaster we know and love today.
I had some design goals when I started the project, most of which came
from my background in doing remote upgrades. The first goal was to make
mm as secure as possible. There are numerous precautions built in to
avoid overwriting files inappropriately, avoid typical /tmp file
overwriting exploits, etc. I also wanted it to be as independent as
possible from having to know anything about specific files. This would
allow it to continue to be useful no matter what got changed or
upgraded. To accomplish this goal I mercilessly cribbed some ideas from
Nik Clayton's make world page. The next goal was to make the program as
simple as possible given the complexity of some of the stuff that it
deals with. I tried to accomplish that by making it as unix like as
possible, with tools, options, etc. that act as much like "regular" unix
tools as my knowledge and ability allowed. Finally, one of the central
design goals was to never take any action that the user didn't
specifically request. ALL of the defaults in the program are to do
nothing, because in almost all cases that's the safest thing to (not)
do.
That final goal has been compromised somewhat in the past year, as
different people have requested various hacks to avoid having to look at
certain files that they never want to upgrade. I'm uncomfortable with
this for several reasons. First, because of the way mm works it's
already possible (and in fact, desirable) to "hide" local modifications
to files by keeping their cvs $FreeBSD ($Id) tags the same as the
current version in source. In this way, you only have to deal with the
file when there are actual updates, which I feel strongly is something
you _should_ do anyway. However, over time I've come to accept the fact
that mm now belongs more to the community than to me exclusively, so
what *I* think you should do is less important than it was.
The other struggle I'm having currently is that the thing is really
becoming a beast. I wanted it to be small, simple and fast, and in the
tradition of unix tools to do one thing and do it well. The thing
already has WAY more options than I ever use, which isn't necessarily
the ultimate barometer, but it is starting to give me the willies. The
other reason I'm growing increasingly concerned is the number of posts
in -questions, et al that go something like; "I see in UPDATING that I
should be using mergemaster, but I tried it and I don't really
understand what the hell it does, so I just didn't bother." I tried to
make the man page as simple as possible, and include stuff that I wish
was present in other man pages, but it's not uncommon for people to not
even bother reading the man page, or read it, get overwhelmed by the
options, give up, and go anyway.
All of that is background to aid in understanding where I'm coming from
when I talk about the following. There are a couple of categories of
proposed changes. Some are things that I'm definitely going to do. Some
are things that I plan to do, and am currently working on and/or have
patches for. I'm including those two categories because if I don't
someone will feel the need to suggest them. :) The final category are
things that I would like comments about, hence the subject of this
e-mail.
First, the things I am definitely going to do. Christian "naddy"
Weisgerber has taken on the task of porting mm to openbsd. He has made
some very reasonable requests that will make his life easier and reduce
gratuitous differences between versions. Also, several people have asked
for
the ability to specify DESTDIR, which is an easy fix and definitely an
understandable request.
Second, the things I am planning to do. Several people have asked for
mergemaster to make use of cvs, going all the way back to freebsdcon.
However, it's only been recently that anyone has given me actual
concrete plans on things to integrate. The first in this category is an
option to check whether the installed version of a file is different
from the stock cvs version of that same file, and if it's not, just go
ahead and install it. The second is to offer the ability to patch the
installed version of the file with the cvs diff as an alternative to
merging with sdiff. Both of these are excellent ideas, and although they
will take a little more work, they are worth it, and will be done soon.
Another option that I've decided to add is cribbed from the dhclient
script. I'm going to add variables to the .mergemasterrc file for
mergemaster-start and mergemaster-end scripts (names may change). These
scripts will run after the temproot environment is built, and after mm
is done with it's run. These scripts would be source'd from within mm so
all the variables will be available to them. This should allow people
to add in their own "options" that I don't want to add to the program.
Last, the things I want comment on. Top of this list is the very
(very!) frequently requested item of having mm automatically eliminate
certain files from consideration. As stated, I am extremely
uncomfortable with this idea. However, it comes up SO often that some
method of dealing with it has to be agreed on. There are two
alternatives that come to mind. The first is to just refer users to the
mergemaster-start script idea above and let them nuke whatever they
want. If the weight of opinion here is to add the feature, here's how it
would look. This is cobbled together from various people's suggestions.
We need a list of files somewhere (probably in .mergemasterrc) that
people don't ever want to have to deal with via the "normal" mm process.
If a file on that list has been updated (according to the cvs tag) then
instead of presenting that file to the user, it will be ignored the
first time through. Then, at the end of the script there will be a
printout similar to what happens when the user leaves a file to do by
hand later. Something like:
*** The following files from your ignore list have newer versions
than the ones you have installed, and will remain for you to
deal with by hand:
master.passwd
crontab
Then the user will have a choice of either deleting the temproot
directory, or dealing with those files by hand. Overall, I prefer the
idea of letting people do it on their own.
Next on the list of things I'm open to suggestions on is a small script
I wrote that compares the variables in /etc/rc.conf[.local] to the ones
in /etc/defaults/rc.conf, and does the same for make.conf. This helps
you catch those situations where variables you have and depend on may
have changed. It's a short script, so I don't see it as really needing
to be on its own, but the question is where in the mm process to put it.
I generally run mm before I build my world, since if things blow up I
have good backups. Other people do it after world & kernel but before
reboot (which is sound advice for new users anyway). That's fine for the
rc.conf variables, but doesn't really help you for the make.conf
variables. I could add more switches (ack!) to just do the conf file
comparisons seperately I suppose...
This also brings up the question of which files in /etc are good to
examine before a world build. In the past there have been things in
make.conf, master.passwd and group that have all bit people and caused
failed builds (and/or installs). I have kicked around the idea of having
a "pre-build" mode for mergemaster that does the temproot build then
checks out these 3 files (any other suggestions?) and does the make.conf
comparison for you. Is this overkill? Too much handholding?
There was a PR filed in regards to the current situation with the new
sendmail moving the aliases file to /etc/mail, and mergemaster's
"failure" to deal with that. In my opinion, I don't think that fixing
this "problem" is in mergemaster's purview. If the site depends on their
aliases file, the administrator should probably take note of the fact
that mm is installing a new aliases file in a new location and take
appropriate steps. There was also a suggestion to have mm create a
symlink in /etc to /etc/mail/aliases, which I definitely do not think is
a good idea. If the site needs this, the administrator should handle it.
If it's not essential, mm shouldn't be messing with it. The analogy in
the PR regarding /.cshrc and /.profile is not really valid, since that
is a default part of the system which has no objectionable alternatives.
My experience is that anything involving mail is likely to start a holy
war, and I want to stay far from it. Additionally there is another
alternative that I haven't seen discussed, namely putting both
/etc/aliases and /etc/mail/aliases in sendmail.cf. This may be
undesirable from the "cleanliness" perspective, but it would solve the
problem.
I have an idea related to getting users to actually READ the man page
which entails checking for the presence of a $HOME/.mergemasterrc file,
and if one is not present to print a helpful message. This could get
annoying though for experienced users doing an update on a new machine.
And the last of the ideas I want feedback on is related to the problem
of determining permissions on the files to be installed. To my knowledge
there is no existing C utility that will tell you the octal permissions
on a file, and preferably also combine them with the umask to spit out
what the permissions should be. My current solution for that is a couple
of one line perl scripts, but I really want to eliminate that as my
contribution to the "don't make anything in the base depend on perl"
campaign. I have written a little tiny program in C to do this which
could be included along with mergemaster, what do people think? It is
possible to do the same thing in sh, but it's very cumbersome, unless
someone can suggest some combination of existing utilities that I
haven't considered yet. The main problem with doing it in sh (which I've
done in the past) is that the umask effectively means "different" things
if it's a binary/directory or if it's a regular file, so even though
getting the permissions is easy (with ls, whatever) combining it with
the umask is hard. Of course, I could always take that whole section
out, and just install the files with the permissions created by
/usr/src/etc/Makefile. I see less need now of custom permissions than I
did ages ago, but what do others think?
Ok... those are all of my ideas. Now comment away! If you have some
additional suggestions of your own, don't hesitate to add those on as
well. I finally have some time to work with this, and I'd really like to
get something that's ready to MFC by the 9/22 deadline for the 4.1.1
release.
Keep those cards and letters coming,
Doug
--
"The dead cannot be seduced."
- Kai, "Lexx"
Do YOU Yahoo!?
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?39C4475A.DBDA6C5>