Date: Wed, 07 Aug 2002 21:18:05 +1000 From: BSD Freak <bsd-freak@mbox.com.au> To: Roger 'Rocky' Vetterberg <listsub@401.cx>, FreeBSD Questions <freebsd-questions@FreeBSD.ORG> Subject: Re: There must be a better way to maintain older systems Message-ID: <e5fe64e5c22e.e5c22ee5fe64@mbox.com.au>
next in thread | raw e-mail | index | archive | help
I think what many don't realise is how much testing, backup, documentation etc. goes into our upgrade process before we roll out a new version of FreeBSD. We only perform binary upgrades (I see building from source, cvsup etc. far too risky). Many of you will no doubt disagree very strongly but we are completely reliant on our FreeBSD boxes being in tip top shape - mistakes cannot be tolerated. These are not machines we run as a hobby, if they are down - our enterprise is down. We also run some complex configurations involving multiple jails etc. Hence our upgrade procedure is as follows for each server (with slight variations): 1. We start off by testing the application on a test server with identical hardware and the new version of FreeBSD we wish to migrate to. 2. We review any new features of the new version we should be implementing including new/improved kernel options, sysctl paramters, optimisations etc. and test these. 3. We do a "clean" install of the new version of FreeBSD and our application on the test server. Thoroughly documenting the whole build process). 4. We copy all the application data from the production to the test server. 5. (This is the tricky bit) We either freeze data (where possible) or take the production server offline and plug into a seperate network with the test server and synchronise the data (usually with rsync). This step is very time critical and we have to do this as fast as possible because the production server is offline. 6. We switch the ip addresses of the production and test server and the test server now becomes our new production server. 7. We keep the old production server around (but off) for about a week to make sure all is well with the new setup. 8. We wipe the old production server and it becomes the new test server for the next box we need to upgrade. * Voila, upgrade complete with minimal possible downtime, minimal risk and a "clean install" of the new version of the OS. * The process now starts again for the next (of our 14) FreeBSD servers. Total time for this process is 2-3 weeks sometimes a month depending on my sysadmin and other workload. Because of the length of this process I have started skipping releases and upgrading servers every second release. Any suggestions on how I can improve this process without introducing higher risks would be *GREATLY* appreciated...... ----- Original Message ----- From: Roger 'Rocky' Vetterberg <listsub@401.cx> Date: Wednesday, August 7, 2002 1:23 pm Subject: Re: There must be a better way to maintain older systems > BSD Freak wrote: > > Hi all, > > > > I am responsible for maintaining 14 FreeBSD, 1 Windows 2000 and > 1 > > Solaris servers at three sites. While I am certianly no fan of > Windows > > 2000 or the commercial UNIX distributions I have to say they > take up a > > lot less of my time to maintain. For example I can download > (binary > > packages) patches and "Service Packs"/hotfixes to patch bugs and > > vulnerabilities and then I forget about it. Upgrades of OS > happen once > > every 3-4 years (and usually accomany a hardware upgrade which > makes it > > a bit neater and less risky). > > > > With FreeBSD however I find myself upgrading every six months or > so > > when a new version is released. I spend half my time upgrading > the 14 > > production servers (in the middle of the night usually!), then > by the > > time I have gotten around to the last system, I'm usually only a > month > > or so away from the next -RELEASE and I I have to do it all > again if I > > am to keep my systems secure and current. > > > > I find myself thinking there *MUST* be a better way. I am quite > happy > > with the stability/features of older versions (ie 4.4-R 4.5-R > etc). > > Surely I don't have go through this upgrade cycle every six > months! It > > would be great to just run a pkg_add which would overwrite any > insecure > > binaries with newer patched ones (and do an actual binary > upgrade only > > when absolutely required - e.g. every 2-3 years). I am even > thinking of > > starting such a project myself. > > > > Am I missing something? (i.e. is there a better way?) > > (If someone tells me to cvsup and do a makeworld on my busy > production > > servers I will scream!) > > I understand that you do not wish to run make buildworld on a lot > of production machines, but there is another way. > I have a machine whichs only task in life is to run make > buildworld. It does nothing but cvsup its sources and build > binaries to share with other machines. Doing a make installworld > takes only a few minutes, reboot included, which is acceptable or > atleast unavoidable even on production machines. Im sure a lot of > the binary patches for your win2k server requires you to reboot > too, dont they? > With 14 machines, I would dedicate one of them as a 'builder'. > Let it buildworld, share /usr/src and /usr/obj via NFS, mount > them on the other machines and I would guess you could upgrade > all 14 machines with 40-50 minutes of work. A few simple scripts > and you could do it in 10. > > -- > R > > > > > > --------------------------------------------------------------------- Never lose a fax again, receive faxes to your personal email account! Visit http://www.mbox.com.au/fax To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e5fe64e5c22e.e5c22ee5fe64>