From owner-freebsd-isp@FreeBSD.ORG Thu May 28 16:35:06 2009 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B4751106566C for ; Thu, 28 May 2009 16:35:06 +0000 (UTC) (envelope-from steve@kcilink.com) Received: from yertle.kcilink.com (thingy.kcilink.com [74.92.149.59]) by mx1.freebsd.org (Postfix) with ESMTP id 614F38FC16 for ; Thu, 28 May 2009 16:35:06 +0000 (UTC) (envelope-from steve@kcilink.com) Received: from steve.int.kcilink.com (steve.int.kcilink.com [192.168.7.99]) by yertle.kcilink.com (Postfix) with ESMTP id 9A9668A253 for ; Thu, 28 May 2009 12:16:26 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=kcilink.com; s=kci0709; t=1243527386; bh=erY7WqqJEuSKNFxYo9NAs9NXfrkkDY84o0RAGoJaCXI=; h=Message-Id:From:To:In-Reply-To:Content-Type: Content-Transfer-Encoding:Mime-Version:Subject:Date:References; b=SZxXEerfxM00cnGJlno2jmF3KFt6v6ntSY5cADGm0NsMmraQkBKJud2roa7pWKKrU KKB/1Lpfi6nJTxwpnus1NF/r+5qeYnuzVPdnt7MHZ/5fzb/7faZvHvl/A/5+wHBCj3 1gTMW5bXbndJLF9XRsMEHzPb9RSegnO33z94ONyk= Message-Id: From: Steve Scally To: freebsd-isp@freebsd.org In-Reply-To: <20090524120020.6B42E10656B9@hub.freebsd.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Date: Thu, 28 May 2009 12:16:26 -0400 References: <20090524120020.6B42E10656B9@hub.freebsd.org> X-Mailer: Apple Mail (2.935.3) Subject: Re: Avoiding source code on production servers (Neil Neely) X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2009 16:35:07 -0000 > > Tonix (Antonio Nati) wrote: >> I'm in the phase of planning my new generation of FreeBSD servers, >> and >> I would love to make them more easy to upgrade. >> Main problem I have currently is I do not want any source code on >> production server, so freebsd-update is welcome, but... what about >> packages? >> I would use packages, but they are not easy to upgrade, while ports >> can be easy to upgrade, but need to have sources an servers. > The weakness of FreeBSD here is very unfortunate and IMO goes far > beyond > just source vs binary distribution. Working in a mixed environment > where we have begun heavily using CentOS and utilizing yum it's > obvious > how far behind FreeBSD has fallen in this space. Ports lack any > kind of > concept of "Long Term Stable", so if you are running anything in > ports > (like say perl...) then when a security issue comes out you end up > having to install new versions - the default is not to patch the older > versions. For non-production environments that is likely fine, but > for > major production services it is a painful scenario. So you aren't > just > fixing security you are mixing in the concept of adjusting > functionality > as well. > > (A recent perl "security" upgrade moved perl to a new version which > broke compatibility with the Crypt::CBC module requiring a reinstall - > the new version of that from ports forced salting when it hadn't > previously and now applications were needing to be recoded to get it > all > working again.) This seems more like a flaw in the upgrade procedure you have setup and not testing before upgrading. I would assume, being this is the ISP list, you have multiple redundant type boxes of which one can be taken out of production and upgraded and if it fails will not disrupt service. One setup, which I can think of, that can't really be tested is if your hosting sites for customers in that case it would be a bit hard to setup a mirrored test environment for but nothing is impossible. > > At the end of the day FreeBSD of course lets you have all the power to > just apply the patches yourself to the source and you would be > fine. At > the cost that you need to be doing all of this work yourself and can't > rely on nice management tools to help you. Every problem I've ever > encountered with FreeBSD can be easily handled by a FreeBSD expert - > but > when I bring in a new green admin they have a heck of a time making > any > sense of it and I'm drug back into the trenches of managing all this. I am coming from the opposite end, I used to do CentOS work and now work on FreeBSD and after getting over the initial learning curve and methodology of FreeBSD I find it much better than CentOS. It is much easier to screw up a system with a yum upgrade, wrong priority, or a package from the centos-plus repo. Also linux in general doesn't have a centralized place to read pr notices or an automated portaudit system. For me I had to follow a security focus list or CERT list to see which packages were vulnerable then go to the maintainers and get information. Then I had to wait for CentOS to push out an update and that to me is a maintenance nightmare. The only way to get your new admin up to speed is to let them experience the problem and solve it. Internal documentation also helps immensely or a wiki. Without the network specific information available it doesn't matter what OS you use there will be a huge hurtle. In fact if you don't already have this setup why not let your junior guys work on it. In our wiki we have all our procedures for backups, nightly processes, system flows, mail paths, port options, etc. We also have pages listing all issues or steps needed for upgrades of all our specific server types, mail, dns, management and so on. > > > Why the contrast is extra frustrating is that it takes considerable > skill and understanding of the details of an environment to safely > update a production FreeBSD server. Internal documentation would help here as well as a repo for all your one-off configs then multiple changes can be tracked. > Contrast this with CentOS where an > extremely green admin can easily manage it with minimal instruction. > Unlike with the FreeBSD process this has no risk that it will cause > cascading complex issues that require application modification to > restore them to operation. > Yum can be be just as finicky and destructive if not setup or executed correctly. Just my thoughts. > I've been using FreeBSD since the 2.x days in '96 or so, and I love > it - > my tone is critical because I'm sad to see the state of things and > doubly sad that I don't have the time to volunteer with the project to > help do something about it. In most ways I consider FreeBSD > superior to > any linux, however this core issue of maintenance over time has been > driving our shift to using CentOS over the last few years. If a "Long > Term Stable Port Tree" concept were to come along I think that would > plug the hole here. While I lack the time to lead such a charge, I > would be happy to assist if such an effort were to get launched. > > -- > Neil Neely > http://neil-neely.blogspot.com/