Date: Tue, 17 Jan 2012 16:18:08 -0800 From: Devin Teske <devin.teske@fisglobal.com> To: "'Julian Elischer'" <julian@freebsd.org>, "'Mark Felder'" <feld@feld.me> Cc: freebsd-hackers@freebsd.org Subject: RE: FreeBSD has serious problems with focus, longevity, and lifecycle Message-ID: <03ab01ccd576$a8b1bee0$fa153ca0$@fisglobal.com> In-Reply-To: <4F15C44F.1030208@freebsd.org> References: <alpine.BSF.2.00.1112211415580.19710@kozubik.com> <op.v78i3yxi34t2sn@tech304> <4F15C44F.1030208@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> -----Original Message----- > From: owner-freebsd-hackers@freebsd.org [mailto:owner-freebsd- > hackers@freebsd.org] On Behalf Of Julian Elischer > Sent: Tuesday, January 17, 2012 10:56 AM > To: Mark Felder > Cc: freebsd-hackers@freebsd.org > Subject: Re: FreeBSD has serious problems with focus, longevity, and life= cycle >=20 > On 1/17/12 7:39 AM, Mark Felder wrote: > > Why is everyone so afraid of running -STABLE? Plenty of stuff gets > > MFC'd. Yeah, I agree -- running -RELEASE is difficult. Hell, it's > > frustrating to us that VMWare only supports -RELEASE and it took until > > ESX 5 to officially support 8.2! > > > > More releases / snapshots of -STABLE helps people on physical servers, > > but anyone who runs VMs on Xen or VMWare won't get any support for > > those versions because they didn't go through the QA process yet. > > FreeBSD is increasingly becoming a third world citizen thanks to > > virtualization efforts being focused on Linux, so I feel that more > > frequent releases won't help as many people as you think. >=20 > I'm going to go both ways on this one. >=20 > Where I used to work (Devin Teske is now there) we used to use the 'stab= le' > branch and rolll our own releases. We still do this today, but have only further enhanced the procedure. Within FreeBSD announcing a new release, we can be ready to ship said-relea= se within 3-6 weeks. However, that's not to say that our customers can take said-new-release eve= ry 3-6 weeks. Our largest customer is not-surprisingly the fastest at turn-= around but at-best could only do maybe 2 releases at-most in a single year.= Our smaller customers are taking OS upgrades much slower (every-other year= if we're lucky; more like once every 3 years). For both our large and small customers, we actively back-port device suppor= t in-accordance with hardware manufacturing windows. This is to explain that our hardware suppliers notify us ahead-of time when= a particular piece of hardware is about to become unavailable. At that tim= e we are usually given a choice of hardware to evaluate as replacements for= the existing EOM production-line. We're usually given at least 3-9 months prior-notice before our current pro= duction-line goes EOL. For each candidate replacement we try each FreeBSD release that we're curre= ntly using in production. If it doesn't work, we try another candidate hard= ware. If we can't seem to find a winning combo that works with our existing= -RELEASE, we then start trying newer releases until we find drivers workin= g for each/every required piece of hardware (network cards, RAID cards, ser= ial ports/cards, Adaptec SCSI cards, Fibre HBAs, etc.). When we find a rele= ase that contains the necessary drivers, we back-port. At this time, it's worth noting that we use ONLY monolithic kernels and we = deliver them via packages. When we back-port new device support, we just ro= ll a new kernel, ship it via package, pkg_add, reboot. Similarly, if the patch is in userland, we package up the replaced binaries= (produced by using the release engineering procedures documented in build(= 7) and [more importantly] release(7)). The net effect is that we run a -RELEASE with patches from either the same = -STABLE, a higher -RELEASE, higher -STABLE, or (as a last-resort) -CURRENT = or HEAD. We've been known to roll FreeBSD-4.11 kernels with bits back-porte= d from RELENG_8. So, I guess what I=E2=80=99m trying to say is that if you're going to take = your production environment extremely seriously (as though 1.5-Trillion glo= bally-economic-dollars depended on it) you-too would take a serious look at= release(7) and the Release Engineer's handbook. It really is worth maintaining your own release, taking required fixes from= -STABLE (preferred) or higher (as necessary) to satisfy your customers (wh= ich are nearly almost always going to have a different release schedule tha= n that-of any OS, be-it FreeBSD, Linux, or other distribution). > the criticality of those systems was hard to over-emphasize. In 2005 we w= orked > out we processed 1.5 trillion dollars of transactions on those systems. >=20 I'm going to have to sit down with DaveR, JPL, and others to get an updated= metric for 2011. I'd be willing to bet that we've increased transactional = load over the years (with respect to FreeBSD, we brought on one new sizable= customer since then and expanded the scope of existing FreeBSD customers t= o new overseas installations as well as several new sites in the States). > The other side of the coin is that we had the resources to have someone (= me) > tracking the branch. That person is now (me) plus I have Dave Robison (DaveR) to lean on occasio= nally (and you're always a great help, DaveR). I'd argue that it's not the man-power... it's whether management sees the i= mportance in allowing one (or two) persons to spin his/her wheels, developi= ng a laudable solution to the problem at-hand (precisely what we've done; m= itigating extra/busy work). However, sometimes you just have to take initiative. The company didn't rea= lly officially "approve" any project that involved re-architecting our inte= rnal release engineering ... I had to take it upon myself over the last 3.5= years to do said monster-undertaking (in my ``spare'' time). > I only spun a release when I thought it was a good time to do > so, but I always had a year or so advance warning of when a new release w= as > likely to be needed so I could select a good moment from over a wide rang= e. Likewise! When Julian was here, the company had the same top-executives (such as Cayf= ord), and likewise, I too have the same guidance. If/When we ever find ourselves in the boat of our deployed release not supp= orting some hardware we ask ourselves three things: 1. Can we supplant missing support by making an out-of-band purchase of old= er hardware? (e.g. an onboard network card doesn't work, so we'll just thro= w an Intel PRO/100 PCI card into one of the extra slots) 2. Can we back-port missing driver support? NOTE: I then go do research to answer yes/no. 3. How hard is it to back-port missing support if above-answer is YES? NOTE: I then look at how hard it is which takes many things into considerat= ion. 4. Is answer to above "days", "weeks", or "fat-chance"? 5. If answer to above is "days", I'm often instructed to do the back-port. 6. If answer is instead "weeks", we way our other options, including... 7. Can we "pre-ship" a newer OS having tested that it "plays-well with othe= rs" (in a heterogeneous environment)(example: shipping 8.1 workstations but= still using a 4.11 server stack because workstations require better Vid-ca= rd support but servers do not). 8. Can we find another supplier that has the ability to source older hardwa= re? 9. Failing any/all "easy" solutions, we then make a blanket statement that = "it's time to move our customers to the next release" in which we then star= t enroads to regression test our product on said newer release as all other= tricks to stay on the current release won't work. It is through the above procedure that we accommodate customers that may or= may-not have the annual budget to either: a. do a "tech refresh" and/or b. update the OS in concordance to abate both EOM and EOL timelines dictated by hardware man= ufacturers (which we can only hope to influence even with *our* sizeable pu= rchasing power; NOTE: We simply can't influence manufacturers like Google, = Microsoft, Apple, etc. can). > Also ran a layer on the top of the sources where I could add cherry-pick= ed back- > ports and changes as part of our release. >=20 Yup, we've maintained that ability to this day. It's the only thing that's = saved us. In transitioning to 8.1 we've already cherry-picked 8 patches in-= wholesale from RELENG_8. > If it came to that maybe all the people who are currently saying they nee= d better > support of the 8.x branch could get together and together, support someon= e to > do that job for them..would 1/5th of a person be too expensive for them? >=20 > if not, what is a reasonable cost? Is it worth 1/20 th of a person? >=20 An interesting thought. Open to discussion on this one (meaning: I'm willin= g to volunteer). --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?03ab01ccd576$a8b1bee0$fa153ca0$>