From owner-freebsd-stable@FreeBSD.ORG Fri Feb 10 23:11:02 2012 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 549A4106564A for ; Fri, 10 Feb 2012 23:11:02 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [76.96.30.56]) by mx1.freebsd.org (Postfix) with ESMTP id 3AE928FC1E for ; Fri, 10 Feb 2012 23:11:01 +0000 (UTC) Received: from omta14.emeryville.ca.mail.comcast.net ([76.96.30.60]) by qmta06.emeryville.ca.mail.comcast.net with comcast id YP5E1i0041HpZEsA6PB1D2; Fri, 10 Feb 2012 23:11:01 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta14.emeryville.ca.mail.comcast.net with comcast id YPAz1i0241t3BNj8aPB0gH; Fri, 10 Feb 2012 23:11:01 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7233B102C1E; Fri, 10 Feb 2012 15:10:59 -0800 (PST) Date: Fri, 10 Feb 2012 15:10:59 -0800 From: Jeremy Chadwick To: Alexander Leidinger Message-ID: <20120210231059.GA25777@icarus.home.lan> References: <20120210145604.Horde.ewjpSpjmRSRPNSH0YRHxgAk@webmail.leidinger.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120210145604.Horde.ewjpSpjmRSRPNSH0YRHxgAk@webmail.leidinger.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: stable@FreeBSD.org Subject: Re: Reducing the need to compile a custom kernel X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 23:11:02 -0000 On Fri, Feb 10, 2012 at 02:56:04PM +0100, Alexander Leidinger wrote: > during some big discussions in the last monts on various lists, one > of the problems was that some people would like to use > freebsd-update but can't as they are using a custom kernel. With all > the kernel modules we provide, the need for a custom kernel should > be small, but on the other hand, we do not provide a small > kernel-skeleton where you can load just the modules you need. My below comments apply to RELENG_8, amd64. Just noting that here. I have no plans to upgrade to RELENG_9 or newer. > This should be easy to change. As a first step I took the generic > kernel and removed all devices which are available as modules, e.g. > the USB section consists now only of the USB_DEBUG option (so that > the module is build like with the current generic kernel). I also > removed some storage drivers which are not available as a module. > The rationale is, that I can not remove CAM from the kernel config > if I let those drivers inside (if those drivers are important > enough, someone will probably fix the problem and add the missing > pieces to generate a module). This is a core (meaning "key") problem as I see it. There is no definitive list of what drivers can/will work as modules vs. being compiled in to the kernel statically. Furthermore, there is evidence on the mailing lists where users experience different behaviour in drivers when built as modules than if statically included: em(4) is one such example (I can dig up the post if needed). Jack often asks me why we include em(4) statically. :-) > Such a kernel would cover situations where people compile their own > kernel because they want to get rid of some unused kernel code (and > maybe even need the memory this frees up). > > The question is, is this enough? Or asked differently, why are you > compiling a custom kernel in a production environment (so I rule out > debug options zhich are not enabled in GENERIC)? Are there options > which you add which you can not add as a module (SW_WATCHDOG comes > to my mind)? If yes, which ones and how important are they for you? Excellent question, and I would love to share with folks the list of things we tweak in our kernel config vs. GENERIC: * Removal: option INET6 * Removal: option SCTP * Removal: option COMPAT_FREEBSD32 * Removal: option COMPAT_FREEBSD4 * Removal: option COMPAT_FREEBSD5 * Removal: option COMPAT_FREEBSD6 * Removal: option COMPAT_FREEBSD7 * Setting: option PRINTF_BUFR_SIZE=256 * Addition: option KDTRACE_HOOKS * Addition: option FLOWTABLE - Note: Despite this, we disable use of flowtable via sysctl due to severe bugs as we have no indication this has been fixed: http://lists.freebsd.org/pipermail/freebsd-stable/2010-November/060289.html * Addition: option BREAK_TO_DEBUGGER * Addition: option ALT_BREAK_TO_DEBUGGER * Addition: option KDB * Addition: option KDB_TRACE * Addition: option DDB * Addition: option DDB_NUMSYM * Addition: option GDB * Addition: option KTR * Addition: option KTR_ENTRIES=262144 * Addition: option KTR_COMPILE=(KTR_SCHED) * Addition: option KTR_MASK=(KTR_SCHED) * Removal: device ata - Note: Our config has the following comment: # NOTE: "device ata" is missing because we use the Modular ATA core # to only include the ATA-related drivers we need (e.g. AHCI). * Removal: device atapifd * Removal: device atapist * Removal: All device/options relating to SCSI controllers * Addition: device atacore * Addition: device ataisa * Addition: device atapci * Addition: device ataahci * Addition: device ataintel * Removal: device ch * Removal: device sa * Removal: All device/options pertaining to RAID controllers * Removal: All device/options pertaining to PCMCIA/CardBus * Removal: All device/options pertaining to parallel ports * Removal: device puc * Removal: All PCI Ethernet NICs (with and without miibus), with the exception being em * Removal: All ISA Ethernet NICs * Removal: All Wireless NICs * Removal: device vlan * Removal: device tun * Removal: device gif * Removal: device faith * Removal: option USB_DEBUG * Removal: device ulpt * Removal: All device/options pertaining to USB serial devices * Removal: All device/options pertaining to USB Ethernet devices * Removal: All device/options pertaining to USB Wireless devices * Removal: All device/options pertaining to FireWire * Addition: device coretemp * Addition: device smbus * Addition: device smb * Addition: device ichsmb * Addition: device ichwd - Note: We do not use features of this driver given known problems with the watchdog firing during ddb> and similar environments. I have no idea if this has been fixed, but I do remember it being confirmed as a problem. * Addition: options ALTQ * Addition: options ALTQ_CBQ * Addition: options ALTQ_RED * Addition: options ALTQ_RIO * Addition: options ALTQ_HFSC * Addition: options ALTQ_CDNR * Addition: options ALTQ_PRIQ * Addition: options ALTQ_NOPCC I want to note here: the pf ALTQ options are a pain in the butt, quite honestly. I've found in the past that removing the ones you don't use won't result in a successful build, thus one must include them all. We do need ALTQ support though, for rate-limiting capability. The NOPCC option is needed for SMP builds, which makes me wonder what the state of SMP is in this regard -- meaning, on non-SMP builds, is it still safe to include ALTQ_NOPCC? Now, most of the device-related ones -- ASSUMING they can be used reliably as kernel modules -- can probably be ignored (meaning if you strip them out, per your proposal, that's fine). However, the list of options above still stands. Also, there are obviously kernel modules which we're using dynamically, such as ahci.ko (/boot/loader.conf ahci_load="yes") and some others: # kldstat Id Refs Address Size Name 1 20 0xffffffff80100000 1d34f30 kernel 2 1 0xffffffff81e36000 11e48 ahci.ko 3 1 0xffffffff82012000 1313a8 zfs.ko 4 1 0xffffffff82144000 1ff1 opensolaris.ko 5 1 0xffffffff82146000 28c93 pf.ko 6 1 0xffffffff8216f000 7f8 accf_http.ko 7 1 0xffffffff82170000 1f0 accf_data.ko So in this case, ZFS is loaded dynamically via zfs_enable="yes" in rc.conf, as is pf. The accf_* combo is pulled in in real-time when one uses Apache with apache22_http_accept_enable="yes". All the other options you see that we adjust we have legitimate need for. I cannot stress (to you and others) how important it is to have many of the debugging options in there; in the case of a kernel panic, in almost every case on the mailing lists, the first response from developers is "can you drop to ddb and..." I realise for folks running remote/embedded environments these probably aren't what they want, but for the majority I see them as being needed more and more I watch the responses from developers. Finally, just throwing this out there: does anyone know if you can include ALT_BREAK_TO_DEBUGGER by itself, without BREAK_TO_DEBUGGER? E.g. we only want ~ inducing a break to ddb, not serial break. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |