From owner-freebsd-stable@freebsd.org Tue Mar 20 18:10:51 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76C71F623E6 for ; Tue, 20 Mar 2018 18:10:51 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from resqmta-po-08v.sys.comcast.net (resqmta-po-08v.sys.comcast.net [IPv6:2001:558:fe16:19:96:114:154:167]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "resqmta-po-01v.sys.comcast.net", Issuer "COMODO RSA Organization Validation Secure Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1736A800AF for ; Tue, 20 Mar 2018 18:10:50 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from resomta-po-08v.sys.comcast.net ([96.114.154.232]) by resqmta-po-08v.sys.comcast.net with ESMTP id yLiZe3rFX3paSyLijeOmGQ; Tue, 20 Mar 2018 18:10:49 +0000 Received: from koitsu.org ([71.198.44.84]) by resomta-po-08v.sys.comcast.net with SMTP id yLihe8HXLQ1ZhyLiie8pr9; Tue, 20 Mar 2018 18:10:48 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 43FB11581F8F; Tue, 20 Mar 2018 11:10:47 -0700 (PDT) Date: Tue, 20 Mar 2018 11:10:47 -0700 From: Jeremy Chadwick To: dewayne.geraghty@heuristicsystems.com.au Cc: freebsd-stable@freebsd.org Subject: Re: Stability of 11.1S Message-ID: <20180320181047.GA55267@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.9.4 (2018-02-28) X-CMAE-Envelope: MS4wfNNNYsnvbI46M16cGhmYjDIirCX18HpqAWWIwDSTdpyEg/MBd1b2JB7qjkJRjHE+vvVyla8ZlwzGrm9duCU7mAfjl7YQdPMLCtjiarQzqszzopElYg4R 3zwJ4vxCgR1CA6d7yyhGdxDbmj8OOVuasPqeOx2b8Rl/ZxukUbl+jjGjQPEo6ifT5V/jt5ygGnRtQWQC3/gG+vz+f83iosdskGp/5DALCoizI3eN1bKZj5Vp X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Mar 2018 18:10:51 -0000 (Please keep me CC'd as I am not subscribed to -stable) I haven't seen any issues, but that means very little. Details: Two boxes -- one bare metal, one VPS (QEMU): $ uname -a FreeBSD XXXXXXXXXXXXXXX 11.1-STABLE FreeBSD 11.1-STABLE #0 r330529: Tue Mar 6 11:36:04 PST 2018 root@XXXXXXXXXXXXXXX:/usr/obj/usr/src/sys/X7SBA_RELENG_11_amd64 amd64 $ uptime 10:33a.m. up 13 days, 18:10, 2 users, load averages: 0.15, 0.19, 0.16 $ uname -a FreeBSD XXXXXXXXXXXXXXXX 11.1-STABLE FreeBSD 11.1-STABLE #0 r330753: Sat Mar 10 21:34:20 PST 2018 root@XXXXXXXXXXXXXXXX:/usr/obj/usr/src/sys/XXXXXXXX_RELENG_11_amd64 amd64 $ uptime 10:33a.m. up 9 days, 10:46, 1 user, load averages: 0.31, 0.35, 0.31 Systems were updated recently because I wanted to test Meltdown/Spectre mitigation (more on that below). Prior to that, bare metal was running 9.x with 200+ day uptimes, VPS was running 10.x with 80-90 day uptimes (VPS providers' HV crashed, i.e. not FreeBSD issues). Since load averages on FreeBSD 10.x onward cannot be trusted[1][2], I have to explain the general system specs and loads: Bare metal box is an Intel Core 2 Quad Q9550, 8GB RAM, doing very little other than running Apache + lots of cron jobs for systems stuff + ZFS with several disks (but not OS disk; that's a dedicated SSD w/ UFS + SU (not SUJ). The cron jobs tend to stress the network and disk I/O a bit; ZFS gets used every day, but only "heavily" during LAN file copies to/from it (Samba is involved), and during nightly backups with rsync. VPS box is some form of QEMU-based Intel Haswell CPU, 1GB RAM, doing general things like Apache + postfix + SpamAssassin + some other daemons, and a lot of Perl. Swap is used heavily on this machine. Disks are all vtblk, and I use multiple to get capacity for the needed space for /usr/src and /usr/obj. Everything is UFS + SU (not SUJ). Things off the top of my head that might be relevant to you: 1. r329462 added Meltdown/Spectre mitigation[3][4]. Bare metal box has the below in /boot/loader.conf, since this is a machine that does not need either given its environment: # Disable PTI (Meltdown mitigation) and IBRS (Spectre mitigation); these # are not relevant on this bare-metal system given its environment and # use case. Details of these tunables is here: # https://lists.freebsd.org/pipermail/freebsd-stable/2018-March/088526.html # vm.pmap.pti="0" hw.ibrs_disable="1" VPS box has no tunings of this sort, and ends up with the below, because the hosting provider has no done BIOS + QEMU updates to add IBRS support (they're very aware of it + have attempted it twice but apparently it didn't go well): vm.pmap.pti: 1 hw.ibrs_disable: 1 hw.ibrs_active: 0 2. If your CPU is an AMD Ryzen, there is a VERY long discussion on -stable about problems with Ryzen manifesting itself in a very uncomfortable way, leading to system lock-ups[5]. There are unofficial patches you can try. I would recommend chiming in there and not here, if relevant to your systems. And yes, the massive number of MFCs that eadler@ is doing make tracking down exact things more tedious than normal, especially when you have sweeping commits like this one[6][7] (which, AFAIK, was acting as a major blocker for several other MFCs and causing general merge problems). However, I commend his efforts; it's a massive undertaking (I would say full-time job). We stable users must accept that we are running stable/11 for a reason -- not only to get fixes faster, but to act a form of "guinea pig" that don't want the risks of HEAD/CURRENT. The more people using stable/11 the better overall feedback devs can get on bugs/issues before making it into the next -RELEASE. This is exactly why, for those of you who have known me over the years, I actually "track" or "follow" commits as they come across. I do this by using the FreshBSD site[8] alongside manual review of svnlite update output. I generally know what files/bits are relevant to my interests. Hope this gives you some things to think about. Good luck! [1]: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=173541#c8 [2]: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=173541#c22 [3]: https://lists.freebsd.org/pipermail/freebsd-stable/2018-February/088396.html [4]: https://lists.freebsd.org/pipermail/freebsd-stable/2018-March/088526.html [5]: https://lists.freebsd.org/pipermail/freebsd-stable/2018-January/thread.html#88174 [6]: http://www.freshbsd.org/commit/freebsd/r330897 [7]: https://svnweb.freebsd.org/base?view=revision&revision=330897 [8]: http://www.freshbsd.org/?branch=RELENG_11&project=freebsd -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |