From owner-freebsd-stable@freebsd.org Wed May 15 16:16:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36F7C1597C3D for ; Wed, 15 May 2019 16:16:41 +0000 (UTC) (envelope-from matt.garber@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8FE6A93D70 for ; Wed, 15 May 2019 16:16:40 +0000 (UTC) (envelope-from matt.garber@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 4E0EA1597C37; Wed, 15 May 2019 16:16:40 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B79B1597C35; Wed, 15 May 2019 16:16:40 +0000 (UTC) (envelope-from matt.garber@gmail.com) Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 83F2493D69; Wed, 15 May 2019 16:16:39 +0000 (UTC) (envelope-from matt.garber@gmail.com) Received: by mail-qt1-x82d.google.com with SMTP id y42so312096qtk.6; Wed, 15 May 2019 09:16:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=BDKCTeHS+/j7SohAx91ExE0xNbRq8raNLmfqada1j6I=; b=Bimpda2lVUDerX6EauT1XIMb0vXrM7QPvOq7KoQS2FJQzQtpPaBvB7ghxizUg99pQS E9Z5cBW3APgaqoy+bDqIEZmu+KfUhg9/NT0/RJCHZh0QE+NEYgHm9cQBGxFiz/SG8nq9 5AeVj0t+TWHBlQpDEv+JpwUkpG65U+D3YSwHEI53a2WHzdQ3MDnm7mMOXq791yFS5esR EoYdNIEgDW4AdgneibmBaRL3Sv5znvRNgq/g35pL5DITXYfKPBS9ChSCNKqWk9fK/cBT KlWcRvaahdaX0kfV2s84k60wMPX2+X4J0kemCYN9Mkpu6652/eYe6SuabgtEKgYZglMK WztA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=BDKCTeHS+/j7SohAx91ExE0xNbRq8raNLmfqada1j6I=; b=kAn7uKNjaZcoypXWRCqSM9Y5cnB8BaLiD0L1TgQ5pMJLdeh2AjWXxoFNrBrQZQw7jw iqsVQAu3wx9BbOZyu2geE8Zqa9vlqJtcZ8UpxfjzPyIJP5K/86GILKVNRF6Uguh92zLH gsY6bU9v22jqTTpJ/WaujQ69FDSe4DXugsE9fYOoKgcVtPRMw8YiqPGxTbyJdsGwm2Tg G/zhZ3KMppiD0GBWvuev2a/uW5zBW4hcHFiIDMRwrK3jABM5yPdOIrme410+dwE/BX7+ Gb/yiCjc92RUJK4O3v1jqA8r59dOMpXVUcLUAFl4w3ZdSa6LZ6aEn941gWyaQSyUl0g6 iu5Q== X-Gm-Message-State: APjAAAWqn+EYQTrBVBILmhrTF3jHJd5/HHGcXQ3Hpcncl9DEtq74ULtT 1NwuYTtJEjeYjNp5SglTFxxDvhFyO84= X-Google-Smtp-Source: APXvYqxr3uLHsV9jjdNSZoMxsLsdndRVznyr6voGLKJjWaNLby18/W19fPIxO2l3G+DKhDaKfE5oBw== X-Received: by 2002:ac8:19ab:: with SMTP id u40mr36483770qtj.229.1557936999079; Wed, 15 May 2019 09:16:39 -0700 (PDT) Received: from [10.100.20.3] ([68.183.62.201]) by smtp.gmail.com with ESMTPSA id 139sm1263618qkm.27.2019.05.15.09.16.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 15 May 2019 09:16:38 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: FreeBSD flood of 8 breakage announcements in 3 mins. From: Matt Garber In-Reply-To: Date: Wed, 15 May 2019 12:16:37 -0400 Cc: FreeBSD Stable ML , "freebsd-hackers@freebsd.org" , FreeBSD Core Team , Alan Somers Content-Transfer-Encoding: quoted-printable Message-Id: <6CE35CEB-C2AB-47B1-AA86-BC9C91B2B8A6@gmail.com> References: <201905151544.x4FFiR0G067138@fire.js.berklix.net> To: Will Andrews , "Julian H. Stacey" X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: 83F2493D69 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2019 16:16:41 -0000 > On May 15, 2019, at 12:12 PM, Will Andrews wrote: >=20 > On Wed, May 15, 2019 at 10:45 AM Julian H. Stacey = wrote: >=20 >> Batching also means some of these vulnerabilities could have been >> fixed earlier & less of a surge of demand on recipient admins time. >>=20 >> An admin can find time to ameliorate 1 bug, not 8 suddenly together. >> Avoidance is called planning ahead. Giving warning of a workload. >> Like an admin plans ahead & announces an outage schedule for planned >> upgrade. >>=20 >> Suddenly dumping 8 on admins causes overload on admin manpower. >> 8 reason for users to approach admin in parallel & say >> "FreeBSD seems riddled, how long will all the sudden unplanned >> outages take ? Should we just dump it ?" >> Dont want negative PR & lack of management. >>=20 >=20 > What admins prefer 8 downtime events instead of 1? >=20 > =E2=80=94Will. Exactly. If batching 8 (or more) individual bugs/issues together into = one release is really causing admin/manpower overload and angst, then = maybe it=E2=80=99s time in your situation to use the binary updates = (which would only be a single `freebsd-update` and reboot, so there = would be no =E2=80=98sudden unplanned outages=E2=80=99) rather than = tracking src and remediating each individual bug at a time. I understand = that might be mutually exclusive with other reasons why you don=E2=80=99t = already use binary updates or prefer to track src for the base system, = but there are always compromises and trade-offs to everything, and = batching seems preferable to any alternatives. You=E2=80=99d seriously want to run reboots across a server fleet every = other day for two weeks if there were 8 separate patches staggered out? = That=E2=80=99s insanity, and is way more of a PR problem from a = =E2=80=98should we just dump it=E2=80=99 perspective. You mention = =E2=80=98announces an outage schedule for planned upgrade=E2=80=99, but = that=E2=80=99s really its own form of internal batching =E2=80=93 it = shouldn=E2=80=99t make any difference if you=E2=80=99re technically = pushing 1 or 8 bug/security fixes during that pre-identified period of = time: all of your other internal processes like maintaining a test group = for detecting regressions, using boot environments (or other storage = features) to allow for rollbacks, etc. all continue to work as intended. Any potential negative PR within your company/organization seems like it = would be related to how else you=E2=80=99re handling the upgrade = process(es), not whether the fixes are batched or not. Whatever other = negative things you can say about them, I don=E2=80=99t hear enterprise = admins begging that Microsoft/Oracle/whoever would dribble out patches = one at a time each week instead of combining them like they do; it seems = like it works just fine for everyone else. =C2=AF\_(=E3=83=84)_/=C2=AF Thanks, =E2=80=94 Matt Garber