From owner-freebsd-arch@FreeBSD.ORG Fri Apr 17 13:30:54 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2542A9A6; Fri, 17 Apr 2015 13:30:54 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF762110; Fri, 17 Apr 2015 13:30:53 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-54-116-245.nwrknj.fios.verizon.net [173.54.116.245]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id EA241B962; Fri, 17 Apr 2015 09:30:52 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Cc: Bryan Drewery Subject: Re: [RFC] SA/EN ABI/Lib flagging for package rebuilding Date: Fri, 17 Apr 2015 09:08:46 -0400 Message-ID: <3952433.yB8VVSoaLQ@ralph.baldwin.cx> User-Agent: KMail/4.14.2 (FreeBSD/10.1-STABLE; KDE/4.14.2; amd64; ; ) In-Reply-To: <5526B8E0.1090403@FreeBSD.org> References: <5526B8E0.1090403@FreeBSD.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 17 Apr 2015 09:30:53 -0400 (EDT) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Apr 2015 13:30:54 -0000 On Thursday, April 09, 2015 12:37:36 PM Bryan Drewery wrote: > Currently for any SA/EN we force rebuild all packages for > pkg.freebsd.org. This is a very time-consuming process at 24 hours for > each set and 10 (non-head) sets to build. > > We really only need to rebuild packages if ABI/KBI is changed (which I > would think would happen almost never) or if a library is updated. The > library being updated needs a rebuild in case of packages linking > against the library statically. There's no quick way to determine what > package may be infected by a static linkage to a library so we just > rebuild them all. > > There are 2 somewhat conflicting goals here. We need head packages to be > usable on all ABI/KBI changes. The simplest here was decided to just > rebuilding head packages always. This is a compromise I am willing to > accept. I had wanted to use __FreeBSD_version for this in the past but > it tends to be missed or bumped gratuitously. > > > My idea to cover both cases so that packages will only rebuild if the > public API or libraries change in the jail or anything else which > affects the resulting binaries. > > I will take a checksum of the following and if anything differs then do > a rebuild: > > /lib /usr/lib /libexec (.so included due to symbol versioning and > detecting added/removed libraries which can change packages) > /usr/include (all public API for the system including kernel API for > modules) > /usr/libdata > > That is all I am aware of for the public API/KPI/ABI/KBI that is > relevant for packages. > > I also realize I need to include some of the binutils files as they may > change how binaries are produced. This is all of 'ar as ld nm objcopy > objdump ranlib readelf strip' in /usr/bin/. Also all of GCC/Clang > binaries in /usr/bin: 'cc clang clang++ clang-cpp clang-tblgen cpp > c++filt CC c++ g++ gcc gnu-ar gnu-ranlib'. > > I don't want to consider all of /bin /usr/bin /sbin and /usr/sbin as it > catches SA that do not really need a rebuild, such as > FreeBSD-SA-15:07.ntp which modified /usr/sbin/ntpd. > > By having a whitelist of files from *bin* I risk not rebuilding in a > rare case where /bin/sh or /bin/cp (random example) had a vulnerability > fixed in them that could change resulting binaries. I consider that to > be very unlikely though. Perhaps for head packages it is more relevant > but I still consider it unlikely other changes will require rebuilding > packages. > > If I forgot other critical pieces from *bin* please list them. > > An alternative would be for there to be a reliable flag in head and > SA/EN noting whether binaries need to be rebuilt. There had been a > technical limitation to bumping __FreeBSD_version for SA as it would > modify all binaries. I think that was fixed. Portmgr fought hard to > always have __FreeBSD_version bumped but that's a lost cause. As I > mentioned I don't trust this flag in head either. I also want __FreeBSD_version always bumped for SA/EN, but secteam@ actively refuses to do this, even after being explicitly told to do so by core@. :( Regardless, I think your approach is more robust than hoping developers remember to bump the version for any API/ABI change. Looking at your proposed list I can only think of a few other binaries that matter, in particular code-generators like lex, yacc, and rpcgen. I don't claim that that list is exhaustive, but I do think it is a class of binaries that were not in your list. -- John Baldwin