From owner-freebsd-hackers@freebsd.org Sat Jul 18 14:10:44 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E8269A3DBE for ; Sat, 18 Jul 2015 14:10:44 +0000 (UTC) (envelope-from holger@layer-acht.org) Received: from alpha.holgerlevsen.de (mail.holgerlevsen.de [62.201.164.66]) by mx1.freebsd.org (Postfix) with ESMTP id 806661BA8; Sat, 18 Jul 2015 14:10:42 +0000 (UTC) (envelope-from holger@layer-acht.org) Received: from localhost (alpha.holgerlevsen.de [62.201.164.66]) by alpha.holgerlevsen.de (Postfix) with ESMTP id 7489ACAD650; Sat, 18 Jul 2015 16:10:34 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at alpha.holgerlevsen.de Received: from alpha.holgerlevsen.de ([62.201.164.66]) by localhost (mail.holgerlevsen.de [62.201.164.66]) (amavisd-new, port 10024) with ESMTP id o8sdsVQHbsG2; Sat, 18 Jul 2015 16:10:24 +0200 (CEST) Received: from matrix.localnet (epsilon.holgerlevsen.de [62.201.164.82]) by alpha.holgerlevsen.de (Postfix) with ESMTP id D49AACAD093; Sat, 18 Jul 2015 16:10:23 +0200 (CEST) From: Holger Levsen To: "freebsd-hackers@freebsd.org" , reproducible-builds@lists.alioth.debian.org Subject: Re: reproducible builds of FreeBSD in a chroot on Linux Date: Sat, 18 Jul 2015 16:09:45 +0200 User-Agent: KMail/1.13.7 (Linux/3.16.0-0.bpo.4-amd64; KDE/4.8.4; x86_64; ; ) References: <201505071122.36037.holger@layer-acht.org> <201506162350.11646.holger@layer-acht.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart4827853.JqjbN4J8qL"; protocol="application/pgp-signature"; micalg=pgp-sha512 Content-Transfer-Encoding: 7bit Message-Id: <201507181609.49815.holger@layer-acht.org> X-Mailman-Approved-At: Sat, 18 Jul 2015 14:45:17 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jul 2015 14:10:44 -0000 --nextPart4827853.JqjbN4J8qL Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, so I made some progress on this: a.) there is a build host running freebsd= =20 10.1 (called freebsd-jenkins.debian.net) now, on which the jenkins user fro= m=20 jenkins.debian.net can login via ssh as jenkins user b.) besides the base=20 system it has "screen git vim sudo denyhosts" installed and c.) the=20 directories /srv/workspace/chroots/ and /srv/reproducible-results have been= =20 created (and are owned by the jenkins user) and d.) /usr/obj/srv is a link = to=20 /srv. With this,=20 http://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/tree/bin/reproduci= ble_freebsd.sh=20 gets as far as=20 https://jenkins.debian.net/view/reproducible/job/reproducible_freebsd/7/con= sole=20 where "stage 2.1: cleaning up the object tree" fails on "make buildworld",= =20 because /srv/workspace/chroots/freebsd- XXXXXXXX.v1adN6Qo/freebsd/lib/libc/tests does not exist. And at this point I'm stuck as to why this happens. Any hint much welcome! (Please note that reproducible_freebsd.sh is just a work-in-progress now an= d=20 there are still some bits from it's source, reproducible_netbsd.sh visible.= =20 This need to be cleaned up, but shouldn't be too confusing know that this i= s=20 clear.) On Mittwoch, 17. Juni 2015, Ed Maste wrote: > > https://wiki.freebsd.org/ReproducibleBuilds claims there are 3 known > > issues (for "make world" AIUI) for HEAD, I would like to build twice and > > verify myself. > I'm interested in fixing the remaining kernel / world issues, with the > kernel being my higher priority. cool! =20 > For the kernel we have the username, hostname, and build timestamp. > The path is included too, but I don't anticipate trying to address it > at first; release builds are done in a consistent location anyhow > (/usr/src). /me nods - that's what we are doing in (reproducible builds for) Debian too= ,=20 the path has to be the same on rebuilds (as it is included in too many buil= d=20 artifacts to deeply.) > These are used only as user-facing strings for the kern.version sysctl > and reported by uname. An example kern.version string: > FreeBSD 10.1-STABLE #28 r280427+86df2de(stable-10): Thu Mar 26 16:07:47 E= DT > 2015 > emaste@feynman:/tank/emaste/obj/tank/emaste/src/git-stable-10/sys/GENERIC >=20 > From a technical perspective they're trivially eliminated. There may > be some 3rd party ports expect the precise format, but probably not > very many (and they should be fixed, anyhow). There's a much larger > social issue in convincing the FreeBSD developer community to accept > their removal, though :-) If any build (of the same sources) results in the exact same bits, the buil= d=20 time becomes meaningless and thus a.) can be dropped or b.) replaced with t= he=20 date of the last modification of the sources - which is meaningful informat= ion=20 again! While this is/was a new thought for most everyone (me included...) in my=20 experience it also has been convincing logic for most everyone. The technic= al=20 details to achieve this are sometimes a bit harder to achieve, but not=20 impossible. (eg they differ whether git, svn or tarballs are the means to g= et=20 access to sources.) In Debian we want 100% bit identical packages (=3D.deb files) as this allow= s us=20 to only require a checksum comparison to see whether two builds created=20 reproducible results. > > https://wiki.freebsd.org/PortsReproducibleBuilds says "Of the 23599 > > packages which were built in both runs, 15164 have the same checksum > > when using the previously mentioned patch, giving 64.25% reproducible > > packages." - I'm also curious to re-confirm this - and set up a test > > bed, which can be triggered regularily and easily. Our jenkins set up > > allows this and I'm interested to do this. >=20 > I'm pleasantly surprised by the ports results -- 64.25% seems quite > good for such a straightforward change. The test there is on the same > host though, and so avoids any non-reproducibility from host/user/path > leaks. ah > > My interest is to help FreeBSD with reproducible builds as I want to see > > reproducible builds become the norm in the free software world and as I > > believe FreeBSD is an important part of this world. And also because I'm > > curious. :) >=20 > Great! Hopefully we can help lend some weight in convincing upstream > projects to accept reproducibility patches (once we get further along > in our ports effort). I'm looking forward to see this happen! ;-) cheers, Holger --nextPart4827853.JqjbN4J8qL Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIVAwUAVapeLQkauFYGmqocAQoKbA/+L//t613SZ3MeYETSLn9fz9jRllyaFLQq USIkeRfahmsGLeRbTqwwosvnAPhUv4xcKAeeC4ofLzo4pb8ySNhpl+JyZ90vMCGj +tiyZCaPwBp9EHiHIup78rsFJZOaBP1CfhnrWOIn1uwQJ7puopQ2pnYfXdydy+uQ grWxY3YzkXTDFLr+biDeaLyg2Qi0MTexvf5udShoEpZdI8AS2+AIVpzwwaCa+2t9 V8VS6x23o4uB65TP+DqKKb+u7Lg43d0/lc+ZdAEHxMFPWSXO2BAVTXfIPW2Nnw1B PbAJJF3jEsZyFVFLrkgKTkQyVH0yK1wFfiQqq8TqWZRca/tL3gi5gcqFMuiD+x9N Cmste0dcUnNv7a4YrcWQB7wGIzlVhQZehkCDoNBdtcJSPOOtCOMBj5Mh2GoSiaMQ dUgWxqy0scWv4tSTCIO8R/J5wa+2hURS2iDc+hMajXSjWgYLlWixMC/uNCDubghA OWVHGoWV1yDAdBkyKMSe2/yysPUP4xmKqCf97fQcyjXHNDbsrsHLabEH30YEn1ML S7mtFBNeSP2Ia6suvgzt9Ugp+7UkwPSYpiVIrRw4Jf+QzZ073BVoto9aq9wc5f8E tt63Yd1jRasmTuB+ZyT0IUO92Sexm9vo7SKl+NAQRWuiUuVxhr99RFxe0vDiudso Mfn7W96ZgOs= =mucW -----END PGP SIGNATURE----- --nextPart4827853.JqjbN4J8qL--