Date: Mon, 11 Oct 2010 11:47:33 +0300 From: Kostik Belousov <kostikbel@gmail.com> To: Erik Cederstrand <erik@cederstrand.dk> Cc: FreeBSD Hackers <hackers@freebsd.org> Subject: Re: Deterministic builds? Message-ID: <20101011084733.GM2392@deviant.kiev.zoral.com.ua> In-Reply-To: <718D8E86-EA2E-4D07-BAFF-5D8D093FD296@cederstrand.dk> References: <718D8E86-EA2E-4D07-BAFF-5D8D093FD296@cederstrand.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
--9CGWabQ8wCOOTGwI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 10, 2010 at 10:51:20PM +0200, Erik Cederstrand wrote: > Hi hackers >=20 > As a followup to the "Timestamps in static libraries" thread which result= ed in a '-D' option to ar(1), I'd like to discuss if it is a worthy goal of= the Project to create deterministic builds. By that I mean for two make bu= ild+install world+kernel+distribution runs, every contained file is bitwise= identical between the two runs. >=20 > Deterministic builds would be useful for me, since I'm creating binary di= ffs against lots of FreeBSD builds, and smaller diffs are good. Also, I'd l= ike to detect which files have changed between two commits. I imagine it wo= uld also be useful for things like IDS and freebsd-update. >=20 > Currently, this does not hold for static libraries (*.a), kernel modules = (*.ko / *.ko.symbols) and the following: >=20 > bthidd > cc1 > cc1obj > cc1plus > clang > clang++ > ctfconvert > freebsd.cf > freebsd.submit.cf > kernel > kernel.symbols > libcrypto.so.6 > libufs.so.5 > loader > pxeboot > sendmail.cf > submit.cf > tblgen > zfsloader >=20 > Most of the libraries can be brought to be identical by using ar -D. Some= record the absolute OBJDIR path to header files, though (libc.a for exampl= e). >=20 > I tried adding 'D' to ARFLAGS in share/mk/sys.mk, but that's only part of= the solution. ARFLAGS are overridden hundreds of places in the source code= , and in some places ARFLAGS isn't even used (or AR for that matter). Is it= worthwhile to go through the whole tree, fixing up these calls to ar? A lo= t of this is in contrib/ code. >=20 > Another option is to add a WITH_DETERMINISTIC_AR knob to the build to com= pile ar with D as default behaviour. This would make the above changes unne= cessary, but is more intrusive. >=20 > A third option is that this is not a priority for the community, or direc= tly unwanted, and that I just post-process my builds myself. >=20 > I don't know what causes the checksum difference in .ko files - there is = no size difference, and no difference according to strings(1). A bsdiff on = the two is typically around 160B. >=20 > .ko.symbols have some unique identifiers or addresses internally. >=20 > kernel, loader, zfsloader and pxeboot have a build date recorded, kernel = also has absolute path to GENERIC. OK for the kernel, I think, although it = would be easier for me if this was just stored in a separate file since bin= ary diffs on large files are expensive. >=20 > clang, clang++ and tblgen store some absolute paths to .cpp files in the = src repo internally, plus unique identifiers. >=20 > freebsd.cf, freebsd.submit.cf, sendmail.cf and submit.cf record the absol= ute OBJDIR path to sendmail >=20 > What do you think? My personal opinion that the feature is nice to have. Unless the changes to get this working are too large, and, more importantly, unless the maintenan= ce cost of having this in good shape is too high, sure we would better have deterministic build results. Also, the deterministic builds require somebody who would monitor the feature, either manually, or by setting some bot that automatically checks it. Otherwise, I suspect, it will degrade. --9CGWabQ8wCOOTGwI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkyyzyQACgkQC3+MBN1Mb4hUwgCcD7slcBfrCl9lb+L3nt9QE0YI NVAAnipFNyVhp8YmQ+C3M1bsK5Ds9C5e =310U -----END PGP SIGNATURE----- --9CGWabQ8wCOOTGwI--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20101011084733.GM2392>