Date: Fri, 17 Jun 2016 01:47:06 -0700 From: Maxim Sobolev <sobomax@freebsd.org> To: Ed Maste <emaste@freebsd.org> Cc: ports@freebsd.org Subject: Re: Some reproducible builds notes Message-ID: <CAH7qZftHcs9gP=htQtJCGqPNJhB6tnVi6bLUAxDiakteWvX_Nw@mail.gmail.com> In-Reply-To: <CAPyFy2DWrGBbk6hFdftixjHyivEGO2dVSCjsbXKNOj3YceMN8A@mail.gmail.com> References: <CAPyFy2DWrGBbk6hFdftixjHyivEGO2dVSCjsbXKNOj3YceMN8A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Ed, I have not got time to look at diffoscope myself yet, but it's definitely in my short-term TODO list. Quick question for you though. Would it work given two tar.bz2 packages or does it need two directories? Just in case it does it as well (although I do not hope for that much), our build system also produces corresponding mtree files so it would be nice if it could also parse those and only extract files that have different checksum. Preferably we would like to diff mtree first, and then run extraction once as extracting files one by one from GB-size .tar.bz2 is painful CPU-wise. We have code to do just that (i.e. inspect mtree and extract files that differ, so if you don't, we can probably contribute that. P.S. For anyone interested in Ed's work, here is his BSDCan talk here live: https://youtu.be/z7pDnBO5wSM?t=5h37m6s -Max On Jun 16, 2016 7:11 AM, "Ed Maste" <emaste@freebsd.org> wrote: > I recently presented on "Reproducible Builds in FreeBSD" at BSDCan. > For anyone unfamiliar with the topic, from > https://reproducible-builds.org/ "Reproducible builds are a set of > software development practices which create a verifiable path from > human readable source code to the binary code used by computers." In > brief, the idea is that building the same binary, software package, > document or other binary artifact twice from the same source produces > identical output. There's good background information, documentation > on making builds reproducible, and links to test results on the > reproducible-builds.org site. > > Many folks have contributed to the reproducible build effort in > FreeBSD src and ports over time -- at least a decade. There are many > practical benefits of reproducible builds (such as bandwidth and > storage savings). However, there's been a growing interest over the > last few years in the broad open source and free software community in > the topic, coming primarily from a software and toolchain integrity > perspective. Over the last few years some Debian folks have been > leading a comprehensive and structured reproducible builds effort. > bapt@ and I attended the first Reproducible Builds Summit in Athens > last year, and I had a talk accepted at BSDCan on it. The BSDCan > schedule page for my talk[1] has a link to the slides[2]. > > I'd like to continue discussing reproducible builds in the FreeBSD > context, but for now just want to capture some data from my talk so > that it's available for interested maintainers of individual ports > who'd like to take a look. I used src r300165 and ports r415464, with > a few patches as described in the talk. > > I've put data from the ports build runs for my talk at [3]. In that > directory nonrepro.1.txt contains the set of packages that built > nonreproducibly (with a patch set the timestamps in pkg's output). > nonrepro.4.txt contains the set of packages that built nonreproducibly > with the patch above, SOURCE_DATE_EPOCH set in the build environment, > a Clang patch[4] to honour SOURCE_DATE_EPOCH, and a change to make GNU > ar default to deterministic archives, since committed as ports > r416639. > > Diffoscope[5] is a tool that attempts to show the differences between > two binary artifacts in a concise and human-readable form. It's > available in ports as sysutils/py-diffoscope and in the > py34-diffoscope package. You can also try it out online[6]. In the > diffoscope/ subdirectory[7] I've put the output for most of the > nonreproducible packages. (Some packages[8] are excluded because of > excessive diffoscope runtime.) > > > [1] http://www.bsdcan.org/2016/schedule/events/714.en.html > [2] > http://www.bsdcan.org/2016/schedule/attachments/375_2016-06-11-BSDCan-2016-Reproducible-Builds.pdf > [3] https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/ > [4] http://reviews.llvm.org/D20791 > [5] https://diffoscope.org/ > [6] https://try.diffoscope.org/ > [7] > https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/diffoscope/ > [8] > https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/excessive-diffoscope-runtime.txt > _______________________________________________ > freebsd-ports@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAH7qZftHcs9gP=htQtJCGqPNJhB6tnVi6bLUAxDiakteWvX_Nw>