From owner-freebsd-ports@freebsd.org Fri Jun 17 08:47:08 2016 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6279FA788CB for ; Fri, 17 Jun 2016 08:47:08 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 402CC27FD for ; Fri, 17 Jun 2016 08:47:08 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mailman.ysv.freebsd.org (Postfix) id 3F96AA788CA; Fri, 17 Jun 2016 08:47:08 +0000 (UTC) Delivered-To: ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F3D0A788C9 for ; Fri, 17 Jun 2016 08:47:08 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0CDB027F8 for ; Fri, 17 Jun 2016 08:47:07 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-oi0-x22b.google.com with SMTP id p204so107821556oih.3 for ; Fri, 17 Jun 2016 01:47:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=yRrs0iMjjCkHIXLplYi+EF/cjjgq26ZqIeyloeAYW80=; b=VN9Xd2lu5DM7wgcVtzg63nntK8geDb1T4UEjL75mg3pO55+P51VsrhVLeLksplYDtu FCV4PwyXNv29eTw7NwRIyN2+2Tpsq1HbB99t6smo08me8lFnYZ4ReK17FARN3tnbCv+G CNjwDANvQ890QNSWQR5I9VnE2vGelzaIuhPyuXqtLh7HxXnKz6yZMpua6MbmVTbkHCDK 7bgpgvMGhc8FD9Yws19Q23qBkP8s9giYcEqZ92HXRytf8SpNLs3RCsmzhvGknVfhk9r5 M1efwxTgnqE6O8DuXthzmyi9FAsM5xhhxWoFNvwdQmApw8o+oa+hSxsfppxaI3zrBfz/ ui3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=yRrs0iMjjCkHIXLplYi+EF/cjjgq26ZqIeyloeAYW80=; b=fgagNE0OMI6NLUbmOgMSsb4lPwlzIHGvqOTFftVTkofhxnWUrah0rYTW3KfpXHEu3+ Gqmj65qE5ZUCnl2xcg6fi72H4fVQOxmNWhq8R3WlRIQp6CWzzHz8nOSKkgqh0Ec3z9wi FrzxySZ0h8iyAfvOfVNSoYDMM1nJ5pgwWgkVN6BVuRJVq2vCy2kZ1//3mqSu8FQ+EO5y +L5H/BwOrPcBTQHDfJsf8SCu3oQz70Evy3AwaTwEMjiliU5CNZDH56CWPkrQSmR/I3LQ hp8vFHbN/Ilj01MUllajzO3l69ggdv7B6pUMndGPefdWXW2z9Q5BxMCQsJpsYM60LxCl twmg== X-Gm-Message-State: ALyK8tJiK0O+fDs3XFFcGqqY9E9YRGv/SctL0QkcEOhavAR34ASkOHMqtfMCcZ7ESiwCa09Tgq52VJpzosFInNPi X-Received: by 10.202.86.13 with SMTP id k13mr401372oib.3.1466153226967; Fri, 17 Jun 2016 01:47:06 -0700 (PDT) MIME-Version: 1.0 Sender: sobomax@sippysoft.com Received: by 10.157.41.209 with HTTP; Fri, 17 Jun 2016 01:47:06 -0700 (PDT) In-Reply-To: References: From: Maxim Sobolev Date: Fri, 17 Jun 2016 01:47:06 -0700 X-Google-Sender-Auth: YlfqjgkRL5S57B_nQ-RALtFdowk Message-ID: Subject: Re: Some reproducible builds notes To: Ed Maste Cc: ports@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2016 08:47:08 -0000 Hi Ed, I have not got time to look at diffoscope myself yet, but it's definitely in my short-term TODO list. Quick question for you though. Would it work given two tar.bz2 packages or does it need two directories? Just in case it does it as well (although I do not hope for that much), our build system also produces corresponding mtree files so it would be nice if it could also parse those and only extract files that have different checksum. Preferably we would like to diff mtree first, and then run extraction once as extracting files one by one from GB-size .tar.bz2 is painful CPU-wise. We have code to do just that (i.e. inspect mtree and extract files that differ, so if you don't, we can probably contribute that. P.S. For anyone interested in Ed's work, here is his BSDCan talk here live: https://youtu.be/z7pDnBO5wSM?t=5h37m6s -Max On Jun 16, 2016 7:11 AM, "Ed Maste" wrote: > I recently presented on "Reproducible Builds in FreeBSD" at BSDCan. > For anyone unfamiliar with the topic, from > https://reproducible-builds.org/ "Reproducible builds are a set of > software development practices which create a verifiable path from > human readable source code to the binary code used by computers." In > brief, the idea is that building the same binary, software package, > document or other binary artifact twice from the same source produces > identical output. There's good background information, documentation > on making builds reproducible, and links to test results on the > reproducible-builds.org site. > > Many folks have contributed to the reproducible build effort in > FreeBSD src and ports over time -- at least a decade. There are many > practical benefits of reproducible builds (such as bandwidth and > storage savings). However, there's been a growing interest over the > last few years in the broad open source and free software community in > the topic, coming primarily from a software and toolchain integrity > perspective. Over the last few years some Debian folks have been > leading a comprehensive and structured reproducible builds effort. > bapt@ and I attended the first Reproducible Builds Summit in Athens > last year, and I had a talk accepted at BSDCan on it. The BSDCan > schedule page for my talk[1] has a link to the slides[2]. > > I'd like to continue discussing reproducible builds in the FreeBSD > context, but for now just want to capture some data from my talk so > that it's available for interested maintainers of individual ports > who'd like to take a look. I used src r300165 and ports r415464, with > a few patches as described in the talk. > > I've put data from the ports build runs for my talk at [3]. In that > directory nonrepro.1.txt contains the set of packages that built > nonreproducibly (with a patch set the timestamps in pkg's output). > nonrepro.4.txt contains the set of packages that built nonreproducibly > with the patch above, SOURCE_DATE_EPOCH set in the build environment, > a Clang patch[4] to honour SOURCE_DATE_EPOCH, and a change to make GNU > ar default to deterministic archives, since committed as ports > r416639. > > Diffoscope[5] is a tool that attempts to show the differences between > two binary artifacts in a concise and human-readable form. It's > available in ports as sysutils/py-diffoscope and in the > py34-diffoscope package. You can also try it out online[6]. In the > diffoscope/ subdirectory[7] I've put the output for most of the > nonreproducible packages. (Some packages[8] are excluded because of > excessive diffoscope runtime.) > > > [1] http://www.bsdcan.org/2016/schedule/events/714.en.html > [2] > http://www.bsdcan.org/2016/schedule/attachments/375_2016-06-11-BSDCan-2016-Reproducible-Builds.pdf > [3] https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/ > [4] http://reviews.llvm.org/D20791 > [5] https://diffoscope.org/ > [6] https://try.diffoscope.org/ > [7] > https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/diffoscope/ > [8] > https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/excessive-diffoscope-runtime.txt > _______________________________________________ > freebsd-ports@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" > >