From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 18 08:38:46 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 066C4DD8 for ; Tue, 18 Mar 2014 08:38:46 +0000 (UTC) Received: from mail-vc0-x22f.google.com (mail-vc0-x22f.google.com [IPv6:2607:f8b0:400c:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BDCBEC06 for ; Tue, 18 Mar 2014 08:38:45 +0000 (UTC) Received: by mail-vc0-f175.google.com with SMTP id lh14so6786726vcb.20 for ; Tue, 18 Mar 2014 01:38:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=wKsXShxQfxlyPii7BTr1jaVLgepuYE3+dCzf0Sz6J7M=; b=Kydejl9LkVUhtbY9vBT23jg0PQUuerMwwWCfo62EdT08OaLw2r5DFp9vryr+Z0Mfs5 ufXqPkpZ6WRoM0v8mH0kOZts+c4OzOFhPAPPMOx/OfnAd8bPhUs12YYgRXs3IzE33sII mK2DlrVrfCZKrNfK0r+sxH8IM94oPhRdHRvDKFaqWTkm4gEQzjslqROegQERJhOslrkM g9zWfjdUrbi9utwn1xHQELSNk7fJEmuUPOTG7K+8G8W0Fl62/pUj9RaayB0XhjVjDdJ7 Zo8J1Xa4M1pF6jkHDVMJKQ3zR3vVQvNdfvwtl91wVlnK9vaU9Ig1idfv66sGJewsZzsh 2mUg== MIME-Version: 1.0 X-Received: by 10.221.29.196 with SMTP id rz4mr23880258vcb.8.1395131924945; Tue, 18 Mar 2014 01:38:44 -0700 (PDT) Received: by 10.58.47.8 with HTTP; Tue, 18 Mar 2014 01:38:44 -0700 (PDT) Date: Tue, 18 Mar 2014 09:38:44 +0100 Message-ID: Subject: GSoC: Deterministic builds From: Adria Garriga To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Mailman-Approved-At: Tue, 18 Mar 2014 11:31:10 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2014 08:38:46 -0000 Hello! I'm Adria (rhaps0dy in IRC). I'd love to participate in the GSoC as a FreeBSD student. Here's the project I have in mind, could you please give me some feedback? Deterministic port builds ========================= Scripts to automatically create a jail or bhyve guest where you can build ports deterministically. On any amd64 host, given an updated ports tree, same CFLAGS, and this jail, the result of the build is byte-by-byte equal. Progress towards this has already been made by several free software development teams, most notably the Debian and Bitcoin teams. Debian have several deterministic build scripts and documentation on what hinders the accomplishment of this goal. It's mainly the time, the build path and the locale, so with libfaketime or an appropriate replacement, canonical build paths inside the jail and using the same locale much would be solved. The Bitcoin team has produced Gitian, (https://gitian.org/), a set of scripts that create an Ubuntu virtual machine on qemu and run deterministic builds on it. On the (few) tests I've run, clang{,++} produce deterministic object files. Seeing that this much of work is already done, I can say that this is not enough for two months of intensive work. That's the reason I've come up with these two extensions. Extension one: -------------- Extend the above procedure to more architectures, ideally all architectures supported by FreeBSD (i386, amd64, IA64, sparc64, powerpc and pc98 if I'm not mistaken). This would require more testing work and almost certainly the use of virtual machines instead of cross-compiling environments, further complicated by the fact that I only have access to i386 and amd64 hardware. Extension two: -------------- Create necessary programs for a volunteer-driven compilation farm. This could help compile 'custom' ports for users with slow machines, or redundantly compile and verify the official binary-distributed packages. It would be either p2p or centralised on a server. I'm aware that this is a huge task, too big for a single month (half a year even). The goal of this extension would be to produce software for a low number (at most ten) of hosts assumed to be always up to share compilation resources and redundantly check for differences (draft at http://cf.bagelbox.org/freebsd-farm.png). To further reduce complexity and make it manageable, it will only compile one reasonably big port (for example Firefox) but will be easily extended. I'm specially interested in the second extension, really. The first one is a bit of unpleasant but necessary hard work. What do you think? Thank you very much! Adria