From owner-soc-status@freebsd.org Wed Aug 7 06:02:03 2019 Return-Path: Delivered-To: soc-status@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A5DA9BFCEB for ; Wed, 7 Aug 2019 06:02:03 +0000 (UTC) (envelope-from therontarigo@gmail.com) Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 463LVy5Jjfz4HWG for ; Wed, 7 Aug 2019 06:02:02 +0000 (UTC) (envelope-from therontarigo@gmail.com) Received: by mail-pl1-f169.google.com with SMTP id c14so38938558plo.0 for ; Tue, 06 Aug 2019 23:02:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:message-id:date:user-agent :mime-version:content-language; bh=WQF5n5sGxLmK+ZOT8y4czaL36a28Tkl7KsLYz82vQMw=; b=IIasQDO+hVXS8aQEDh/uYpUKsLuXP8sPHZbHFUyC72e1d0ReJAmWPwpH/eyJRylaE7 b0aCqvWukGp2GQoEacZsNlrvbNxAyB3ef7/YFOfLgFaWFRdaOzdOWUzrcJB1ic8LTfJZ 1ftLzFFe1xQKEZn8+b1tKWz0thtHTOovtBpPI7ZopYcO3p7DPW442z/PMIsu+QQqsqwV oA3fvD9khzXiwdW9Eip1y02YUnEa5FE1aY1mGQApAo0dNb+BTNsmb7HrrdE+RonJzfSd HsVdRtMVF0CX+BdMy5FdPeXUvyvB8GxS7+MYIXqQ/uQLJc8nENbZvegmUsprUalbfqOm BuoA== X-Gm-Message-State: APjAAAXoillTVh0lfZjeKMZwiuUVgAx19l/3MO7QgvIHkX2JOvmnVFn+ B99MRNRtcD0i1cGuP6u4A5laCKZ0 X-Google-Smtp-Source: APXvYqxUZYcXpONLGYDXc1qY9OY4NAeTaEv4K7vbxzehuMpckN5i5NxUB5s4d6+B1ksfJU9br9R38g== X-Received: by 2002:a17:90b:95:: with SMTP id bb21mr6896621pjb.8.1565157720930; Tue, 06 Aug 2019 23:02:00 -0700 (PDT) Received: from [192.168.1.23] (c-73-170-47-221.hsd1.ca.comcast.net. [73.170.47.221]) by smtp.gmail.com with ESMTPSA id v8sm75496034pgs.82.2019.08.06.23.01.59 (version=TLS1_3 cipher=AEAD-AES128-GCM-SHA256 bits=128/128); Tue, 06 Aug 2019 23:01:59 -0700 (PDT) From: Theron Tarigo To: soc-status@freebsd.org Subject: GSoC: Separation of Ports Build Process from Local Installation Message-ID: <55ccc3d8-4af8-6c98-60e2-8bb522023866@freebsd.org> Date: Tue, 6 Aug 2019 23:01:58 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 Content-Language: en-US X-Rspamd-Queue-Id: 463LVy5Jjfz4HWG X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of therontarigo@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=therontarigo@gmail.com X-Spamd-Result: default: False [-6.11 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RWL_MAILSPIKE_GOOD(0.00)[169.214.85.209.rep.mailspike.net : 127.0.0.18]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-1.00)[-0.997,0]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[theron@freebsd.org,therontarigo@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[221.47.170.73.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[theron@freebsd.org,therontarigo@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[soc-status@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[169.214.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-3.11)[ip: (-9.68), ipnet: 209.85.128.0/17(-3.39), asn: 15169(-2.45), country: US(-0.05)]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Aug 2019 06:02:03 -0000 Hello all, This project aims to decouple the build process of the ports framework from the local installation.  To work around the assumptions made by ports about dependency file locations, a userspace tool to remap processes' filesystem namespaces is under development: freebsd-user-namespace, provided as port devel/userns. This past week, I have made the following progress: - Modified bsd.port.mk to run pre-* do-* post-* ports targets in a sub-make in userns environment, only when needed (when a port defines those targets itself). - Fixed some non-fatal errors typically seen during dependency installation. - Added support to userns's execve implementation to run ELF executables which use DT_RUNPATH by translating this to an appropriate LD_LIBMAP environment value. - Attributed some failures to a faulty FreeBSD installation on my test system, which is now fixed. Latest summary of port building success: - 5610 ports considered for testing (2000 requested + their dependencies) - 4321 (77%) succeeded - 140 (2%) failed - 1149 (20%) skipped due to failed dependencies - Success rate: 97% (portion succeeded of those which were started, not skipped) Compare to last summary: - 5613 ports tried - 2958 (53%) succeeded - 177 (3%) failed - 2478 (44%) skipped due to failed dependencies - Success rate: 94% Although success rate is similar, overall success is much higher (77% vs. 53%) due to significantly fewer failed dependencies. What I am working on next: - Fix known issues in userns: resolve symbolic link targets, resolve ".." in pathnames. - Eliminate remaining dependency failures. - Compare the packages built by separated-build method to either official packages or ports built by original FreeBSD method.  Any differences should be attributable to built times and hostnames. - Provide more thorough documentation of PORTS_SEPARATED_BUILD, in particular its usage and its limitations, so that users and devs interested in this project can more easily test it for themselves. Project goals and status are kept at https://wiki.freebsd.org/SummerOfCode2019Projects/PortsSeparatedBuild Source of the userspace filesystem namespace tool is shared at https://github.com/therontarigo/freebsd-user-namespace . Changes to ports framework are shared at https://github.com/freebsd/freebsd-ports/compare/master...therontarigo:separated. Theron Tarigo From owner-soc-status@freebsd.org Fri Aug 9 08:49:44 2019 Return-Path: Delivered-To: soc-status@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 18F58C2716 for ; Fri, 9 Aug 2019 08:49:44 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [173.228.5.8]) by mx1.freebsd.org (Postfix) with ESMTP id 464f7V6tSWz4Ft8; Fri, 9 Aug 2019 08:49:42 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mob.bitblocks.com (mob.bitblocks.com [192.168.125.11]) by mail.bitblocks.com (Postfix) with ESMTP id 560151570CE9; Fri, 9 Aug 2019 01:49:21 -0700 (PDT) From: Bakul Shah Message-Id: Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: GSoC: Separation of Ports Build Process from Local Installation Date: Fri, 9 Aug 2019 01:49:20 -0700 In-Reply-To: <55ccc3d8-4af8-6c98-60e2-8bb522023866@freebsd.org> Cc: soc-status@freebsd.org To: Theron Tarigo References: <55ccc3d8-4af8-6c98-60e2-8bb522023866@freebsd.org> X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: 464f7V6tSWz4Ft8 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of bakul@bitblocks.com designates 173.228.5.8 as permitted sender) smtp.mailfrom=bakul@bitblocks.com X-Spamd-Result: default: False [-3.56 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.945,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:173.228.5.8/29]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MV_CASE(0.50)[]; DMARC_NA(0.00)[bitblocks.com]; URI_COUNT_ODD(1.00)[13]; NEURAL_HAM_LONG(-0.99)[-0.991,0]; IP_SCORE(-1.96)[ip: (-6.05), ipnet: 173.228.0.0/20(-3.03), asn: 46375(-0.66), country: US(-0.05)]; NEURAL_HAM_SHORT(-0.97)[-0.970,0]; RCPT_COUNT_TWO(0.00)[2]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:46375, ipnet:173.228.0.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Aug 2019 08:49:44 -0000 Is there any way Theron can have access to a set of build machines? That = would allow him to do some regression testing as well as try out his = scheme on a much larger number of ports. > On Aug 6, 2019, at 11:01 PM, Theron Tarigo wrote: >=20 > Hello all, >=20 > This project aims to decouple the build process of the ports framework = from the local installation. To work around the assumptions made by = ports about dependency file locations, a userspace tool to remap = processes' filesystem namespaces is under development: = freebsd-user-namespace, provided as port devel/userns. >=20 > This past week, I have made the following progress: > - Modified bsd.port.mk to run pre-* do-* post-* ports targets in a = sub-make in userns environment, only when needed (when a port defines = those targets itself). > - Fixed some non-fatal errors typically seen during dependency = installation. > - Added support to userns's execve implementation to run ELF = executables which use DT_RUNPATH by translating this to an appropriate = LD_LIBMAP environment value. > - Attributed some failures to a faulty FreeBSD installation on my test = system, which is now fixed. >=20 > Latest summary of port building success: > - 5610 ports considered for testing (2000 requested + their = dependencies) > - 4321 (77%) succeeded > - 140 (2%) failed > - 1149 (20%) skipped due to failed dependencies > - Success rate: 97% (portion succeeded of those which were started, = not skipped) >=20 > Compare to last summary: > - 5613 ports tried > - 2958 (53%) succeeded > - 177 (3%) failed > - 2478 (44%) skipped due to failed dependencies > - Success rate: 94% >=20 > Although success rate is similar, overall success is much higher (77% = vs. 53%) due to significantly fewer failed dependencies. >=20 > What I am working on next: >=20 > - Fix known issues in userns: resolve symbolic link targets, resolve = ".." in pathnames. > - Eliminate remaining dependency failures. > - Compare the packages built by separated-build method to either = official packages or ports built by original FreeBSD method. Any = differences should be attributable to built times and hostnames. > - Provide more thorough documentation of PORTS_SEPARATED_BUILD, in = particular its usage and its limitations, so that users and devs = interested in this project can more easily test it for themselves. >=20 > Project goals and status are kept at = https://wiki.freebsd.org/SummerOfCode2019Projects/PortsSeparatedBuild = =20= >=20 > Source of the userspace filesystem namespace tool is shared at = https://github.com/therontarigo/freebsd-user-namespace = .=20 > Changes to ports framework are shared at = https://github.com/freebsd/freebsd-ports/compare/master...therontarigo:sep= arated = . >=20 > Theron Tarigo >=20