Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 22 Jul 2019 10:55:49 -0700
From:      Theron Tarigo <theron@freebsd.org>
To:        soc-status@freebsd.org
Subject:   GSoC: Separation of Ports Build Process from Local Installation
Message-ID:  <0016ceb4-b36e-8fa8-6597-8256541a96a8@freebsd.org>

next in thread | raw e-mail | index | archive | help
Hello all,

This project aims to decouple the build process of the ports framework 
from the local installation.  To work around the assumptions made by 
ports about dependency file locations, a userspace tool to remap 
processes' filesystem namespaces is under development: 
freebsd-user-namespace, provided as port devel/userns.

This past week, I have made the following progress:
- Eliminated the need for any special setup to be able to use 
PORTS_SEPARATED_BUILD.  The mode may be enabled by this one variable, at 
the command line or in /etc/make.conf.
- Created a lightweight framework for testing large numbers of ports 
under the separated build mode.  It does the following:
       - Handles FLAVORS.
       - Saves success/failure/skipped status.
       - Skips ports which have failed dependencies.
       - Builds each port in its own pristine environment.
         (This allows concurrent building of unrelated ports)
       - Saves success/failure/skipped status.
       - Skips ports which have failed dependencies.
       - Builds each port in its own pristine environment.
         (This allows parallel (concurrent) port build to work)
- Attempted build of 2000 ports, selected at random, and their dependencies.
- Identified (and applied temporary fixes for) some ports causing 
highest number of skipped dependents.
- Retried the skipped ports.
- Developed a plan of a general fix for the most common cause of 
failure: custom target definitions in port Makefiles running build tools 
(also from ports) without the userns wrapper.

Latest summary of port building success:
- 5613 ports tried
- 2958 (53%) succeeded
- 177 (3%) failed
- 2478 (44%) skipped due to failed dependencies
- Success rate: 94%

What I am working on next:

- Implement the planned solution for running build tools inside userns 
wrapper.
- Compare the packages built by separated-build method to either 
official packages or ports built by original FreeBSD method.  Any 
differences should be attributable to built times and hostnames.
- Debug the ports which fail by segfaut or buserror; this implicates 
userns library.
- Provide more thorough documentation of PORTS_SEPARATED_BUILD, in 
particular its usage and its limitations, so that users and devs 
interested in this project can more easily test it for themselves.

Project goals and status are kept at 
https://wiki.freebsd.org/SummerOfCode2019Projects/PortsSeparatedBuild .

Source of the userspace filesystem namespace tool is shared at 
https://github.com/therontarigo/freebsd-user-namespace .
Changes to ports framework are shared at 
https://github.com/freebsd/freebsd-ports/compare/master...therontarigo:separated.

Theron Tarigo




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0016ceb4-b36e-8fa8-6597-8256541a96a8>