From owner-soc-status@freebsd.org Wed Aug 21 17:14:36 2019 Return-Path: Delivered-To: soc-status@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 60765CE895; Wed, 21 Aug 2019 17:14:36 +0000 (UTC) (envelope-from shivankgarg98@gmail.com) Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46DDmW1NWRz4F28; Wed, 21 Aug 2019 17:14:34 +0000 (UTC) (envelope-from shivankgarg98@gmail.com) Received: by mail-ed1-f68.google.com with SMTP id z51so3756735edz.13; Wed, 21 Aug 2019 10:14:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=5DytjjhbIngv/VSn0G4tbBSoOT5HLY+UGUMDExp7FIY=; b=R04MUQ1UoMflpwhjOFKIwZmyhrH7praFeoRwV3TCjEm7CRL5ICZntGIqqOpWHYOOID hJJMrDpKpvyemIMVhUSU7erOqlfeXJ1PClIQKplkSMEYSfZe+HlNqaPxBA0jWCWg5RCF +2wQ6fxkYV5Ay/swCPRjUxRTUu2cLKGSvOkr2xZDlXEmfvKMkROEaICe1roLwV00tAn5 YxxNe0z0xdP88yeR5A4bKqJlla+M2VMKHSOV7NVsbctJkzYsFYbHDf3II5qXCN2JK0c4 BJJ+pZDy61BjnhD3Nc9VJzzyC0hojaQlQqhyo8JkIitsTdAtYenSB66GzQvYPpdbSHz3 8Yxg== X-Gm-Message-State: APjAAAXbpXm/Rt8yl/j708CdfGmfMPQDmG7MiTcKx3c2pywGY2bQwluc An5Q+CADUqUBE91bjUQ56gHtap2I3GfkrQ== X-Google-Smtp-Source: APXvYqxLjSJ3k6MXPF2SnNhTV/DjXetvm+SlCIXIWmlFyhKw+QqdqvCdb/lFlncU2qpU1kE7G3v0og== X-Received: by 2002:a05:6402:1346:: with SMTP id y6mr37743575edw.27.1566407672286; Wed, 21 Aug 2019 10:14:32 -0700 (PDT) Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com. [209.85.208.52]) by smtp.gmail.com with ESMTPSA id c14sm4259617edb.5.2019.08.21.10.14.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 21 Aug 2019 10:14:32 -0700 (PDT) Received: by mail-ed1-f52.google.com with SMTP id f22so3819461edt.4; Wed, 21 Aug 2019 10:14:31 -0700 (PDT) X-Received: by 2002:a17:906:759:: with SMTP id z25mr31498271ejb.72.1566407671545; Wed, 21 Aug 2019 10:14:31 -0700 (PDT) MIME-Version: 1.0 From: Shivank Garg Date: Wed, 21 Aug 2019 22:43:45 +0530 X-Gmail-Original-Message-ID: Message-ID: Subject: MAC Policy on IP addresses in Jails To: freebsd-hackers@freebsd.org, freebsd-jail@freebsd.org, trustedbsd-discuss@freebsd.org, "Bjoern A. Zeeb" , soc-status@freebsd.org X-Rspamd-Queue-Id: 46DDmW1NWRz4F28 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of shivankgarg98@gmail.com designates 209.85.208.68 as permitted sender) smtp.mailfrom=shivankgarg98@gmail.com X-Spamd-Result: default: False [-5.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-2.12)[ip: (-4.84), ipnet: 209.85.128.0/17(-3.36), asn: 15169(-2.35), country: US(-0.05)]; NEURAL_HAM_SHORT(-0.99)[-0.985,0]; RCVD_IN_DNSWL_NONE(0.00)[68.208.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FORGED_SENDER(0.30)[shivank@freebsd.org,shivankgarg98@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[shivank@freebsd.org,shivankgarg98@gmail.com] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Aug 2019 17:14:36 -0000 Hi Everyone, I am a fourth-year undergraduate student in Department of EE at IIT Kanpur, India. I am an open-source enthusiast and interested in Operating Systems, Computer Networks, and system security. As a part of Google Summer of Code'19, I wrote a loadable kernel MAC module with the TrustedBSD MAC framework to limit the set of IP addresses for a VNET-enabled Jail to choose from. I was mentored by Bjoern A. Zeeb (bz@FreeBSD.org). *About the project:* With the introduction of VNET(9) in FreeBSD, Jails are free to set their IP addresses. However, this privilege may need to be limited by the host as per its need for multiple security reasons. This project uses mac(9) for an access control framework to impose restrictions on FreeBSD jails according to rules defined by the root of the host using sysctl(8). It involves the development of a dynamically loadable kernel module (mac_ipacl) based on The TrustedBSD MAC Framework to implement a security policy for configuring the network stack. This project allows the root of the host to define the policy rules to limit a jail to a set of IP (v4 or v6) addresses and/or subnets for a set of interfaces. Features this new MAC policy module are: - Host can define the list(multiple lists) of IP addresses/subnets for the jail to choose from. - Host can restrict the jail from setting the certain IP addresses or prefixes(subnets). - Host can restrict this privilege to a few networks interfaces. *How to use the module:* I have also wrote a man page for the module. Please refer to the mac_ipacl(4) for using the new MAC module and examples on it. *Test Plan:* Test Scripts integrated with kyua and ATF are included with the module. *Review Link:* This module has been reviewed and revision has been accepted and is ready to land. To check the review: https://reviews.freebsd.org/D20967 *Download Patch/Raw diff from here: * https://reviews.freebsd.org/file/data/udbhpp4gvffsqbqkkekc/PHID-FILE-wun5bhf4qlx6677fdd73/D20967.diff *Wiki and other links:* Please refer to wiki page from more detailed description of the project: *Project FreeBSD Wikipage*: https://wiki.freebsd.org/SummerOfCode2019Projects/MACPolicyIPAddressJail GitHub: https://github.com/shivankgarg98/freebsd/tree/shivank_MACPolicyIPAddressJail/sys/security/mac_ipacl I'll be be very thankful if you can give this module a try and share your valuable experience about it. Please be free to share your ideas and feedback on this module. Regards, Shivank Garg From owner-soc-status@freebsd.org Wed Aug 21 20:56:18 2019 Return-Path: Delivered-To: soc-status@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3B9B6D3F38; Wed, 21 Aug 2019 20:56:18 +0000 (UTC) (envelope-from therontarigo@gmail.com) Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46DKhK19bSz4XVM; Wed, 21 Aug 2019 20:56:16 +0000 (UTC) (envelope-from therontarigo@gmail.com) Received: by mail-pg1-f174.google.com with SMTP id n4so2053192pgv.2; Wed, 21 Aug 2019 13:56:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=1OfV25YDlo/eZgzCzyB35o7pxcP0Xi+wRIMycIxpi8U=; b=nKrMsolIq6dypMlnaCpc9zg22zGsCgH4zN5MmIo4XGS63XClCbyoljW//jBAgzWI7+ BD96lTtmEp50fU3duIA2zZEPEUEj+6COyThdNRGDIZIBHIMTS6dtnhBxLhOZUDjYuRif F5aDfr+ZPoQsEEcmFVJTycE+WKFiO+IEWipX+M0poU71zEo3c8e4OeRZ5N/ZJRHqaEoF +oEs62d3vje42UEGrA942TbMa3WKm5rjk4pumNRgysZOyRwnD1LF+x3rLsk+BDveHZSX jnaLAWaNvSvOQHkkzglTsDZdRffICcHKp8BmRfxgMdRaP7d/ZLXVs7MpUcyRcpSJYgDz AqUg== X-Gm-Message-State: APjAAAXVDH9ndVzf5404G3PhVHwrSD8p9+zmFEyjlE7WUY5cwjFH7xhh 9r6uSZ6LMCOhK1dIeQHcPUH1Jv4b8WI= X-Google-Smtp-Source: APXvYqzeruajs3Smf/ayLNaIXiodBRrfX3qhJgVCx7e1x4Rix2bjhAr5UcUa9/RNnq2EqDlOiTYS7A== X-Received: by 2002:a17:90a:9cf:: with SMTP id 73mr1908612pjo.90.1566420975437; Wed, 21 Aug 2019 13:56:15 -0700 (PDT) Received: from [192.168.1.21] (c-73-170-47-221.hsd1.ca.comcast.net. [73.170.47.221]) by smtp.gmail.com with ESMTPSA id s67sm679345pjb.8.2019.08.21.13.56.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 21 Aug 2019 13:56:15 -0700 (PDT) From: Theron Tarigo Subject: Building ports without needing to installing dependencies into /usr/local To: freebsd-ports@freebsd.org Cc: soc-status@freebsd.org, Bakul Shah Message-ID: Date: Wed, 21 Aug 2019 13:56:14 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 46DKhK19bSz4XVM X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of therontarigo@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=therontarigo@gmail.com X-Spamd-Result: default: False [-6.12 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; RCVD_IN_DNSWL_NONE(0.00)[174.215.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-3.13)[ip: (-9.89), ipnet: 209.85.128.0/17(-3.36), asn: 15169(-2.35), country: US(-0.05)]; FORGED_SENDER(0.30)[theron@freebsd.org,therontarigo@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[221.47.170.73.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[theron@freebsd.org,therontarigo@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Aug 2019 20:56:18 -0000 Hello all, I'd like to share with you the following summary of my progress in adapting the FreeBSD ports framework to gain the capability to build packages (including dependencies) in isolation of the local system configuration in /usr/local, whereas the existing behavior is to require dependencies installed there.  This work is the result of my participation in FreeBSD's Google Summer of Code. Due to the widespread assumption by ports that dependencies are installed at build time into their final deployed locations, this has been a non-trivial undertaking.  Whereas Poudriere remedies this by running the ports framework entirely within Jails, this project attempts to patch all build-time scripts and tooling to access required files from an environment-controlled location.  In most cases this is accomplished by a userspace library to catch and rewrite file paths in POSIX API calls, which has also been developed as part of this project: https://github.com/therontarigo/freebsd-user-namespace/ The project was motivated by the observation that the ports framework as-is (without external tools) fundamentally lacks the capability of completing all build work before installation is performed, which is found in nearly all single-project build systems including FreeBSD base.  This is different from tools such as Poudriere or Synth as it is designed as a core dependency-handling mechanism of the ports build system rather than as an external management or automation tool. The need to touch some core parts of the FreeBSD ports framework means that readiness of the project for adoption may be a long time from now.  That said, I feel confident in declaring it a successful experiment, after testing a limited number of ports under the new scheme (in which ${LOCALBASE} i.e. /usr/local is never touched): Of 5638 ports known (1979 ports selected randomly from ports tree, and their dependencies), 75% were built successfully, since 23% were skipped due to failed dependencies. Of 4230 ports that could be tested (all dependencies were satisfied), 97% succeeded. I would have liked to have tested the entire ports tree, but haven't had access to sufficient machine-time. I've discussed these success rates with my mentor, Bakul Shah, and we agreed that this demonstrates the usefulness of the method. Of course the project is not ready for adoption as the default way of building ports until 100% success here is achieved, but merging of the code on an earlier schedule should be feasible since it is implemented as an option which can remain turned off by default without changing existing behaviors. From a user's perspective, the project currently provides a lighter-weight alternative to one of the core features of Poudriere: A port and all its dependencies are buildable in isolation (like Jail) to create one or more packages to be later installed on one or more systems. It is light-weight because: - Creation of virtual environments for dependencies is done purely in userspace, so Jail and Nullfs are not used. - The feature is usable through the standard Make targets; no top-down separate script or program is used to start the builds. - No preparation other than checking out the (modified, to be eventually merged) ports tree is required. - Direct changes to existing files in the ports framework are kept to a minimum. Other points of potential interest: - Superuser is not required at any part of the process (other than the installation of the resulting packages). It should not be misunderstood as trying to be a potential replacement for Poudriere: It does not perform jailed testing, incremental rebuilds, utilization of ZFS, or many other advanced features, nor are such features planned, nor even appropriate for direct inclusion into /usr/ports/Mk. The scheme currently works only on amd64, due to a small piece of the userspace virtual environment implementation involving machine code manipulation, but this can be extended to support other architectures. For those interested in helping to test, the work is available on Github as follows.  The ports are synchronized to quarterly 2019Q3 as of Aug 15. An example of how it can be tested (no need for superuser): (note that /usr/lib/debug/libexec/ld-elf.so.1.debug (from base-dbg or from installworld) must exist) $ git clone https://github.com/therontarigo/freebsd-ports -b separated --depth 1 $ cd freebsd-ports $ make PORTS_SEPARATED_BUILD=1 PORTSDIR=$PWD PORTBLDBASE=$HOME/ports -C devel/gmake config-recursive package-recursive To be extra sure it is not relying on /usr/local: try instead with LOCALBASE=/usr/nlocal (of course then packages won't install to the default local prefix). Resulting packages will be in $HOME/ports/packages/All/ This modified ports tree is intended to behave exactly as the official one when PORTS_SEPARATED_BUILD=1 is NOT used.  Anywhere that this is violated is a bug that I must fix. Much cleanup and far more thorough testing beyond what I have accomplished in the time of this project are needed before this should be used in production or considered for merging into official ports tree. Your feedback would be much appreciated and will help me to prepare a report on the project's successes, shortcomings, and future directions with respect to the community's needs. Theron From owner-soc-status@freebsd.org Wed Aug 21 21:02:22 2019 Return-Path: Delivered-To: soc-status@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 52BEBD4173; Wed, 21 Aug 2019 21:02:22 +0000 (UTC) (envelope-from therontarigo@gmail.com) Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46DKqK39d0z4XrB; Wed, 21 Aug 2019 21:02:21 +0000 (UTC) (envelope-from therontarigo@gmail.com) Received: by mail-pf1-f172.google.com with SMTP id q139so2228880pfc.13; Wed, 21 Aug 2019 14:02:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:cc:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=1OfV25YDlo/eZgzCzyB35o7pxcP0Xi+wRIMycIxpi8U=; b=Adzzj7/LLYoQ34CCYbXCyrYHIP/ZuDeFD0SQFwYYQyRNKsFBvJH1ewS8vfi8Op+tpV qNzNdfVJQLSdvUaMhzJ2gNliL+3ucjvKZN8HdKict4sLrH225EI+C6z9WG50pq8C0wAn ZVKsq1hJ9ex+PsAnXjsuH9V2VX2GloE1zoHpYCOKOnnPt1qCI5exCcMGIcvKJIrVoKZN hDJB/gXwh+aOceoydsCDss5l5EsZ5aVwXt4PLWyGBiZeedWcdC0x2MmW8PEQPyGukqlC 7d1iC97nO6huFh90UYnkp3XizeNJizdERmgHUtgXouynhLnODfkCKiZKzLeto5IxlYAB i9PA== X-Gm-Message-State: APjAAAXh3kRhi/2W78GMY9W846hFcXnwpjnY6Vrx7BZxQrDMqPlLJdts xpO5xGhBSg1nPqk3r/wmsPk= X-Google-Smtp-Source: APXvYqwI4RyCKDiog6bxWXmwqBeZN/gedu/3hJWuw9lg1LfniSqLzPuxluV57TWFeGOrTRLS6wob2w== X-Received: by 2002:a17:90a:ec12:: with SMTP id l18mr1901048pjy.6.1566421339825; Wed, 21 Aug 2019 14:02:19 -0700 (PDT) Received: from [192.168.1.21] (c-73-170-47-221.hsd1.ca.comcast.net. [73.170.47.221]) by smtp.gmail.com with ESMTPSA id 185sm26555553pfd.125.2019.08.21.14.02.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 21 Aug 2019 14:02:19 -0700 (PDT) To: freebsd-ports@freebsd.org Cc: soc-status@freebsd.org, Bakul Shah From: Theron Tarigo Subject: Building ports without needing to installing dependencies into /usr/local Message-ID: <06401c42-96cd-c558-a9ac-1a6b949354f5@freebsd.org> Date: Wed, 21 Aug 2019 14:02:18 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 46DKqK39d0z4XrB X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of therontarigo@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=therontarigo@gmail.com X-Spamd-Result: default: False [-6.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; RCVD_IN_DNSWL_NONE(0.00)[172.210.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-3.11)[ip: (-9.81), ipnet: 209.85.128.0/17(-3.35), asn: 15169(-2.35), country: US(-0.05)]; FORGED_SENDER(0.30)[theron@freebsd.org,therontarigo@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[221.47.170.73.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[theron@freebsd.org,therontarigo@gmail.com] X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Aug 2019 21:02:22 -0000 Hello all, I'd like to share with you the following summary of my progress in adapting the FreeBSD ports framework to gain the capability to build packages (including dependencies) in isolation of the local system configuration in /usr/local, whereas the existing behavior is to require dependencies installed there.  This work is the result of my participation in FreeBSD's Google Summer of Code. Due to the widespread assumption by ports that dependencies are installed at build time into their final deployed locations, this has been a non-trivial undertaking.  Whereas Poudriere remedies this by running the ports framework entirely within Jails, this project attempts to patch all build-time scripts and tooling to access required files from an environment-controlled location.  In most cases this is accomplished by a userspace library to catch and rewrite file paths in POSIX API calls, which has also been developed as part of this project: https://github.com/therontarigo/freebsd-user-namespace/ The project was motivated by the observation that the ports framework as-is (without external tools) fundamentally lacks the capability of completing all build work before installation is performed, which is found in nearly all single-project build systems including FreeBSD base.  This is different from tools such as Poudriere or Synth as it is designed as a core dependency-handling mechanism of the ports build system rather than as an external management or automation tool. The need to touch some core parts of the FreeBSD ports framework means that readiness of the project for adoption may be a long time from now.  That said, I feel confident in declaring it a successful experiment, after testing a limited number of ports under the new scheme (in which ${LOCALBASE} i.e. /usr/local is never touched): Of 5638 ports known (1979 ports selected randomly from ports tree, and their dependencies), 75% were built successfully, since 23% were skipped due to failed dependencies. Of 4230 ports that could be tested (all dependencies were satisfied), 97% succeeded. I would have liked to have tested the entire ports tree, but haven't had access to sufficient machine-time. I've discussed these success rates with my mentor, Bakul Shah, and we agreed that this demonstrates the usefulness of the method. Of course the project is not ready for adoption as the default way of building ports until 100% success here is achieved, but merging of the code on an earlier schedule should be feasible since it is implemented as an option which can remain turned off by default without changing existing behaviors. From a user's perspective, the project currently provides a lighter-weight alternative to one of the core features of Poudriere: A port and all its dependencies are buildable in isolation (like Jail) to create one or more packages to be later installed on one or more systems. It is light-weight because: - Creation of virtual environments for dependencies is done purely in userspace, so Jail and Nullfs are not used. - The feature is usable through the standard Make targets; no top-down separate script or program is used to start the builds. - No preparation other than checking out the (modified, to be eventually merged) ports tree is required. - Direct changes to existing files in the ports framework are kept to a minimum. Other points of potential interest: - Superuser is not required at any part of the process (other than the installation of the resulting packages). It should not be misunderstood as trying to be a potential replacement for Poudriere: It does not perform jailed testing, incremental rebuilds, utilization of ZFS, or many other advanced features, nor are such features planned, nor even appropriate for direct inclusion into /usr/ports/Mk. The scheme currently works only on amd64, due to a small piece of the userspace virtual environment implementation involving machine code manipulation, but this can be extended to support other architectures. For those interested in helping to test, the work is available on Github as follows.  The ports are synchronized to quarterly 2019Q3 as of Aug 15. An example of how it can be tested (no need for superuser): (note that /usr/lib/debug/libexec/ld-elf.so.1.debug (from base-dbg or from installworld) must exist) $ git clone https://github.com/therontarigo/freebsd-ports -b separated --depth 1 $ cd freebsd-ports $ make PORTS_SEPARATED_BUILD=1 PORTSDIR=$PWD PORTBLDBASE=$HOME/ports -C devel/gmake config-recursive package-recursive To be extra sure it is not relying on /usr/local: try instead with LOCALBASE=/usr/nlocal (of course then packages won't install to the default local prefix). Resulting packages will be in $HOME/ports/packages/All/ This modified ports tree is intended to behave exactly as the official one when PORTS_SEPARATED_BUILD=1 is NOT used.  Anywhere that this is violated is a bug that I must fix. Much cleanup and far more thorough testing beyond what I have accomplished in the time of this project are needed before this should be used in production or considered for merging into official ports tree. Your feedback would be much appreciated and will help me to prepare a report on the project's successes, shortcomings, and future directions with respect to the community's needs. Theron