From owner-freebsd-ports@freebsd.org Mon Jul 20 13:24:18 2020 Return-Path: Delivered-To: freebsd-ports@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B491435E997 for ; Mon, 20 Jul 2020 13:24:18 +0000 (UTC) (envelope-from svysh.fbsd@gmail.com) Received: from mail-lj1-x232.google.com (mail-lj1-x232.google.com [IPv6:2a00:1450:4864:20::232]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9Mrd5cdqz428D for ; Mon, 20 Jul 2020 13:24:17 +0000 (UTC) (envelope-from svysh.fbsd@gmail.com) Received: by mail-lj1-x232.google.com with SMTP id r19so20131560ljn.12 for ; Mon, 20 Jul 2020 06:24:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:subject:to:message-id:date:user-agent:mime-version :content-transfer-encoding:content-language; bh=8E3UoQxsWbTQFlM4Pdie2x7/MxI79bz/os18ONJTgIs=; b=AWWATsJ15XjMkLyNpOSUMnpeVb61s1iXdHAoymDTR7zQcTJdZib0ASRu7V3uJhl8mO FZy0rG5TNzfc3q2MwZVt4fS7fVZTu3DSbodSwei5ECv4EyMry5Ug3DVoM52aUiXw53Cw GL63+m8Pnl+swXT6+8L82nfX2xKJYIqRukxBmGqPFFCViKACU7sphe8p+vSK57oz0d9X Jkz6UM/BM4w/rUzuK8NeYPEIHd+/kZiC6wMOyB7b7rPXV9NwvDjah5ExkOSDMY80ee5i dHCgnynEi7rM5liQYF2gQPUFP4fP0Eql1DL9PAgysXFyU2EWx+UKsgj0miB345YmNssU D1/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=8E3UoQxsWbTQFlM4Pdie2x7/MxI79bz/os18ONJTgIs=; b=eCcPi6dAr6sR55bNWooWczx9ALHArUT2BFRR41v6KrwZK7vqdudKyaHmlHswiqvyrG Pnz2lwp1C+cM3RrEDUQvOeX3DZN7sK+uPsLsWPumuUxCl9270ovBWu6PeuC4mmJ8nBvX qIiEoemrYUn7oz4lt4O2USpdAuCIFYgZPx5pqGWhNMVWJYGaP3hXNQuqS1tvPhdZpqS9 KJJ1PNafGmqwr2t+gd8qP7ExoLvEUaBt7NJhlLIzl7OgrWYIRDbjPVBS1PvINf7c6REz KqReOhvsLBzcf+paBGdY83fktjYpMB3+efdkex/ZsgJd+MxLkChPYliZpAI+PpsNVawP GkBw== X-Gm-Message-State: AOAM532DeuEMEZJHyBCjbeHK1qebtVzIcGJcyynLOvU/0OX5VINNHIm4 3eg0h824cgYeY6KYViZARfEeEas3AsQ= X-Google-Smtp-Source: ABdhPJybSD2lAVKyl9eqZW7iIcgIvSo71b6lhTHTSwp7CLN0q8bAw6R4ARKPu8C63iSDAdNUVowHDw== X-Received: by 2002:a2e:160f:: with SMTP id w15mr10931567ljd.28.1595251453078; Mon, 20 Jul 2020 06:24:13 -0700 (PDT) Received: from [192.168.169.21] ([91.231.113.113]) by smtp.gmail.com with ESMTPSA id t6sm3061495lfc.62.2020.07.20.06.24.12 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 20 Jul 2020 06:24:12 -0700 (PDT) From: Sergei Vyshenski Subject: Autogenerated versus handmade distributions from Github (possible new section to PHB) To: FreeBSD Ports Message-ID: <0202562f-8035-03b7-d308-e2a2faf30b3e@gmail.com> Date: Mon, 20 Jul 2020 16:24:11 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4B9Mrd5cdqz428D X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=AWWATsJ1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of svyshfbsd@gmail.com designates 2a00:1450:4864:20::232 as permitted sender) smtp.mailfrom=svyshfbsd@gmail.com X-Spamd-Result: default: False [-2.84 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.02)[-1.023]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.991]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ports@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_SHORT(0.18)[0.176]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::232:from]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 13:24:18 -0000 Hi, Please comment. Regards, Sergei ============================= See PDF version: https://drive.google.com/file/d/1ly9ylKJehzcqXxy3NhqN3Pywy0iqZwn9/view?usp=sharing The text below is being offered for discussion as a candidate for a new section of PHB, which could be placed after the existing section"5.17. How to Use USE_GITHUB with Git Submodules?"  Writing of this text has been advised by Adam Weinberger (adamw@): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221634 and is based on posts by Yasuhiro KIMURA (yasu at utahime.org): https://lists.freebsd.org/pipermail/freebsd-ports/2017-December/111758.html Mathieu Arnold (mat@): https://lists.freebsd.org/pipermail/freebsd-ports/2017-December/111759.html and Sergei Vyshenski (svysh.fbsd at gmail.com): https://lists.freebsd.org/pipermail/freebsd-ports/2017-December/111753.html https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221634 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247459 (this one has status: open) === A proposed new section of PHB starts here === *Autogenerated versus handmade distributions from Github* *A-distributions* USE_GITHUB triggers a nice feature of automatic distribution generating, which is provided by the Github infrastructure. Upon user request it automatically generates distribution archive (aka git archive or source code archive or tarball) from the master branch of a specified project. It also accepts wanted format (tgz, zip etc) and wanted tag (commit number) of the master branch. Author of the project could label some commit numbers with version numbers, thus performing version release. Version numbers are published on a special web pages like: https://github.com/openxpki/openxpki/tags https://github.com/openxpki/openxpki/releases Let us call such automatically (dynamically) generated distributions "A-distributions". A-distributions do not exist on the Github portal. They are generated upon user request. This request can come in two ways: 1)Click on an URL (shown on the Web page) like this: https://github.com/openxpki/openxpki/archive/v3.6.1.tar.gz Note a keyword "archive" in this type of URL. 2)Through USE_GITHUB the same file is accessible like this: https://codeload.github.com/openxpki/openxpki/tar.gz/v3.6.1?dummy=/openxpki-openxpki-v3.6.1_GH0.tar.gz Note a keyword "codeload" in this type of URL. Both these URLs are virtual ones in a sense that the referenced file actually does not exist, and is generated only upon user request. *H-distributions* Sometime getting of an A-distribution could place a user in a difficult position. For illustration consider two such cases: 1) cyclic dependencies and 2) excessive complexity. 1)Project P is called to have a cyclic dependency, if project P depends on project P1, which in turn depends on project P. In the simplest case of cyclic dependency, project P can depend on itself (autodependency). Imagine that project P is dedicated to building a library P_L, which is meant for text processing. Usually author of such a project P illustrates use of the library P_L by employing this library while building documentation for the same project P. Straightforward porting of the project P for FreeBSD will bring ports infrastructure on halt with complaint about not installed library P_L, which is needed to build documentation, which in turn is part of the project P. 2)Project P is called to have excessive complexity, if a tiny (and not essential) part of it requires huge amounts of dependencies. Imagine that a project uses a high-level workflow language to describe logic of some complex business process. This project provides docs and examples, where workflow language is used as a source for graphical visualization. Then building this docs and examples would require tons of other packages and would spend much more time that the all the rest of the project P. Autodependency mentioned on points 1 above could be avoided in two ways: * *Porter's way:*Porter can split a port for the project P into two ports: P_library and P_doc, where port P_doc depends on port P_library. * *Author's way:*Author of the project P could prepare a documentation in a form, which does not require further use of library P_L. To achieve this, he makes use of the library P_L, which is available on his personal host anyway. Then author by hand prepares a distribution, which contains sources for building the library P_L, and pre-built documentation in a ready-to-use form. Let us call such distributions "H-distributions". Example of project with autodependency is a Github project libexpat/libexpat, which is ported to FreeBSD as port textproc/expat2 with use of H-distribution. Excessive complexity mentioned in point 2 above could be avoided in the Author's way as follows. Author pre-builds docs and examples with fancy pictures himself. Then author by hand prepares an H-distribution, which contains pre-built docs and examples, along sources for the rest of the project (that is no sources for docs and examples). Example of project with excessive complexity is a Github project tdiary/tdiary-core, which is ported to FreeBSD as port www/tdiary with use of H-distribution. H-distributions are not available via USE_GITHUB. They could only be downloaded from the pages where author has put them, like: https://github.com/tdiary/tdiary-core/tags https://github.com/tdiary/tdiary-core/releases using clickable URL like: https://github.com/tdiary/tdiary-core/releases/download/v5.1.2/tdiary-full-v5.1.2.tar.gz Note keywords "releases" or "download" in this type of URL. URLs for H-distributions are real ones in a sense that the referenced file actually exists, and has been generated and uploaded by author's hand. To fetch the H-distribution from port, porter should remove USE_GITHUB, and add explicit MASTER_SITES, and maybe DISTNAME or DISTPREFIX. *Examples of ports using H-tarballs:* textproc/expat2 https://github.com/libexpat/libexpat www/tdiary https://github.com/tdiary/tdiary-core net/uriparser https://github.com/uriparser/uriparser (will use H-tarball, if PR#247459 is committed) *A- and H-distributions mixed* Ultimate example of simultaneous use of both A- and H-distributions can be seen on page https://github.com/libexpat/libexpat/releases/tag/R_2_2_8 which provides URL for H-distribution: https://github.com/libexpat/libexpat/releases/download/R_2_2_8/expat-2.2.8.tar.gz and provides URL for A-distribution (it is called "Source code" here): https://github.com/libexpat/libexpat/archive/R_2_2_8.tar.gz Please compare both types of distribution and see how very different they are. This H-distribution is employed for port /textproc/expat2, while this A-distribution is left for a special connoisseurs. *A- and H-distributions compared* You should always use released H-distributions, if they are available, authors go through much efforts to make them, so that other people can build their software more easily. The git archive (A-distribution) you get from USE_GITHUB should only be used if nothing else is available. === A proposed new section of PHB ends here ===