From nobody Thu Jan 13 23:13:04 2022
X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A61981959AC8;
	Thu, 13 Jan 2022 23:13:16 +0000 (UTC)
	(envelope-from kevans@freebsd.org)
Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "smtp.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4JZgG44GHKz3JLt;
	Thu, 13 Jan 2022 23:13:16 +0000 (UTC)
	(envelope-from kevans@freebsd.org)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim;
	t=1642115596;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=jsE86FyAoG6v4urrPu0WtWgnfS+YjPqx+xogIBYQKwk=;
	b=Dh+2FVojoEIoAuZNzrEZaNd2+KJTjVqLkVvCxInwBNl1Al1TmRbIqCaIGsU2/x2M4JDNfd
	gDzpGK9cc9+svPImKmWRaQ1l76go4CPs5Sgyd6mpcO9gm1s7fIHAsrMx915Swn157baUHs
	jiOsZCycJSNitMyXVbp4C0LJjjO+vNhs9ExTV4EQiu0A4v8RlrIkkDSqI+JkYeBisRd26y
	G38rj2rQ28PqeOfsr/5FwfsBUmTiuTuHaR1JNilvkc1cNaLBpyQUid7RRd3c0a/bBHm74m
	jH9QYHqLZPHHh54tPQ91EnQ6jGLIZpdhB9ZqUAw0H6LS1EpRMByh9kM1vudTbg==
Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	(Authenticated sender: kevans)
	by smtp.freebsd.org (Postfix) with ESMTPSA id 6E4A3AA7;
	Thu, 13 Jan 2022 23:13:16 +0000 (UTC)
	(envelope-from kevans@freebsd.org)
Received: by mail-qk1-f171.google.com with SMTP id c190so7571124qkg.9;
        Thu, 13 Jan 2022 15:13:16 -0800 (PST)
X-Gm-Message-State: AOAM5325E8jqnKzb4lrmAoUvo1mou96w5MQIEFMO9p95z8wKgKRT3X6F
	K89+jmzYwt7SaOCsS++4lKZoiM8+TJhYQbfTc6A=
X-Google-Smtp-Source: ABdhPJz9aJHtJYKRLpgXciPDy9gEttxEJqjhjsGgLil2Ustj0qvauxatmC+cxiyW2saZUBPwQYU7tRBPF9FnMay9x10=
X-Received: by 2002:a05:620a:f0e:: with SMTP id v14mr4832992qkl.365.1642115595962;
 Thu, 13 Jan 2022 15:13:15 -0800 (PST)
List-Id: Commit messages for the main branch of the src repository <dev-commits-src-main.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main
List-Help: <mailto:dev-commits-src-main+help@freebsd.org>
List-Post: <mailto:dev-commits-src-main@freebsd.org>
List-Subscribe: <mailto:dev-commits-src-main+subscribe@freebsd.org>
List-Unsubscribe: <mailto:dev-commits-src-main+unsubscribe@freebsd.org>
Sender: owner-dev-commits-src-main@freebsd.org
X-BeenThere: dev-commits-src-main@freebsd.org
MIME-Version: 1.0
References: <202201132138.20DLccSU044307@gitrepo.freebsd.org>
In-Reply-To: <202201132138.20DLccSU044307@gitrepo.freebsd.org>
From: Kyle Evans <kevans@freebsd.org>
Date: Thu, 13 Jan 2022 17:13:04 -0600
X-Gmail-Original-Message-ID: <CACNAnaF+pWor8tsih95tdsgWmcpskqz2ajt5==_Y74CsVy3VXw@mail.gmail.com>
Message-ID: <CACNAnaF+pWor8tsih95tdsgWmcpskqz2ajt5==_Y74CsVy3VXw@mail.gmail.com>
Subject: Re: git: 82bfeeff10da - main - tools/portconflicts/port_conflicts_check.lua:
 detect port conflicts
To: =?UTF-8?B?U3RlZmFuIEXDn2Vy?= <se@freebsd.org>
Cc: src-committers <src-committers@freebsd.org>, 
	"<dev-commits-src-all@freebsd.org>" <dev-commits-src-all@freebsd.org>, dev-commits-src-main@freebsd.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1642115596;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=jsE86FyAoG6v4urrPu0WtWgnfS+YjPqx+xogIBYQKwk=;
	b=UqmEVswAIF/JPlqhER6V9y114/g3OwlVJvaMACXwi/gTGvmqRNGSXfUFdUBEnth1zgff5b
	zgl4lc9cx1MRHNMF36KH+XyTi8zskcMrrQsNgtAw1ycJiICh/frrQn1iSFHMLDYvg0vVmL
	+TEu/z26ftbtFtD3EF8X/4OApvzdDyjwxnjV45cIPGc1S+1YslRSKw+9yMuN5SOYQu6ChP
	0lc8pgXkrtCyf+oG4coVb1dRDPKbOC0R0Myqg9wqcMkN0YYK5p5SzkVTVnCX2+5pF8ujIa
	VdtIB0K/3Y6Bo3GAvqXbCesLy1aDZ9tqfsLKbMgbqsJLEZ65bOzGVnkuNNjcQw==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642115596; a=rsa-sha256; cv=none;
	b=E+QDIGvpvY8QJvR1FHrzD/24zxh0r1D08unWwvyJXEdorgRYz6gbnqshDWa++LlgoKwiPy
	Xs9uZ/fAYmt4DJdTF2pBCKOdP0OPgN52m8o5exyby+V/n6a3WaeH9yNXirEHtDHEG5Ells
	nM9gNRlevzwu2ubaODJDvZDxw8lor/xMghv3Mtb9yD7nZoo9Zg9lx+ilYBkYwsdWbKbrlU
	FY9nU0a52s/qsZ8m5p6c6hN7vLYlwlWya1i1IbeT/1Z7kzLTp4LYNx93INv9R97vCQIHVV
	AJWfW5GRlzUlQMOxFdJ1WG4sMZ4twS/MBwMZUSZ3bVfRJqLQq+lSnq3PLe+tmg==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
X-ThisMailContainsUnwantedMimeParts: N

 and runOn Thu, Jan 13, 2022 at 3:38 PM Stefan E=C3=9Fer <se@freebsd.org> w=
rote:
>
> The branch main has been updated by se:
>
> URL: https://cgit.FreeBSD.org/src/commit/?id=3D82bfeeff10da860b1ed9f03a01=
f3282b42b991be
>
> commit 82bfeeff10da860b1ed9f03a01f3282b42b991be
> Author:     Stefan E=C3=9Fer <se@FreeBSD.org>
> AuthorDate: 2022-01-13 21:36:09 +0000
> Commit:     Stefan E=C3=9Fer <se@FreeBSD.org>
> CommitDate: 2022-01-13 21:36:09 +0000
>
>     tools/portconflicts/port_conflicts_check.lua: detect port conflicts
>
>     This tool can be used to detect install conflicts (ports/packages tha=
t
>     conflict with each other because of identically named files).
>
>     MFC after:      3 days
>

Hi,

Good to see more use of flua coming into the tree. :-) I recommend
grabbing devel/lua-luacheck and running that over the file -- it's
mostly pointing out complaints about things that should be declared
`local`, but also notes some style-type issues and a write-only
variable.

> ---
>  tools/tools/portconflicts/port_conflicts_check.lua | 184 +++++++++++++++=
++++++
>  1 file changed, 184 insertions(+)
>
> diff --git a/tools/tools/portconflicts/port_conflicts_check.lua b/tools/t=
ools/portconflicts/port_conflicts_check.lua
> new file mode 100755
> index 000000000000..ced7cc619f4d
> --- /dev/null
> +++ b/tools/tools/portconflicts/port_conflicts_check.lua
> @@ -0,0 +1,184 @@
> +#!/usr/libexec/flua
> +
> +--[[
> +SPDX-License-Identifier: BSD-2-Clause-FreeBSD
> +
> +Copyright (c) 2022 Stefan Esser <se@FreeBSD.org>
> +
> +Generate a list of existing and required CONFLICTS_INSTALL lines
> +for all ports (limited to ports for which official packages are
> +provided).
> +
> +This script depends on the ports-mgmt/pkg-provides port for the list
> +of files installed by all pre-built packages for the architecture
> +the script is run on.
> +
> +The script generates a list of ports by running "pkg provides ." and
> +a mapping from package base name to origin via "pkg rquery '%n %o'".
> +
> +The existing CONFLICTS and CONFLICTS_INSTALL definitions are fetched
> +by "make -C $origin -V CONFLICTS -V CONFLICTS_INSTALL". This list is
> +only representative for the options configured for each port (i.e.
> +if non-default options have been selected and registered, these may
> +lead to a non-default list of conflicts).
> +
> +The script detects files used by more than one port, than lists by
> +origin the existing definition and the list of package base names
> +that have been detected to cause install conflicts followed by the
> +list of duplicate files separated by a hash character "#".
> +
> +This script uses the "hidden" LUA interpreter in the FreeBSD base
> +systems and does not need any port except "pkg-provides" to be run.
> +
> +The run-time on my system checking the ~32000 packages available
> +for -CURRENT on amd64 is 150 seconds.
> +--]]
> +
> +require "lfs"
> +
> +local index_file =3D "/usr/ports/INDEX-14"
> +
> +local function read_index ()
> +   local ORIGIN =3D {}
> +
> +   local pipe =3D io.popen("pkg rquery '%n %o'")
> +   for line in pipe:lines() do
> +      local pkgbase, origin =3D string.match(line, "(%S+) (%S+)")
> +      ORIGIN[pkgbase] =3D origin
> +   end
> +   pipe:close()
> +   return ORIGIN
> +end
> +
> +local function read_files()
> +   local FILES_TABLE =3D {}
> +
> +   local pkgbase, version
> +   local pipe =3D io.popen("pkg provides .")
> +   for line in pipe:lines() do
> +      local label =3D string.sub(line, 1, 10)
> +      if label =3D=3D "Name    : " then
> +        name =3D string.sub(line, 11)
> +        pkgbase, version =3D string.match(name, "(.*)-([^-]*)")
> +      elseif label =3D=3D "          " or label =3D=3D "Filename: " then
> +        local file =3D string.sub(line, 11)
> +        if file:sub(1, 10) =3D=3D "usr/local/" then
> +           file =3D file:sub(11)
> +        else
> +           file =3D "/" .. file
> +        end
> +        local t =3D FILES_TABLE[file] or {}
> +        t[#t + 1] =3D pkgbase
> +        FILES_TABLE[file] =3D t
> +      end
> +   end
> +   pipe:close()
> +   return FILES_TABLE
> +end
> +
> +local PKG_PAIRS =3D {}
> +
> +for file, pkgbases in pairs(read_files()) do
> +   if #pkgbases > 1 then
> +      for i =3D 1, #pkgbases -1 do
> +        local pkg_i =3D pkgbases[i]
> +        for j =3D i + 1, #pkgbases do
> +           local pkg_j =3D pkgbases[j]
> +           if pkg_i ~=3D pkg_j then
> +              p1 =3D PKG_PAIRS[pkg_i] or {}
> +              p2 =3D p1[pkg_j] or {}
> +              p2[#p2 + 1] =3D file
> +              p1[pkg_j] =3D p2
> +              PKG_PAIRS[pkg_i] =3D p1
> +           end
> +        end
> +      end
> +   end
> +end
> +
> +local CONFLICT_PKGS =3D {}
> +local CONFLICT_FILES =3D {}
> +
> +for pkg_i, p1 in pairs(PKG_PAIRS) do
> +   for pkg_j, p2 in pairs(p1) do
> +      CONFLICT_PKGS[pkg_i] =3D CONFLICT_PKGS[pkg_i] or {}
> +      CONFLICT_PKGS[pkg_j] =3D CONFLICT_PKGS[pkg_j] or {}
> +      CONFLICT_FILES[pkg_i] =3D CONFLICT_FILES[pkg_i] or {}
> +      CONFLICT_FILES[pkg_j] =3D CONFLICT_FILES[pkg_j] or {}
> +      table.insert(CONFLICT_PKGS[pkg_i], pkg_j)
> +      table.insert(CONFLICT_PKGS[pkg_j], pkg_i)
> +      for _, file in ipairs(p2) do
> +        table.insert(CONFLICT_FILES[pkg_i], file)
> +        table.insert(CONFLICT_FILES[pkg_j], file)
> +      end
> +   end
> +end
> +
> +local function table_sorted_keys(t)
> +   result =3D {}
> +   for k, _ in pairs(t) do
> +      result[#result + 1] =3D k
> +   end
> +   table.sort(result)
> +   return result
> +end
> +
> +local function table_sort_uniq(t)
> +   local result =3D {}
> +   local last
> +
> +   table.sort(t)
> +   for _, entry in ipairs(t) do
> +      if entry ~=3D last then
> +        last =3D entry
> +        result[#result + 1] =3D entry
> +      end
> +   end
> +   return result
> +end
> +
> +local ORIGIN =3D read_index()
> +
> +local RESULT_PATTERN =3D {}
> +
> +for pkg, pkgs in pairs(CONFLICT_PKGS) do
> +   local origin =3D ORIGIN[pkg]
> +
> +   if origin then
> +      table.sort(pkgs)
> +      RESULT_PATTERN[origin] =3D table.concat(pkgs, " ")
> +   end
> +end
> +
> +local FILE_LIST =3D {}
> +
> +for pkg, files in pairs(CONFLICT_FILES) do
> +   local origin =3D ORIGIN[pkg]
> +
> +   if origin then
> +      FILE_LIST[origin] =3D table.concat(table_sort_uniq(files), " ")
> +   end
> +end
> +
> +for _, origin in ipairs(table_sorted_keys(RESULT_PATTERN)) do
> +   local pipe =3D io.popen("make -C /usr/ports/" .. origin .. " -V CONFL=
ICTS -V CONFLICTS_INSTALL 2>/dev/null")
> +   local conflicts_table =3D {}
> +   local seen =3D {}
> +   for line in pipe:lines() do
> +      for word in line:gmatch("(%S*)%s?") do
> +        if word ~=3D "" and not seen[word] then
> +           table.insert(conflicts_table, word)
> +           seen[word] =3D true
> +        end
> +      end
> +   end
> +   pipe:close()
> +   table.sort(conflicts_table)
> +   conflicts_string =3D table.concat(conflicts_table, " ")
> +   local conflicts_new =3D RESULT_PATTERN[origin]
> +   if conflicts_string ~=3D conflicts_new then
> +      print("< " .. origin, conflicts_string)
> +      print("> " .. origin, conflicts_new .. " # " .. FILE_LIST[origin])
> +      print()
> +   end
> +end