From nobody Sat Nov 4 22:54:42 2023 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SNCcQ3yLjz504mP for ; Sat, 4 Nov 2023 22:55:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SNCcQ020Pz3F7x for ; Sat, 4 Nov 2023 22:55:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1699138499; bh=Cm/8+eatMUA/Adlf34inE6K9vrVCGN+lOrjSaGDVD+M=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=uegcKW/IAvkUEC+96UfnRrPXINOwXH2eW9j7q67xnu6855D4pdJeSWzjQxDGObs2DRvH1Or8l0bH0PMgD9ZFI8SUUYdizYhtENIOmmtqM3rh3tIB6pSXZCROyw92q/D0c0ZoYcNvdpCLr4bJ3/MoaVv+ahXYpvhHAUyYXFv3scOX3Q5gKl6ALo6Af9PYc1q6rG1LJCYy0WDrxbsZ5oa5OCWlwXrenSciA8vewPj/nnp7ho5kYx8N7db3kRXKrNehDDDHL8Sm7grbo8boPdrSY+Ay+kSUVEPlwjEApYQR0OavlQ7x99rtygGDiz38n6Rz4dUz8SdHv+Ld1ABBzgGdkw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1699138499; bh=rxe9L5mZPizGU8jg9GyL+u3JbCDn11urOc//0sNWtDx=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=Bsv6kvcX90QDnN6/Cl3xOIPZfmVm+UvsdSMDF76bLzwVwuTBmXwy6NRfFl0px6cqhTUvH0HYv2o/hGxprpLEz+nlSyW9xgdltSFQj8LGC5A36noyUCiT4eRzvbnTwcFuCuQfOt5hD/RDoyxpIy0RK+aQ6BSPNq0SiBP2v7wvx5uHb7ewziq2TcLpudl2BJAEeJqXoGC9zHMGOrqtT1M9+I2SveK+Is8KDRaOqzIX1hTWw+6SqLMa/Hi+yDjb4X/gBrdUg2CPahSBFLlFHSgrdXE7m1tZyTN/HljaKtUNgI/0MejgM/cT0OrLpPwNLmOz2kCrEafgvxiDJu84uezwkw== X-YMail-OSG: iRktnb0VM1mowjr240PGQnf8Qz_4HgD_WU_yi_tW3fHN3QcsC0YoVYmSEevb44T HhucYVMk3rTyhgv70kqif3_LYVd9bpIOX9NBRrvaW3k_tPeZNNyRTOYoALEDr6byc0vLXarbffe0 XmNyPaeVZGe43f30drOXnhIdMKcmCjEDpOrKc3eEueHKLXY8b_9dkN5mmFD5sGvcovRN7BiGFoO9 5qyYWFL4styPV7BxSPt9aoGkOo13qtI0wjnC1I.i2JEg26J7G9.LqK4muyqMD7vzUW4PvP.64w_T DK1On9USTtKjJOP03Hh25leYB5qbgGItohpflAfyiZrmluSXcnzHRk527mH33d8TiRM0QXDoxVaP QdWIaVlfrmUq2CMqyufdtlqeV5jdV_lW9Wjl_LG9ufZIuVNuqZSBjeKUb8bhRaQeUTn6NVQI8Lqg Ju9M4hOmN1Nmu1z0kSVqq.GBMqbsIiq2XLOKdTiKkHQCpAgLHMUsUw3BSolJSVwZVJwxJfKoZ2Zo 0RSOzkd.Pc1n34nUve7mK_Zd346zw8MAzTMhx5YUnNuVwEvfUb0F9gS4pBkD5HtY3H6YtYHwTvIy ieG0wFQIyE8MLbqZvCqwP880nHKFl0K6TDHKRZHH6.jzy2ex8DspGC5wdKNmDwFlAL9JlbDIcXF1 JRoWONnIlzkwzZpodFDZdtiL0pBkQDyguyxRMlY1ZImwCmsxHySzDmKvMUsUIzwqSmqK38i9jv_6 26Sq9JqQrGrVEU0_Dd_OjGq0wus6ePyK0tKyxgqJJao_KEdgZhudQfTBh5_bpK9BBs2w41g0Rr.. KVY63M_1tHiTjrqiasfwSoxygsUsMTIhDwG28Kb_Yglm8d0OiR0QI3_OPhPtayCgxPO6lUCfAVX3 chjWzt_g1VnV958eP0cqYWYYVVUzfi2FUvmZ_U63cjSX5vsiEtkh8hC4xY0NXpvWPv.ZU2rD.P4o VklYATo2U0uzVIJ0CyjFZyXGXL8mM.ROe1pMJFzU9e3FMfvhZ.77Ca1bNtfyxVL.4lvF8tzGtd8O uKkTGfHEs91lPM6aZCNH3uMC6flb8EVyhp5siXK0nDJ1rjVCy_qrM05MbbDkoNV2_7QKU96XoMIX 4jw2Uh6hTlN4Z207RVgZC_8BMtZ5B_OxiryWQJpo5CTEBGhh5L6RaGUcDXL0FHQ_tNpcmVL7AFep XaIMFaXzU6Gm1zuOY7gZ5GG4ygYc8A4r0DDmAaZh1KZFne1liImZZXZcKUhF6ybeq2z.xmY2wQTX FqE2.Nw6HjXPWfCP73n0SM4FpvKdN3Bql4fu0Hk9LURNLgO29j8r0tNAQOmeTrmWAmpyfCCu0IHw yY6Y9ZMKnpWyjZJROPePBh5loPitTcg3BgvcdpVmf6_9IfFUjIOV.KH6xIwEQ3I8t_8m93wYGAsK 8_xwfMj777n5b7MLXo7uTVWTrZC_eW1aa7NLSuY9JOTUT4VzT0zKYdmgYDldwiAV3yY2wgccLnAt oSAbhFZOdm..5ek902ZDgcVER948DvCv68A.F91QS4P7G_XmioWVHAI6Fhsjk70w6c98BIuIPJ_M 0.triT82046YINfTlS0henVGlr9zOoNopLY3Y4OScP23Al9byzzN7tpwv5zhaTuhz0tYYIkg_j88 vMOpGG_otv4ZN4oDzcMpb7DxoU5tsetQCA_9XXqJ8y7BuaWxTtTnH35wpfkzIzdG5obvWlyodk4q V2u9p8pL7SQZMkXReGKWbSsCR6jE7Opntf8MJvkg28lqGDtXaRZRTlajSOnGDD_t9FkqF81UZy4j 5MRDLtVnH8gUTgI2vFMqsGkdzzi0oQBq.hFP7iT0cSBSoIiGGqWFFQxZPJblN6Ls3b7L.lNH75Zf ogJiUdXpjDd08dyGJ.12IEi65jYxcOOcMyjkT09P3gga4xQ7r376D8O.msIX7m9xAKwEzcFiAh_0 nQFy2RPD351VlvnJqbzjEjUfqHF6_TaWTNhhXyAEFCVS3LtBRsFI_T0kUU9nl2tzpqp90VxHDPGy cuBEXovi61Oh5uMrKW9a.OoVlT2O4K4MHLSzNiev5ufsXmQNNS0uk3qHCSDs1TZKZYguxVbqQJ4C m7OqNshDYj9Tn6r7PWwyPkauYiW2oxtRJxhbqWUpTtvyj.L8cHw7l2gfmWqMgEHOqNWymq1MOUi5 cQut1wQMGw3Z26hXZKvqmU8MYwnOu7MR6v1AAVudeEFHhLF9TzWS0rOhl743iY1cyY9Oy9sCFU0R brFLGGTaoxHT6JKqSiUtOEDcjWUCkXSjydrTQzhP6Uup1xIQfhS.I3yYQ4sDqog7THYq69ybvHqA - X-Sonic-MF: X-Sonic-ID: b409e03b-729c-4fd7-8d8a-7082304c282e Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Sat, 4 Nov 2023 22:54:59 +0000 Received: by hermes--production-ne1-56df75844-n6ltj (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID a2c024cb2f96a1198ada8c35e5e3d78f; Sat, 04 Nov 2023 22:54:53 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: Re: poudriere bulk -a fails on UFS: "Too many links" under logs/bulk/latest-per-pkg/ and then "Failed: starting" From: Mark Millard In-Reply-To: Date: Sat, 4 Nov 2023 15:54:42 -0700 Cc: "freebsd-ports@FreeBSD.org" Content-Transfer-Encoding: quoted-printable Message-Id: <17C21264-D87B-40D3-9228-579F7BC3DD7C@yahoo.com> References: To: Tatsuki Makino X-Mailer: Apple Mail (2.3774.200.91.1.1) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Queue-Id: 4SNCcQ020Pz3F7x On Nov 4, 2023, at 13:56, Tatsuki Makino = wrote: > Hello. > (I am replying to freebsd-ports only.) Seems reasonable. > Mark Millard wrote on 2023/11/04 21:00: >> # ls -a /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ | wc -l >> 32767 >=20 > The number of packages will be inflated relative to the number of = origins because FLAVORS uses PKGNAMEPREFIX and PKGNAMESUFFIX. /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ contains direcotries that have the names of the packages with log files form bulk runs, not origins. > Therefore, something like BUILD_ALL_PYTHON_FLAVORS should not be used = continuously for purposes other than development and debugging. The from scratch bulk -a run is a kind of rare experiment that I do, not a regular thing. It is deliberately a simple bulk -a run when I start such an experimental sequence. > Ports for which the origin directory no longer exists, and whose = description no longer exists from MOVED (e.g. www/seamonkey), seem to = remain in the package forever. > It is better to run poudriere-pkgclean regularly. The sequence used to start my experiments is as shown below: #! /bin/sh poudriere pkgclean -jmain-amd64-bulk_a -A poudriere bulk -jmain-amd64-bulk_a -a So pkgclean had been run. However, as I remember, this does not manage log files. I do that separately. For /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ being empty at the start, the failure would still happen. The 32767 is a UFS limitation via a 16 bit field associated with: /usr/include/ufs/ufs/dinode.h:#define UFS_LINK_MAX 32767 Specifically di_nlink in sys/ufs/ufs/dinode.h : struct ufs2_dinode { uint16_t di_mode; /* 0: IFMT, permissions; see = below. */ int16_t di_nlink; /* 2: File link count. */ . . . Having more than 32767 packages attempted to be built leads to: /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ trying to go beyond that UFS_LINK_MAX limit in what latest-per-pkg/ contains. ZFS does not have the limitation and the prior from-scratch ZFS based bulk -a experiment was able to finish its build. bulk -a in the ZFS context historically used somewhat more RAM+SWAP space than a UFS based bulk -a would. > Would checking that area reduce that number? For amd64, there are over 32767 packages that should get directories in /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ . The usage poudriere puts them to counts them in di_nlink in UFS's directory inode and the UFS limitation is reached before all the packages are present. /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ ends up just not fitting in UFS's limitations for how things are currently organized. I used to be able to compare/contrast the ZFS vs. UFS RAM+SWAP usage for bulk -a runs. But last I did such was before the builds going beyond 32767 packages with logs. (I've not checked if /usr/local/poudriere/data/logs/bulk/latest-per-pkg/ is the only issue. It is just the first context to stop the build.) =3D=3D=3D Mark Millard marklmi at yahoo.com