From nobody Tue Oct 31 08:57:48 2023 X-Original-To: ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKPDX2Tvbz4y5vW for ; Tue, 31 Oct 2023 08:58:04 +0000 (UTC) (envelope-from tatsuki_makino@hotmail.com) Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01olkn2019.outbound.protection.outlook.com [40.92.53.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKPDW09Lgz4GpF for ; Tue, 31 Oct 2023 08:58:03 +0000 (UTC) (envelope-from tatsuki_makino@hotmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=CPG6L12z; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (mx1.freebsd.org: domain of tatsuki_makino@hotmail.com designates 40.92.53.19 as permitted sender) smtp.mailfrom=tatsuki_makino@hotmail.com; dmarc=pass (policy=none) header.from=hotmail.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OygzJB2dDL9BYF9EmUPzcMEOMJ78K5Wvpda1oPGNPcpnPa33qi7ZSusybsdv0mpwf/+SSrp/2H7+KogxLFeU2Ivb4zT/gCrYPpNkF6YcTFg5nf3dsl/nXT2y2HFNyeA+lRFaZLGmTeX+z5xIu2CabM1xx/OEZ8HPHMajz/WnGO8ieTJ9J2fiuyCdWjRtrjCPXox/by0NuQ/gBNbg73w813NRWKayAbARIDiXFs8XbCx2c+8yzqIBs/+HgtEaZOg4E61z6l3XAmWzPQFcV9Uc/Csn3GELrpDDNoytsE7ov43uFFznaLl+RPIZcAB8zi6rdONQDntcpjdOvgT/I/5tGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YA9asVsDezJ4icu+Yk1Rc0a67qSTrhxFrguNiAgQQ9M=; b=cUqwJLFWAWuxyNeNPYppMPvxZNMmchs0KqLz4QudsxnOzVImqsSU1E5FTItLFvpoEXgs12XbQgj54aGeOOsBewA+tOmn8SXylweFpgQ3cy44HV5Jmf8vlWMiEKn4mGc25q9kKRyR2MYo5rx40+kZS3SAn337GXj5BP9efwGTxrlKleDVGBwt/j7LZF4LOHG9LbiXySCOkYdOODFkk8j7V409YMoht6TUy5JjxJ3x8TS7Bi4sA8tVQ4vzQyruqwEn0BCrKlkdbKoXHNaAmexfp/KX8JC50cVQpiuYpP8p388/KaUBaTpB3/MH3lQG5WfCNKVthk1m0MNOsqkttCpZxw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YA9asVsDezJ4icu+Yk1Rc0a67qSTrhxFrguNiAgQQ9M=; b=CPG6L12zxeFb3Q92POn00VQihzLg9n/bk8A6OwIeJR+WdMnv0zV4IbTVQcT/29sqX7YulqZHwzz1Mw/LGXE36wBmwR7cc44HBRLuA3Rqj3ajrxPGKtMKSEXLIMfJ0IHVPUqDeflQFyveBzccn5F6bsPsmus6vMc0CStaqx1H6UMAgfkxmVXA4IVKlVZGGEoeNltftLXwFaj/L0Cf0fLndiBFX0ENqbWk4i2hOgF/86plTxLc3eKCmLGcBvRk7ifzFUTL9/v1cfJKyu/b4cCDwtlRp+gpQMg96TTTGsWzokg5TrwXZ2RcpvktVNesFZKMml8dY8bH8R4fiLvE3r2/6w== Received: from SI2PR01MB5036.apcprd01.prod.exchangelabs.com (2603:1096:4:1f8::9) by PUZPR01MB4731.apcprd01.prod.exchangelabs.com (2603:1096:301:101::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.29; Tue, 31 Oct 2023 08:57:58 +0000 Received: from SI2PR01MB5036.apcprd01.prod.exchangelabs.com ([fe80::fb1a:653f:16a:b4d2]) by SI2PR01MB5036.apcprd01.prod.exchangelabs.com ([fe80::fb1a:653f:16a:b4d2%5]) with mapi id 15.20.6933.024; Tue, 31 Oct 2023 08:57:57 +0000 Subject: Re: We need to do something about build times To: ports@freebsd.org References: From: Tatsuki Makino Message-ID: Date: Tue, 31 Oct 2023 17:57:48 +0900 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.4 In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-TMN: [iXYZsT82M+3x86BI8aHfQVIrD7v5IhY8] X-ClientProxiedBy: SG2PR04CA0154.apcprd04.prod.outlook.com (2603:1096:4::16) To SI2PR01MB5036.apcprd01.prod.exchangelabs.com (2603:1096:4:1f8::9) X-Microsoft-Original-Message-ID: <2adb72e0-2f33-aa02-4580-af6bc916a5b3@hotmail.com> List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SI2PR01MB5036:EE_|PUZPR01MB4731:EE_ X-MS-Office365-Filtering-Correlation-Id: 00c1a289-8676-4e52-f27b-08dbd9ef7a1c X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3APJYLQ+3FCFUZJGmbCE0G11YUNsE4ksO7x80e3yTCASM66istQgy4eufiK0eW/qElrg8E8kBaCAjMY29uczS3N1tFieM55p0wtvmTLWeapsoejWCf31X/7mIJinT5m+BGkVGH3/ZjYWx1aU/s/mfWGDyglP9LoqEp+yJ7ENxsjD/QxQWhLRHa/H9JTuvcdl690yss+1aYuXjUphDMBF2xbN67fw3T4iV9TYBNXfTOHuJG3DwBefguaXHDAjVkmdhqyWYKcngiVx6JFfSUNBlqNT0RcHFKKvtWyk2zcbCn15OL1gJj9E+5DNzFNsAMP3yLKx7A0m909Vyg14oDWKJAa19kpejH+op9Rgjn1x6Nb7IOKcX+qaEaQ8TAjNgkG8VwXqGCD4EjqmUgEpy0v6foViE0Aq4me4Uu8eDXmOEfjJNEONfQnDFh+X0uWnPJolGlgAboLvLfJar78BcDnyTkp+qKTVHLSPVhz5WnyVQKmpS5QHf9eX8+bGc9d/9o/EHxsHEuRKijwhDB9YV07S9IwkiGaTjF4VPcZmpvR546NKzrLY5Hn6N+ny5o8eOrQQf/8BGCfGtVVltYR+ESjY0r4q3SKkOBqaNBDWJTH9LPc= X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dnhtQjhCVkFIczdRTTd6MlRnSURjdGRyeEJZbjU5d1pjWmRtWHZ5R0EzTG5M?= =?utf-8?B?dFlZc1lPSkQwNnMzSTE1MmV0MEFwdHFHWDV0SE1GOXAyYkhJbHJNa3FiZWN0?= =?utf-8?B?MUZqdWZBV09uV051YjVidW1COU9jbHUwbnQxSGY0YzVmWW9MTndhRko4Ympa?= =?utf-8?B?WFd1K1dLbkkzS2dsSWxLOUJPYzZkdU5TSk5Mb2tHOS93MVlGVmFBampDOG90?= =?utf-8?B?T2Mwb0dENzFvVk51T3NsYnNaUytBR2t0TGRMdzJmanpJV2Y3SVFQZUpoWUhj?= =?utf-8?B?WkNPNXFaWU5jL1ByTlF0QXQwTUVXclJZL1MvdGxFSEdYbFlyTHpiOGUwaWpX?= =?utf-8?B?eXZMNnFLbWc1MVFCN25DN2pqQndVc3NmdlRVa0dqUm80NWRGMkppOTlLWjRr?= =?utf-8?B?SHVwTGphSTVjUzRUWVJsa2VQRUozcXNxaTFmcmRkOVpPWitwWmwxQTJjc2pw?= =?utf-8?B?SWhqRm0zOWdHRGlLU1BXS1N2ZFlEeDladnZHTzN2TWU1NTNqMzZEMEw3QzNQ?= =?utf-8?B?aUUvbGViZHZFeXlRS3p2TWsxSmxjbmhiMVZBSXhidEpPZVVIYUE2WHd1alY4?= =?utf-8?B?TlpnNVNUenlTTHdITlpwSGxMeE90U2NiR3F6bFJqK1JJdzNLMXFaWFhwOWMw?= =?utf-8?B?dkNXT1JKTzNzYTRQNnlGd3d2V3M1SDdxcEJuVjd3R25pK0x5cndGU1BDUVE0?= =?utf-8?B?eE5SZkhWY3VIOE1CT0JmcS9zUGlDN2R3MDNubFR1OTkxcTJkd0w4Mms0VVh3?= =?utf-8?B?aFFKbDdBcnVhSEFxVTBNOGpCVnJCTjF6cFhpb0poeHE5YlRlc3NOZ3N1eFhj?= =?utf-8?B?OHoyTng0Mkh2eFBEaENkTE1wZldCVE13NG0wcERxaEtsMzhSY05ZekkwZmxM?= =?utf-8?B?VWo3UzhnL2hwT2U0NmV6UW9ES2FNL0hKTllBcFNGOEViUmJGMURiaXhyd3NB?= =?utf-8?B?RnBkMTBzT09PcGNHcDFrY0QxdlZiZUkxSmpDVGhsUkswWEFsOHVjaU92SmJT?= =?utf-8?B?MVg1OTdKbFE3NFBSdGgrWkRSY3FvdHZNQjhaSVJyZUtlOGZleGdKaDJPekNC?= =?utf-8?B?TURoOFhrYzNzTHNFMzBZajB0aVJodTRFbEwrdDh0eGV3VTg4M2RwZm9yeG9D?= =?utf-8?B?U2NwU3kzM3Nvb3VWQVBZTm5rK2RqRUYra3psY3VhSFJwMVdENjJJQkdkQnh5?= =?utf-8?B?VGtDVzVnOHR0RTByVjgwYzhXOHMxZWxybnB1QUZHTktWd1hFU1cyOE5Na2U1?= =?utf-8?B?SzJSa3ZuK3Z1dmVpdzdqTnd2dnZkR2NCa2ZPYnNGRFhEYU81S1VQaytlT1d1?= =?utf-8?B?WnU1bW1PVndoZ2JidjIyTzVTTzlkNXNsdEdhTVNHa3oyK041a0tzUmxUWWNi?= =?utf-8?B?WkdjaHBrbzdBVlhqTlBLcnJrZElpbW4yMWV0SVFMVnkxQlZYR3BhalV1MVR6?= =?utf-8?B?U0pHUk1IMFkvQlBjSHo4dzdYMkpwd0VqZUp6NjFKdnViY040VEVzaGo2NHh0?= =?utf-8?B?Z1I4QnFpKzZnUm5zUGUxQTdUN1dOd0ptMlJ4KzhROFcrbnNMUGN4NHZ6bTZE?= =?utf-8?B?UHMxUnlKS1NjQTBldWhRSVBRLzRzWUJ4RU5KdjlRNlJ4MlpudnI1a2gvY3pH?= =?utf-8?Q?NMy1tia0Df88zDjbTVkc1LrYR08gWJRaRkqy4NmFG/S8=3D?= X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-d8e84.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: 00c1a289-8676-4e52-f27b-08dbd9ef7a1c X-MS-Exchange-CrossTenant-AuthSource: SI2PR01MB5036.apcprd01.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Oct 2023 08:57:57.6575 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PUZPR01MB4731 X-Spamd-Result: default: False [-1.93 / 15.00]; FORGED_MUA_SEAMONKEY_MSGID_UNKNOWN(2.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; NEURAL_HAM_SHORT(-0.43)[-0.428]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/16]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[ports@freebsd.org]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.53.19:from]; FREEMAIL_FROM(0.00)[hotmail.com]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[hotmail.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_NONE(0.00)[]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim] X-Rspamd-Queue-Id: 4SKPDW09Lgz4GpF X-Spamd-Bar: - Hello. Robert Clausecker wrote on 2023/10/25 04:12: > And it seems I'm slowly killing my build SSD like that. After just about > 9 months, it is already at 100 TB of writes just from port builds. > Building with workdirs in memory is no longer an option as that frequently > kills my build server by filling all its RAM with build files until no > processes can be started anymore. Poudriere does not have an effective > mechanism to prevent this (tmpfs limits don't work as the ports in > question require very large workdirs, tend to take very long to build and > tend to be built all at the same time for multiple jails). Modern compression algorithms are so efficient that some distfiles inject source code into WRKDIR like mentos cola. FreeBSD is not MentOS. If wrkdir is placed in tmpfs by USE_TMPFS, that alone will cause excessive memory usage. I don't know how effective this would be, but I have thought of the following setup. The goal is not to reduce the amount of reading and writing, but to change the speed of reading and writing. Right now I am trying this on poudriere-3.3.7_4. Changes for /usr/local/etc/poudriere.conf Set USE_TMPFS to the following settings USE_TMPFS="data localbase" Changes for /usr/local/etc/poudriere.d/make.conf The following block will be added so that the stage can also be placed in tmpfs. .if 1 STAGEDIRPREFIX= ${LOCALBASE}/.stage STAGEDIR= ${STAGEDIRPREFIX}${.CURDIR}/${_WRKDIR}/stage _PORTS_DIRECTORIES+= ${STAGEDIRPREFIX} _USES_fetch+= 980:..stagedirprefix ${STAGEDIRPREFIX}/😈: ${STAGEDIRPREFIX} @${CHMOD} -R 1777 ${STAGEDIRPREFIX} @${TOUCH} ${TOUCH_FLAGS} ${.TARGET:Q} ..stagedirprefix: ${STAGEDIRPREFIX}/😈 .endif It is a bit tricky to make it so that even nobody can write on it :) I don't know how well this is working as I am now having chromium made on the same machine as poudriere :) As far as df -h -t tmpfs is concerned, it does not seem to be allocating a lot of memory to tmpfs. Regards.