From nobody Mon Oct 7 04:07:59 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XMQcD2fPJz5Yxf4 for ; Mon, 07 Oct 2024 04:08:12 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [IPv6:2001:678:618::40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "plan-b.pwste.edu.pl", Issuer "GEANT OV RSA CA 4" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XMQc93Gx7z4YFX for ; Mon, 7 Oct 2024 04:08:09 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=plan-b.pwste.edu.pl header.s=plan-b-mailer header.b=UoneKOJb; spf=pass (mx1.freebsd.org: domain of zarychtam@plan-b.pwste.edu.pl designates 2001:678:618::40 as permitted sender) smtp.mailfrom=zarychtam@plan-b.pwste.edu.pl; dmarc=pass (policy=quarantine) header.from=plan-b.pwste.edu.pl Received: from [192.168.7.70] (dom.potoki.eu [62.133.140.50]) (authenticated bits=0) by plan-b.pwste.edu.pl (8.18.1/8.17.2) with ESMTPSA id 497480lQ003800 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Mon, 7 Oct 2024 06:08:00 +0200 (CEST) (envelope-from zarychtam@plan-b.pwste.edu.pl) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=plan-b.pwste.edu.pl; s=plan-b-mailer; t=1728274082; bh=ARh71w+WVtQWR3ezIdm+agwClkOAPpwBSh6G1cVRbII=; h=Date:Subject:To:References:From:In-Reply-To; b=UoneKOJbsAFqEO08sqz6PHtKMrUA5fFz9R5fkua5wqhznKirYOr+uRe9ixfSrYeqi IEaOr8x+O/J3Qmkrpy722PgJ0gYB4LWUJre66kOpP4VXUzIF3TEUSa4iKAkhpNj4BE QrzyNFQLZswJeBvBCvoSLnw0FPtmejvFaY22OowyqwSP29/As/bhfatSMvkoiWxhdy UVD2K6FSQJuKIOcA8WxyylxJF2VLOUuFliDc1B2KnGJhjUdZW0XLpwpqhWOg3S82tx BDlk+WtzpWbdkFedrxhV8ul4HorjXpLBGbxfHXHrOCGthbwA7cSNYhjXc5lqnJnRqt /xXZdINRaZ8Ng== X-Authentication-Warning: plan-b.pwste.edu.pl: Host dom.potoki.eu [62.133.140.50] claimed to be [192.168.7.70] Message-ID: <1fd47603-0bf2-4fcf-a556-22335d99e203@plan-b.pwste.edu.pl> Date: Mon, 7 Oct 2024 06:07:59 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Review D38047 ... and then there was one.... To: freebsd-hackers@freebsd.org References: <43F7106E-C5C5-4467-9B72-1D7C51E5430B.ref@yahoo.com> <43F7106E-C5C5-4467-9B72-1D7C51E5430B@yahoo.com> Content-Language: en-US From: Marek Zarychta Autocrypt: addr=zarychtam@plan-b.pwste.edu.pl; keydata= xsBNBFfi3cMBCADLecMTFXad4uDXqv3eRuB4qJJ8G9tzzFezeRnnwxOsPdytW5ES2z1ibSrR IsiImx6+PTqrAmXpTInxAi7yiZGdSiONRI4CCxKY9d1YFiNYT/2WyNXCekm9x29YeIU7x0JB Llbz0f/9HC+styBIu2H+PY/X98Clzm110CS+n/b9l1AtiGxTiVFj7/uavYAKxH6LNWnbkuc5 v8EVNc7NkEcl5h7Z9X5NEtzDxTOiBIFQ/kOT7LAtkYUPo1lqLeOM2DtWSXTXQgXl0zJI4iP1 OAu4qQYm2nXwq4b2AH9peknelvnt1mpfgDCGSKnhc26q6ibTfMwydp+tvUtQIQYpA6b9ABEB AAHNN01hcmVrIFphcnljaHRhIChQbGFuLWIpIDx6YXJ5Y2h0YW1AcGxhbi1iLnB3c3RlLmVk dS5wbD7CwHcEEwEIACEFAlfi4LkCGwMFCwkIBwIGFQgJCgsCBBYCAwECHgECF4AACgkQHZW8 vIFppoJXdgf8D9X3VRFSNaR9lthSx/+uqas17J3FJKBo1xMQsC2a+44vzNvYJSuPGLLJ+LW2 HPVazjP/BWZJbxOYpliY4zxNRU0YCp0BLIVLibc//yax+mE42FND/+NiIZhqJscl6MLPrSwo sIwXec4XYkldkyqW/xBbBYXoIkBqdKB9j5j42Npy1IV/RizOSdmvTWY27ir8e/yGMR1RLr4F 8P5K3OWTdlGy2H2F/3J8bIPBLG6FpaIyLQw4dHSx8V02PYqDxK1cNo2kAOnU8PnZL/AGuMOH iv3MN1VYL8ehcmpBBsrZGebQJxrjY2/5IaTSgp9xHYT70kshuU6Qb97vk1mOjNZxgc7ATQRX 4t3DAQgA10h6RCXuBLMHxq5B8X/ZIlj9sgLoeyfRdDZEc9rT2KUeUJVHDsbvOFf4/7F1ovWY hJbA6GK/LUZeHHTjnbZcH1uDYQeHly4UOLxeEvhGoz4JhS2C7JzN/uRnwbdOAUbJr8rUj/IY a7gk906rktsc/Ldrxrxh7O6WO0JCh2XO/p4pDfEwwB37g4xHprSab28ECYJ9JMbtA8Sy4M55 g3+GQ28FvSlGnx48OoGXU2BZdc1vZKSQmNOlikB+9/hDX8zdYWVfDaX1TLQ8Ib4+xTUmapza mV/bxIsaZRBw+jFjLQHhTbIMfPEU+4mxFDvTdbKPruKPqVf1ydgMnPZWngowdwARAQABwsBf BBgBCAAJBQJX4t3DAhsMAAoJEB2VvLyBaaaC6qkIAJs9sDPqrqW0bYoRfzY6XjDWQ59p9tJi v8aogxacQNCfAu+WkJ8PNVUtC1dlVcG5NnZ80gXzd1rc8ueIvXlvdanUt/jZd8jbb3gaDbK3 wh1yMCGBl/1fOJTyEGYv1CRojv97KK89KP5+r8x1P1iHcSrunlDNqGxTMydNCwBH23QcOM+m u4spKnJ/s0VRBkw3xoKBZfZza6fTQ4gTpAipjyk7ldOGBV+PvkKATdhK2yLwuWXhKbg/GRlD 1r5P0gxzSqfV4My+KJuc2EDcrqp1y0wOpE1m9iZqCcd0fup5f7HDsYlLWshr7NQl28f6+fQb sylq/j672BHXsdeqf/Ip9V4= In-Reply-To: <43F7106E-C5C5-4467-9B72-1D7C51E5430B@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.89 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[plan-b.pwste.edu.pl,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[plan-b.pwste.edu.pl:s=plan-b-mailer]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:206006, ipnet:2001:678:618::/48, country:PL]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[plan-b.pwste.edu.pl:+] X-Rspamd-Queue-Id: 4XMQc93Gx7z4YFX X-Spamd-Bar: --- W dniu 7.10.2024 o 00:36, Mark Millard pisze: > Marek Zarychta wrote on > Date: Sun, 06 Oct 2024 20:13:58 UTC : > >> W dniu 6.10.2024 o 22:04, David Cross pisze: >>> Here’s the thing. The current implementation of nscd DOESN’T WORK at all. There is a symbol that nscd exports that libc is supposed to use as a flag to bypass lookups for nscd itself. But that symbol isn’t exported right. >>> >>> You will need to recompile libc and nscd. (I just do a buildworld to make sure i get everything as there are makefile changes related to the aforementioned symbol changes. >> Yes, without world installed this patched nscd won't even start: >> >> [host] /usr/src# service nscd start >> Starting nscd. >> limits: setrlimit pipebuf: Invalid argument >> /etc/rc.d/nscd: WARNING: failed to start nscd > . . . > > This note is only about the "limits: setrlimit pipebuf: > Invalid argument" notice. > > The main [so: 15] pipebuf related commits were done during > 2024-Sep-20 UTC. If one has a kernel that predates those but > a world for which limits now tries to use the new pipebuf > material, the result is messages like that: > > limits: setrlimit pipebuf: Invalid argument > > (or related such messages). > > > For reference for main [so: 15]: > > Fri, 20 Sep 2024 > . . . > • git: 3458bbd39778 - main - kernel: add RLIMIT_PIPEBUF Konstantin Belousov > • git: 54a8d1fbbf65 - main - getrlimit(2): document RLIMIT_PIPEBUF Konstantin Belousov > • git: a4c04958f526 - main - libutil: support RLIMIT_PIPEBUF Konstantin Belousov > • git: 5d92f20c7d31 - main - bin/sh: support RLIMIT_PIPEBUF Konstantin Belousov > • git: f54f41403d14 - main - usr.bin/limits: support RLIMIT_PIPEBUF Konstantin Belousov > • git: b029e29e0d8b - main - login.conf: add a placeholder for the pipebuf limit Konstantin Belousov > • git: 80133d678ecb - main - procstat: support RLIMIT_PIPEBUF Konstantin Belousov > • git: 8ae779832c6f - main - privs: add PRIV_PIPEBUF Konstantin Belousov > • git: 7672cbef2c1e - main - pipes: reserve configured percentage of buffers zone to superuser Konstantin Belousov > . . . > • git: d6074f73af5c - main - pipe: use pipe subsystem KVA counter instead of pipe_map size Konstantin Belousov > • git: 40769168a5ee - main - pipespace_new(): decrease uidinfo pipebuf usage if reservation check failed Konstantin Belousov > . . . > • git: a52b30ff98cd - main - sys_pipe: consistently use cr_ruidinfo for accounting of pipebuf Konstantin Belousov > • git: af96ccc6a508 - main - uifree(9): report non-zero values for all shared resources Konstantin Belousov > • git: 2c1963d46335 - main - procfs rlimit: handle pipebuf Konstantin Belousov > • git: c84d8db0ab3d - main - procfs: ensure that RLIMIT_IDENT is properly updated when a limit is added Konstantin Belousov > > The combination of an older kernel and a newer world will not be > nicely behaved when any non-kernel code from the above ends up > involved. > > > stable/14 has now also had the commits: > > Sat, 05 Oct 2024 > • git: 1508dce2502d - stable/14 - procfs: ensure that RLIMIT_IDENT is properly updated when a limit is added Konstantin Belousov > . . . > • git: b7eecc86c3bd - stable/14 - kernel: add RLIMIT_PIPEBUF Konstantin Belousov > • git: d20f0dae2f97 - stable/14 - getrlimit(2): document RLIMIT_PIPEBUF Konstantin Belousov > • git: a03f7c040ce7 - stable/14 - libutil: support RLIMIT_PIPEBUF Konstantin Belousov > • git: d5ed8778bf3b - stable/14 - bin/sh: support RLIMIT_PIPEBUF Konstantin Belousov > • git: 25902860b270 - stable/14 - usr.bin/limits: support RLIMIT_PIPEBUF Konstantin Belousov > • git: 524b9810de6a - stable/14 - login.conf: add a placeholder for the pipebuf limit Konstantin Belousov > • git: 6600090e678e - stable/14 - procstat: support RLIMIT_PIPEBUF Konstantin Belousov > • git: fd9babb1b85f - stable/14 - privs: add PRIV_PIPEBUF Konstantin Belousov > • git: d532d9926ee7 - stable/14 - pipes: reserve configured percentage of buffers zone to superuser Konstantin Belousov > • git: 6536b979b856 - stable/14 - pipe: use pipe subsystem KVA counter instead of pipe_map size Konstantin Belousov > • git: a8c663bb4261 - stable/14 - pipespace_new(): decrease uidinfo pipebuf usage if reservation check failed Konstantin Belousov > • git: c15b2e046e8c - stable/14 - sys_pipe: consistently use cr_ruidinfo for accounting of pipebuf Konstantin Belousov > . . . > • git: fc9070bf1d16 - stable/14 - procfs rlimit: handle pipebuf Konstantin Belousov > . . . > > Again, the combination of an older kernel and a newer world will not be > nicely behaved. > > === > Mark Millard > marklmi at yahoo.com > > > Thank you for this deep investigation. Anyway, after the kernel and world update, I could test the patch, but there was no significant progress, at least in lookup timing (as I pointed out before). -- Marek Zarychta From nobody Mon Oct 7 05:05:19 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XMRtS6Sl3z5Z1fr for ; Mon, 07 Oct 2024 05:05:36 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-qv1-xf36.google.com (mail-qv1-xf36.google.com [IPv6:2607:f8b0:4864:20::f36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XMRtS4hMTz4cR9 for ; Mon, 7 Oct 2024 05:05:36 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qv1-xf36.google.com with SMTP id 6a1803df08f44-6cb3d2ce75eso43941096d6.0 for ; Sun, 06 Oct 2024 22:05:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728277536; x=1728882336; darn=freebsd.org; h=to:in-reply-to:cc:references:message-id:date:subject:mime-version :from:content-transfer-encoding:from:to:cc:subject:date:message-id :reply-to; bh=GAnTsx8737U0yzWk935qMkt/0nfXNdy1bIjUZAzffFQ=; b=m6xekl7fBWq3RzeFjsBXerqgwUn5VQOFVuEtuF7WA3l+BjzTw6L/Msh37hGamHSWn8 ARy+KwFKZVGCKlCJVNNgZFqbIht/LCs3aWBilgL7D1UI7ipUacWmNRkVuCTaOOdoC5ZC NNmHMJGy58GjyNDXQC8vi6JEdMo7y7eApqsFuEP7eJJttCUZDPa75bK2cXCXoRtNjocq xJzSK8rL4KEaGUQKx66gglYP706PaBBMT4lWndG0dPPZW8/ci8RCl8xsrC5TeFEBtcvr f50KCzqT3qmMTcA7w1kN7MQXUnuypgVDFyziHrUfIJEQcPDrD9z8BwTaFgQ0tbTavBm1 u4Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728277536; x=1728882336; h=to:in-reply-to:cc:references:message-id:date:subject:mime-version :from:content-transfer-encoding:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GAnTsx8737U0yzWk935qMkt/0nfXNdy1bIjUZAzffFQ=; b=t/qThifRAjH8tTZkY7GNoTGtpU3yOag0YBvpdemMJNa+5057GguX/aWP1K9dxUz6PZ kZgAo/PiAjzrkhuEqg/qFZxRY/i/Lz4kjdW9n/mxcEmsSO2Hb/DjzFWNacJdnCCx1bzH FnPRBumOuEdWAkPlhFx5Do/Q1QooUwnFILwaRNB7OtFZE1I9i+aqioRaW1HLpDGM/NbU fuVDFVS/1iRhDWojvDCv653k9eqs1JvwjsAB7OhsJj7mmidlYxIcgyW9vvSpPkgrCJrw 9KIEzXRh2NOT5G7TLD8wtAlpAKo5da6kZs0OKobsgVcIeCILxRfz4qD7LtRIVQDys+tu YjtQ== X-Gm-Message-State: AOJu0Ywy2L8mdqF4AB4Tocn2dj64VujBZRlzH/eWJWjmHok2X+5ETdXF cbLMzTi2wHIjFhSlH9wmq4UCC6bbzrJbMXGFSEFjsdjAd4Mh38l2Rt9Sqg== X-Google-Smtp-Source: AGHT+IEsfrxMZ3G/1K+OJDawEx5ybUGPFTsharMZu8R0Z/Fp1zz9eA+x24pyr+RuELZkzFYGyDPh0Q== X-Received: by 2002:a05:6214:5c44:b0:6cb:584f:ec22 with SMTP id 6a1803df08f44-6cb9ae07b6fmr156818766d6.21.1728277535862; Sun, 06 Oct 2024 22:05:35 -0700 (PDT) Received: from smtpclient.apple (syn-024-097-005-251.biz.spectrum.com. [24.97.5.251]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6cba47513acsm22050856d6.101.2024.10.06.22.05.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 06 Oct 2024 22:05:34 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: David Cross List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: Review D38047 ... and then there was one.... Date: Mon, 7 Oct 2024 01:05:19 -0400 Message-Id: References: <1fd47603-0bf2-4fcf-a556-22335d99e203@plan-b.pwste.edu.pl> Cc: freebsd-hackers@freebsd.org In-Reply-To: <1fd47603-0bf2-4fcf-a556-22335d99e203@plan-b.pwste.edu.pl> To: Marek Zarychta X-Mailer: iPhone Mail (22A3370) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4XMRtS4hMTz4cR9 X-Spamd-Bar: ---- How many entries are in your ldap structure? I can attempt a replication he= re > On Oct 7, 2024, at 12:08=E2=80=AFAM, Marek Zarychta wrote: >=20 > =EF=BB=BFW dniu 7.10.2024 o 00:36, Mark Millard pisze: >> Marek Zarychta wrote on >> Date: Sun, 06 Oct 2024 20:13:58 UTC : >>=20 >>> W dniu 6.10.2024 o 22:04, David Cross pisze: >>>> Here=E2=80=99s the thing. The current implementation of nscd DOESN=E2=80= =99T WORK at all. There is a symbol that nscd exports that libc is supposed t= o use as a flag to bypass lookups for nscd itself. But that symbol isn=E2=80= =99t exported right. >>>>=20 >>>> You will need to recompile libc and nscd. (I just do a buildworld to ma= ke sure i get everything as there are makefile changes related to the aforem= entioned symbol changes. >>> Yes, without world installed this patched nscd won't even start: >>>=20 >>> [host] /usr/src# service nscd start >>> Starting nscd. >>> limits: setrlimit pipebuf: Invalid argument >>> /etc/rc.d/nscd: WARNING: failed to start nscd >> . . . >>=20 >> This note is only about the "limits: setrlimit pipebuf: >> Invalid argument" notice. >>=20 >> The main [so: 15] pipebuf related commits were done during >> 2024-Sep-20 UTC. If one has a kernel that predates those but >> a world for which limits now tries to use the new pipebuf >> material, the result is messages like that: >>=20 >> limits: setrlimit pipebuf: Invalid argument >>=20 >> (or related such messages). >>=20 >>=20 >> For reference for main [so: 15]: >>=20 >> Fri, 20 Sep 2024 >> . . . >> =E2=80=A2 git: 3458bbd39778 - main - kernel: add RLIMIT_PIPEBUF Konst= antin Belousov >> =E2=80=A2 git: 54a8d1fbbf65 - main - getrlimit(2): document RLIMIT_PI= PEBUF Konstantin Belousov >> =E2=80=A2 git: a4c04958f526 - main - libutil: support RLIMIT_PIPEBUF K= onstantin Belousov >> =E2=80=A2 git: 5d92f20c7d31 - main - bin/sh: support RLIMIT_PIPEBUF K= onstantin Belousov >> =E2=80=A2 git: f54f41403d14 - main - usr.bin/limits: support RLIMIT_P= IPEBUF Konstantin Belousov >> =E2=80=A2 git: b029e29e0d8b - main - login.conf: add a placeholder fo= r the pipebuf limit Konstantin Belousov >> =E2=80=A2 git: 80133d678ecb - main - procstat: support RLIMIT_PIPEBUF= Konstantin Belousov >> =E2=80=A2 git: 8ae779832c6f - main - privs: add PRIV_PIPEBUF Konstant= in Belousov >> =E2=80=A2 git: 7672cbef2c1e - main - pipes: reserve configured percen= tage of buffers zone to superuser Konstantin Belousov >> . . . >> =E2=80=A2 git: d6074f73af5c - main - pipe: use pipe subsystem KVA cou= nter instead of pipe_map size Konstantin Belousov >> =E2=80=A2 git: 40769168a5ee - main - pipespace_new(): decrease uidinf= o pipebuf usage if reservation check failed Konstantin Belousov >> . . . >> =E2=80=A2 git: a52b30ff98cd - main - sys_pipe: consistently use cr_ru= idinfo for accounting of pipebuf Konstantin Belousov >> =E2=80=A2 git: af96ccc6a508 - main - uifree(9): report non-zero value= s for all shared resources Konstantin Belousov >> =E2=80=A2 git: 2c1963d46335 - main - procfs rlimit: handle pipebuf Ko= nstantin Belousov >> =E2=80=A2 git: c84d8db0ab3d - main - procfs: ensure that RLIMIT_IDENT= is properly updated when a limit is added Konstantin Belousov >>=20 >> The combination of an older kernel and a newer world will not be >> nicely behaved when any non-kernel code from the above ends up >> involved. >>=20 >>=20 >> stable/14 has now also had the commits: >>=20 >> Sat, 05 Oct 2024 >> =E2=80=A2 git: 1508dce2502d - stable/14 - procfs: ensure that RLIMIT_= IDENT is properly updated when a limit is added Konstantin Belousov >> . . . >> =E2=80=A2 git: b7eecc86c3bd - stable/14 - kernel: add RLIMIT_PIPEBUF K= onstantin Belousov >> =E2=80=A2 git: d20f0dae2f97 - stable/14 - getrlimit(2): document RLIM= IT_PIPEBUF Konstantin Belousov >> =E2=80=A2 git: a03f7c040ce7 - stable/14 - libutil: support RLIMIT_PIP= EBUF Konstantin Belousov >> =E2=80=A2 git: d5ed8778bf3b - stable/14 - bin/sh: support RLIMIT_PIPE= BUF Konstantin Belousov >> =E2=80=A2 git: 25902860b270 - stable/14 - usr.bin/limits: support RLI= MIT_PIPEBUF Konstantin Belousov >> =E2=80=A2 git: 524b9810de6a - stable/14 - login.conf: add a placehold= er for the pipebuf limit Konstantin Belousov >> =E2=80=A2 git: 6600090e678e - stable/14 - procstat: support RLIMIT_PI= PEBUF Konstantin Belousov >> =E2=80=A2 git: fd9babb1b85f - stable/14 - privs: add PRIV_PIPEBUF Kon= stantin Belousov >> =E2=80=A2 git: d532d9926ee7 - stable/14 - pipes: reserve configured p= ercentage of buffers zone to superuser Konstantin Belousov >> =E2=80=A2 git: 6536b979b856 - stable/14 - pipe: use pipe subsystem KV= A counter instead of pipe_map size Konstantin Belousov >> =E2=80=A2 git: a8c663bb4261 - stable/14 - pipespace_new(): decrease u= idinfo pipebuf usage if reservation check failed Konstantin Belousov >> =E2=80=A2 git: c15b2e046e8c - stable/14 - sys_pipe: consistently use c= r_ruidinfo for accounting of pipebuf Konstantin Belousov >> . . . >> =E2=80=A2 git: fc9070bf1d16 - stable/14 - procfs rlimit: handle pipeb= uf Konstantin Belousov >> . . . >>=20 >> Again, the combination of an older kernel and a newer world will not be >> nicely behaved. >>=20 >> =3D=3D=3D >> Mark Millard >> marklmi at yahoo.com >>=20 >>=20 >>=20 > Thank you for this deep investigation. Anyway, after the kernel and world u= pdate, I could test the patch, but there was no significant progress, at lea= st in lookup timing (as I pointed out before). >=20 > -- > Marek Zarychta >=20 >=20 From nobody Mon Oct 7 14:07:08 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XMgvP4scqz5YBgS for ; Mon, 07 Oct 2024 14:07:13 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XMgvP1mRMz4Z4j for ; Mon, 7 Oct 2024 14:07:13 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qk1-x732.google.com with SMTP id af79cd13be357-7a9b72749bcso382829685a.0 for ; Mon, 07 Oct 2024 07:07:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728310032; x=1728914832; darn=freebsd.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :from:to:cc:subject:date:message-id:reply-to; bh=TdnejahIwdWqiDOsYkIGgBoICtEzokNbHClxF6+8bqY=; b=PwqbwVDwtaw9zfzkewub5P/GzyNwzcBXRaK+Z+gCfeSxYBpa6eP7PGhUf/fcvtrCfF yOcRSA+K0e05avi6qFCj31SEALC8bIAOSAT6bUecm3uHgh4vWpPA/CH9Gz3+8fMqeUei xBb0ez66zmortBTATYTb69IJwZzY/xmBcGRZr+TgvVzx753YRl1SEB1FTWK5iGs5yMxD P60Yqk6rnTEtw13/81O/Gee5Z7fSHZcmJhbya1Ixmel7erxwReQ+JSVn7/C7Zjqk33tE MVJ57ywu8H00ShkAhLlPd0wNyYd1nHFrd+ZlfFWHVdtUCGkZvdDVi6sLPYF1ssIRhOOt m/PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728310032; x=1728914832; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TdnejahIwdWqiDOsYkIGgBoICtEzokNbHClxF6+8bqY=; b=vBLOP2mUbcNR8LlpPuG8Oo/yqTUm2b6W0GsIDGx6Og1t9ie71w4ILgTKsBAqf3+ARe 9mcHVti7VdxqLle3/3PP482CWkt38gi1E4siSBawQ0mycdsTxnWcsY0tp4TVmo4OA9aY kBT9W9HZ78eWUFAOHWJyjGvxcxq97/6z19rh+o2qOkq5qBrCLtcMMe9BsevhxDu6DZWd cw9pSod54KJh7WfpCZ7NPOdLytFvNk1zYeSKGi1L+OtrKXYYY6gLKAMrxgLPVagM6C+P IHopzR/nH+aqOcSrS945TIyU3Sag2fT12LUbQbacxUOGhTlrn3YUrBmG6IahNl/CjIbC r5tg== X-Gm-Message-State: AOJu0Yxm70mp+Bbdpdh1T5nIbn4E1KOMUHZbbUPCN/mltylCi2dmJ8YD PRXeXPOSi3nbTzNsXEtLmBXmakLhpDxoJgLT71ZMYQb0BPcV38YTsyKBog== X-Google-Smtp-Source: AGHT+IHlkIGy+UGzGcoVXZaW+ujZpy6lcQZBGRT1QzlV6Y5VDSRQYRevbRAYtLVOwRhZL4s+xdXlsQ== X-Received: by 2002:a05:620a:4108:b0:7a9:bf2a:d7c8 with SMTP id af79cd13be357-7ae6f486015mr1809873185a.41.1728310031667; Mon, 07 Oct 2024 07:07:11 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7ae7562bdf6sm258668185a.43.2024.10.07.07.07.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Oct 2024 07:07:11 -0700 (PDT) Date: Mon, 7 Oct 2024 10:07:08 -0400 From: Mark Johnston To: "David E. Cross" Cc: FreeBSD Hackers Subject: Re: Review D38047 ... and then there was one.... Message-ID: References: <21941f7f-ce32-e277-a565-b1db3b3841ab@crossfamilyweb.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <21941f7f-ce32-e277-a565-b1db3b3841ab@crossfamilyweb.com> X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4XMgvP1mRMz4Z4j X-Spamd-Bar: ---- On Sun, Oct 06, 2024 at 02:35:18PM -0400, David E. Cross wrote: > As I have been prodding about open reviews, there is now only one that > hasn't had any action.  One that is complete and in main (OMGYES), and one > that is at the finish line (I think). > > > That leaves just D38047 ( https://reviews.freebsd.org/D38047 ) ... Submitted > Jan 13, **2023**, had initial peer review requests, I made all requested > changes (or explained why I didn't)  Made my last substantive changes Aug 4, > **2023**, I have since then semi-regularly updated with fresh rebases, but > no other changes. > > > Please, love to get some eyes on this.  As it stands nscd is completely > useless for LDAP for getgroupmembership (and really ANY implementation that > defines a specific implementation of getgroupmembership, since it will then > bypass the non-existent NSCD version).  Additionally it fixes bugs with > negative caching as well as increases thread safety. One problem with this patch is that it fixes several bugs at once. Though the process can be a bit tedious, it's much easier to review smaller, self-contained patches which fix a single problem. The change to the nscd Makefile plus the cycle prevention symbol rename, for instance, can be committed on its own. I left some comments on the review in any case. If you could split the patch into several smaller ones, it'd be greatly appreciated. > I've been running this successfully with LDAP for years now. From nobody Mon Oct 7 15:08:30 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XMjG96hwhz5YJrd for ; Mon, 07 Oct 2024 15:08:33 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta002.cacentral1.a.cloudfilter.net (omta002.cacentral1.a.cloudfilter.net [3.97.99.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XMjG945w9z4NBL; Mon, 7 Oct 2024 15:08:33 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; none Received: from shw-obgw-4002a.ext.cloudfilter.net ([10.228.9.250]) by cmsmtp with ESMTPS id xmhfsBU5TMArNxpLcsRFuV; Mon, 07 Oct 2024 15:08:32 +0000 Received: from spqr.komquats.com ([70.66.152.170]) by cmsmtp with ESMTPSA id xpLbsTNXS2M9qxpLcsuImy; Mon, 07 Oct 2024 15:08:32 +0000 X-Auth-User: cschuber X-Authority-Analysis: v=2.4 cv=ce5xrWDM c=1 sm=1 tr=0 ts=6703f970 a=y8EK/9tc/U6QY+pUhnbtgQ==:117 a=y8EK/9tc/U6QY+pUhnbtgQ==:17 a=IkcTkHD0fZMA:10 a=DAUX931o1VcA:10 a=6I5d2MoRAAAA:8 a=YxBL1-UpAAAA:8 a=EkcXrb_YAAAA:8 a=Rqjl16dpY3m5TfOLMpoA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id 9AC16254A; Mon, 07 Oct 2024 08:08:30 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id 939DC334; Mon, 07 Oct 2024 08:08:30 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Mark Johnston cc: "David E. Cross" , FreeBSD Hackers Subject: Re: Review D38047 ... and then there was one.... In-reply-to: References: <21941f7f-ce32-e277-a565-b1db3b3841ab@crossfamilyweb.com> Comments: In-reply-to Mark Johnston message dated "Mon, 07 Oct 2024 10:07:08 -0400." List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Date: Mon, 07 Oct 2024 08:08:30 -0700 Message-Id: <20241007150830.939DC334@slippy.cwsent.com> X-CMAE-Envelope: MS4xfD7VxpbJ0IPl/Z7CWCRXBHcNnu6tMbEoB2Fp7Wx0WuAOQBwR6Dn9QVt40P+pjdJrlV2tBEzRiymdmJWMZsj8pH52dPN+Yga5TjPL8ZdrqU89P9whozm1 HtQnQqUFyYiwnrc9xmxzW4JQUbpmzJLEME5uD1TyvTs3l1GO5PVv2GoNs2lleh6R1HeOwnnUYT8X2iU3I5kMy52DMDQaPzC9RvgikqBo9jHS6T08r7IbU3Rf 3LRsRIKpSfdHemNdF6YqygmfQ1QaF8KJOCJ74YqS55w= X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US] X-Rspamd-Queue-Id: 4XMjG945w9z4NBL X-Spamd-Bar: ---- In message , Mark Johnston writes: > On Sun, Oct 06, 2024 at 02:35:18PM -0400, David E. Cross wrote: > > As I have been prodding about open reviews, there is now only one that > > hasn't had any action.  One that is complete and in main (OMGYES), and one > > that is at the finish line (I think). > > > > > > That leaves just D38047 ( https://reviews.freebsd.org/D38047 ) ... Submitte > d > > Jan 13, **2023**, had initial peer review requests, I made all requested > > changes (or explained why I didn't)  Made my last substantive changes Aug 4 > , > > **2023**, I have since then semi-regularly updated with fresh rebases, but > > no other changes. > > > > > > Please, love to get some eyes on this.  As it stands nscd is completely > > useless for LDAP for getgroupmembership (and really ANY implementation that > > defines a specific implementation of getgroupmembership, since it will then > > bypass the non-existent NSCD version).  Additionally it fixes bugs with > > negative caching as well as increases thread safety. > > One problem with this patch is that it fixes several bugs at once. > Though the process can be a bit tedious, it's much easier to review > smaller, self-contained patches which fix a single problem. The change > to the nscd Makefile plus the cycle prevention symbol rename, for > instance, can be committed on its own. Yes. I was about to suggest this. Plus, any proposed commit log message must answer the questions why, what and how. With special attention to why. > > I left some comments on the review in any case. If you could split the > patch into several smaller ones, it'd be greatly appreciated. > > > I've been running this successfully with LDAP for years now. > -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0 From nobody Mon Oct 7 15:33:51 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XMjqY2vWKz5YLKC for ; Mon, 07 Oct 2024 15:34:01 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [IPv6:2001:678:618::40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "plan-b.pwste.edu.pl", Issuer "GEANT OV RSA CA 4" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XMjqX6HxVz4RTr for ; Mon, 7 Oct 2024 15:34:00 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Authentication-Results: mx1.freebsd.org; none Received: from [IPV6:2001:678:618:402f:c1d6:e548:55ce:baed] ([IPv6:2001:678:618:402f:c1d6:e548:55ce:baed]) (authenticated bits=0) by plan-b.pwste.edu.pl (8.18.1/8.17.2) with ESMTPSA id 497FXt96004518 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Mon, 7 Oct 2024 17:33:55 +0200 (CEST) (envelope-from zarychtam@plan-b.pwste.edu.pl) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=plan-b.pwste.edu.pl; s=plan-b-mailer; t=1728315235; bh=X1vBL4VGs0cc++qaJc7+tCvvQzRlWSXC7YIcnh7n8es=; h=Date:From:Subject:To:References:Cc:In-Reply-To; b=fDTEuV7WtBMxd6NymKIthe5+g42sB//LpFgw8wlMBkC6l0xL/fX+EATwwKtcXp+rN OH/fyiW3OQpVM0VgaYZXJPHtvgvtxOYSaRFan3vNpk1Kq15Zm7q8t85HUX6UVMDoXf Xch/2zB29jZGAihtg7OioCNmOtOqyx0aMAHLtDp1taIOXCXnF4ifrSggh1V1CG1U51 l70hbn1i4Ttt3y5V7A2ADndhFSJzGffAm+T5yAdo+3/7RHGk0ZZnq+iCJo6sPqX0h4 LEsaiuQx7NouokZ8QVVFf70TTTHeo7K53ctKDr+wVfnVPSnP8K7oqKvq3yAUR7Wh8U UUK2lWBOf2rLw== Content-Type: multipart/alternative; boundary="------------D1rFfEvTaTiecJtZmNdQ5XEB" Message-ID: Date: Mon, 7 Oct 2024 18:33:51 +0300 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Marek Zarychta Subject: Re: Review D38047 ... and then there was one.... To: David Cross References: <1fd47603-0bf2-4fcf-a556-22335d99e203@plan-b.pwste.edu.pl> Content-Language: en-US Cc: freebsd-hackers@freebsd.org In-Reply-To: X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:206006, ipnet:2001:678:618::/48, country:PL] X-Rspamd-Queue-Id: 4XMjqX6HxVz4RTr X-Spamd-Bar: ---- This is a multi-part message in MIME format. --------------D1rFfEvTaTiecJtZmNdQ5XEB Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit W dniu 7.10.2024 o 07:05, David Cross pisze: > How many entries are in your ldap structure? I can attempt a replication here Hello David, I will rather not expose it publicly. Whole LDAP directory contains few thousand entries - and it was was used for the tests mentioned in this thread. With the filters applied I see below 1k entries, and then lookup with nsdc running takes: first lookup 0.16s, next lookups 0.09s, while without nscd it varies from 0.12 to 0.08 - so nscd performs OK. I have your patch applied and I am still testing it with net/nss-pam-ldapd from ports with patch for login classes applied (it's present in port but not enabled by default). So far it works without issues. -- Marek Zarychta --------------D1rFfEvTaTiecJtZmNdQ5XEB Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

W dniu 7.10.2024 o 07:05, David Cross pisze: 

How many entries are in your ldap structure?  I can attempt a replication here

Hello David,

I will rather not expose it publicly. Whole LDAP directory contains few thousand entries - and it was was used for the tests mentioned in this thread.

With the filters applied I see below 1k entries, and then lookup with nsdc running takes: first lookup 0.16s, next lookups 0.09s, while without nscd it varies from 0.12 to 0.08 - so nscd performs OK.

I have your patch applied and I am still testing it with net/nss-pam-ldapd from ports with patch for login classes applied (it's present in port but not enabled by default). So far it works without issues.

-- 
Marek Zarychta
--------------D1rFfEvTaTiecJtZmNdQ5XEB-- From nobody Tue Oct 8 20:17:43 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XNS4q5B6Pz5YLVV for ; Tue, 08 Oct 2024 20:18:03 +0000 (UTC) (envelope-from david@crossfamilyweb.com) Received: from mail.dcrosstech.com (syn-024-097-005-251.biz.spectrum.com [24.97.5.251]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.dcrosstech.com", Issuer "DCrossTech.com LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XNS4p2C7Yz4QyQ for ; Tue, 8 Oct 2024 20:18:01 +0000 (UTC) (envelope-from david@crossfamilyweb.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of david@crossfamilyweb.com designates 24.97.5.251 as permitted sender) smtp.mailfrom=david@crossfamilyweb.com; dmarc=none X-Virus-Scanned: amavisd-new at dcrosstech.com Received: from [10.1.12.130] (d130.office.dcrosstech.com [10.1.12.130]) (authenticated bits=0) by mail.dcrosstech.com (8.15.2/8.15.2) with ESMTPSA id 498KHhUR069087 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Tue, 8 Oct 2024 20:17:45 GMT (envelope-from david@crossfamilyweb.com) X-Authentication-Warning: mail.priv.dcrosstech.com: Host d130.office.dcrosstech.com [10.1.12.130] claimed to be [10.1.12.130] Content-Type: multipart/alternative; boundary="------------pqdn9J0qCc3irc6FbzLakvWV" Message-ID: <553ea3d5-c94e-9c2f-c044-db7986625c74@crossfamilyweb.com> Date: Tue, 8 Oct 2024 16:17:43 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: Review D38047 ... and then there was one.... Content-Language: en-US To: Marek Zarychta Cc: freebsd-hackers@freebsd.org References: <1fd47603-0bf2-4fcf-a556-22335d99e203@plan-b.pwste.edu.pl> From: "David E. Cross" In-Reply-To: X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.80)[-0.804]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ONCE_RECEIVED(0.10)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEFALL_USER(0.00)[david]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:11351, ipnet:24.97.0.0/16, country:US]; RCPT_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[crossfamilyweb.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4XNS4p2C7Yz4QyQ X-Spamd-Bar: --- This is a multi-part message in MIME format. --------------pqdn9J0qCc3irc6FbzLakvWV Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Ok, I looked a bit into this and for the case of 'getent *' it really is not (currently) a fair comparison to speed. For 'getent password' the system currently works as follows, for each datasource in the list fully iterate over EVERY datasource, and 'cache' is a datasource, but so is ldap. What you wind up getting is a list of EVERYTHNG in files, then a list of everything in cache, and then a list of everything in LDAP. (or whatever).   SO every time it will always go back to origin, so caching effectively doesn't matter except to duplicate the data. I remember this when I was doing the initial development and I looked into ways to NOT have it do it but for some reason I didn't think it was possible without a substantial rewrite, I am taking another look to see if that is still true or if there is a way around it. Going on my vague (it has been multiple years now), I think in the GENERAL case it is unavoidable.  The way NSCD typically operates is that looked up values are PUSHED into the cache from the client.  That is the client says 'do you have X'? nscd replies 'no', then the CLIENT falls back, does the lookup, get the value and pushes it into nscd.  nscd additionally has a 'perform_lookups' flag that will have it do the lookup itself and then tell the client the result.  The interaction of this variable behavior is that there is no way to programatically shortcircuit it without libc knowing how nscd is optionally configured.  If libc knew that nscd would perform the lookups itself then it could for getent type calls just return immediately after the cache layer enumeration.  if libc knew that nscd would NOT perform lookups then it could bypass it and do the normal. I guess I could implement it as follows: nscd retruns NS_SUCCESS if it performs its own lookups and then in the case of getent NS_SUCCESS is treated as a return step for the cache layer only (since otherwise getent calls are treated as continue otherwise you'd never enumerate anything after files). and NS_NOTFOUND if it doesn't.. and then the libc layer would treat that as a continue.  .. I think that may do it... I need to refamiliarize myself with that code. In the meantime, checking basic lookups (not enumerations) is a more fair test.  Also keep in mind that without [notfound=return] that misses will always fall back to origin, which is probably what you want with nscd in the default configuration, but not with nscd doing its own lookups. On 10/7/24 11:33, Marek Zarychta wrote: > > W dniu 7.10.2024 o 07:05, David Cross pisze: > >> How many entries are in your ldap structure? I can attempt a replication here > > Hello David, > > I will rather not expose it publicly. Whole LDAP directory contains > few thousand entries - and it was was used for the tests mentioned in > this thread. > > With the filters applied I see below 1k entries, and then lookup with > nsdc running takes: first lookup 0.16s, next lookups 0.09s, while > without nscd it varies from 0.12 to 0.08 - so nscd performs OK. > > I have your patch applied and I am still testing it with > net/nss-pam-ldapd from ports with patch for login classes applied > (it's present in port but not enabled by default). So far it works > without issues. > > -- > Marek Zarychta --------------pqdn9J0qCc3irc6FbzLakvWV Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Ok, I looked a bit into this and for the case of 'getent *' it really is not (currently) a fair comparison to speed.

For 'getent password' the system currently works as follows, for each datasource in the list fully iterate over EVERY datasource, and 'cache' is a datasource, but so is ldap.

What you wind up getting is a list of EVERYTHNG in files, then a list of everything in cache, and then a list of everything in LDAP. (or whatever).   SO every time it will always go back to origin, so caching effectively doesn't matter except to duplicate the data.

I remember this when I was doing the initial development and I looked into ways to NOT have it do it but for some reason I didn't think it was possible without a substantial rewrite, I am taking another look to see if that is still true or if there is a way around it.

Going on my vague (it has been multiple years now), I think in the GENERAL case it is unavoidable.  The way NSCD typically operates is that looked up values are PUSHED into the cache from the client.  That is the client says 'do you have X'? nscd replies 'no', then the CLIENT falls back, does the lookup, get the value and pushes it into nscd.  nscd additionally has a 'perform_lookups' flag that will have it do the lookup itself and then tell the client the result.  The interaction of this variable behavior is that there is no way to programatically shortcircuit it without libc knowing how nscd is optionally configured.  If libc knew that nscd would perform the lookups itself then it could for getent type calls just return immediately after the cache layer enumeration.  if libc knew that nscd would NOT perform lookups then it could bypass it and do the normal.


I guess I could implement it as follows:

nscd retruns NS_SUCCESS if it performs its own lookups and then in the case of getent NS_SUCCESS is treated as a return step for the cache layer only (since otherwise getent calls are treated as continue otherwise you'd never enumerate anything after files). and NS_NOTFOUND if it doesn't.. and then the libc layer would treat that as a continue.  .. I think that may do it... I need to refamiliarize myself with that code.


In the meantime, checking basic lookups (not enumerations) is a more fair test.  Also keep in mind that without [notfound=return] that misses will always fall back to origin, which is probably what you want with nscd in the default configuration, but not with nscd doing its own lookups.

On 10/7/24 11:33, Marek Zarychta wrote:

W dniu 7.10.2024 o 07:05, David Cross pisze: 

How many entries are in your ldap structure?  I can attempt a replication here

Hello David,

I will rather not expose it publicly. Whole LDAP directory contains few thousand entries - and it was was used for the tests mentioned in this thread.

With the filters applied I see below 1k entries, and then lookup with nsdc running takes: first lookup 0.16s, next lookups 0.09s, while without nscd it varies from 0.12 to 0.08 - so nscd performs OK.

I have your patch applied and I am still testing it with net/nss-pam-ldapd from ports with patch for login classes applied (it's present in port but not enabled by default). So far it works without issues.

-- 
Marek Zarychta
--------------pqdn9J0qCc3irc6FbzLakvWV-- From nobody Wed Oct 9 16:12:50 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XNz6q41Jyz5YkWx for ; Wed, 09 Oct 2024 16:36:35 +0000 (UTC) (envelope-from yuri@FreeBSD.org) Received: from shell1.rawbw.com (shell1.rawbw.com [198.144.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id 4XNz6p1SRYz3ypW for ; Wed, 9 Oct 2024 16:36:34 +0000 (UTC) (envelope-from yuri@FreeBSD.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 198.144.192.42 is neither permitted nor denied by domain of yuri@FreeBSD.org) smtp.mailfrom=yuri@FreeBSD.org; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none) Received: from [192.168.5.3] (c-98-42-44-116.hsd1.ca.comcast.net [98.42.44.116]) (authenticated bits=0) by shell1.rawbw.com (8.15.1/8.15.1) with ESMTPSA id 499GCqvn072462 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 9 Oct 2024 09:12:52 -0700 (PDT) (envelope-from yuri@FreeBSD.org) X-Authentication-Warning: shell1.rawbw.com: Host c-98-42-44-116.hsd1.ca.comcast.net [98.42.44.116] claimed to be [192.168.5.3] Message-ID: <2a759f5e-a9fb-4ad3-ac88-d75e81f5140c@FreeBSD.org> Date: Wed, 9 Oct 2024 09:12:50 -0700 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Freebsd hackers list From: Yuri Subject: Why is the process gets killed because "a thread waited too long to allocate a page"? Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [1.73 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.99)[-0.985]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; RCVD_NO_TLS_LAST(0.10)[]; XM_UA_NO_VERSION(0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; ARC_NA(0.00)[]; GREYLIST(0.00)[pass,body]; RCPT_COUNT_ONE(0.00)[1]; FREEFALL_USER(0.00)[yuri]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; R_SPF_SOFTFAIL(0.00)[~all]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:7961, ipnet:198.144.192.0/23, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org]; TO_DOM_EQ_FROM_DOM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; HAS_XAW(0.00)[] X-Rspamd-Queue-Id: 4XNz6p1SRYz3ypW X-Spamd-Bar: + Hi, When I tried to build lang/rust in the 14i386 poudriere VM the compiler got killed with this message in the kernel log: > Oct  9 05:21:11 yv kernel: pid 35188 (rustc), jid 1129, uid 65534, was killed: a thread waited too long to allocate a page The same system has no problem building lang/rust in the 14amd64 VM. What does it mean "waited too long"? Why is the process killed when something is slow? Shouldn't it just wait instead? Thank you, Yuri From nobody Wed Oct 9 17:53:29 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XP0qr4k4Kz5YpVQ for ; Wed, 09 Oct 2024 17:53:44 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XP0qr2rVTz472h; Wed, 9 Oct 2024 17:53:44 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-537a399e06dso11213e87.1; Wed, 09 Oct 2024 10:53:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728496422; x=1729101222; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=49Vz6f2NDHH25TUDdNQ34r0EUnyq053D/oIAHuu5tB4=; b=U27ORGRM2AaFC8GpYwEJyj/iecqKXQRZfb92U3EpdurV+QoGBX00OGAZ0+gKlURaxB 5JeFRNLU8k8NLEPpeZUsRLrMxyHGzPUWgObniTH9VxKlOSukOmfuKCnGr9n+gnjtaEUd WhizKCVi0wfEEvYYqb/eW99MyoC15vaynO4k7YxVfKdF/Frkmsh6v2nxsymrA39zYKCa y9EZvlgrzH6gQz2nSq+WLGlxLniO13xoveHQrJcu9tLMHd2Qp4glKnv214n/HZB4geRE 1ycwCsx3gSm3pwDeiTid2cY2QFcf+aA/Rb9mOnZ8sUluyvrhBIzDphRm/3l4SNBi0oTk R89g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728496422; x=1729101222; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=49Vz6f2NDHH25TUDdNQ34r0EUnyq053D/oIAHuu5tB4=; b=BqraCuLTeAIw99Xws/9JaiZqrgJojCHedsiLm/agoX3GIfBswOEpApCeTvhRt9vwJ0 HN3MZLokZWD8vsV5h+i968glqgG7Fkce2iFqmcgYfzLxPcVLQBKz1Ag6HBdtE4X1TKwO iADSaSKnUA9DlUUXY7G30pAWCjr/avXggf6Cykcl5FZQ8yaQ+TL4jb4RI0j8AfeZ9f4G kcF9zEoJWlcQVFEp45L4BJIyT66S10GRpwYbA2z4n0qN+nm/Lc+SQ9udYaqn7F1SRVd+ fZ/Q1wqzVig88m/ydN4Wkhvq9BHwl2KzWBBx7twInyCRe5N7InVodCo5ECfoxFF35xpR L54A== X-Gm-Message-State: AOJu0Yzcb3Mw91nOIv1TP1ooY5j3VhU4xQ3VJcex+KmFjWZ4rXggOSJP gVptR8k7zjnbPfcI2U+emrvoywz4/8b14mirPQNJ3RNbY4VVQ2hbkicSAewVdYGIAujOKEpDX97 4wAOrWxaHg72SMGGudVzWXqIIB2SpGg== X-Google-Smtp-Source: AGHT+IEvMC8NPLgYZTV8wfx9rgfICYrzptgs9cemT1SNvjAeTwGyc7reULfogV/81hHmcyfeh2mGof6MeiJbJzsVy1U= X-Received: by 2002:a05:6512:b8d:b0:52e:7656:a0f4 with SMTP id 2adb3069b0e04-539c4946004mr1997925e87.41.1728496421625; Wed, 09 Oct 2024 10:53:41 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: <2a759f5e-a9fb-4ad3-ac88-d75e81f5140c@FreeBSD.org> In-Reply-To: <2a759f5e-a9fb-4ad3-ac88-d75e81f5140c@FreeBSD.org> From: =?UTF-8?Q?Fernando_Apestegu=C3=ADa?= Date: Wed, 9 Oct 2024 19:53:29 +0200 Message-ID: Subject: Re: Why is the process gets killed because "a thread waited too long to allocate a page"? To: Yuri Cc: Freebsd hackers list Content-Type: multipart/alternative; boundary="00000000000034e6e206240eef75" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4XP0qr2rVTz472h X-Spamd-Bar: ---- --00000000000034e6e206240eef75 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable El mi=C3=A9, 9 oct 2024, 18:36, Yuri escribi=C3=B3: > Hi, > > > When I tried to build lang/rust in the 14i386 poudriere VM the compiler > got killed with this message in the kernel log: > > > > Oct 9 05:21:11 yv kernel: pid 35188 (rustc), jid 1129, uid 65534, > was killed: a thread waited too long to allocate a page > > > > The same system has no problem building lang/rust in the 14amd64 VM. > > > What does it mean "waited too long"? Why is the process killed when > something is slow? > Shouldn't it just wait instead? > Did you run out of memory? It tends to happen with rust especially depending on how many things you put in TMPFS in poudri=C3=A8re. > > > Thank you, > > Yuri > > > > --00000000000034e6e206240eef75 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


El mi=C3=A9, 9 oct 2024, 18:36, Yuri <yuri@freebsd.org> escribi=C3=B3:
Hi,


When I tried to build lang/rust in the 14i386 poudriere VM the compiler got killed with this message in the kernel log:


=C2=A0> Oct =C2=A09 05:21:11 yv kernel: pid 35188 (rustc), jid 1129, uid= 65534,
was killed: a thread waited too long to allocate a page



The same system has no problem building lang/rust in the 14amd64 VM.


What does it mean "waited too long"? Why is the process killed wh= en
something is slow?
Shouldn't it just wait instead?

Did you run out of memory? It tends to h= appen with rust especially depending on how many things you put in TMPFS in= poudri=C3=A8re.




Thank you,

Yuri



--00000000000034e6e206240eef75-- From nobody Thu Oct 10 02:21:02 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XPD5W4H9Sz5ZL1N for ; Thu, 10 Oct 2024 02:21:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-55.consmr.mail.gq1.yahoo.com (sonic315-55.consmr.mail.gq1.yahoo.com [98.137.65.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XPD5V4l4Wz46Cv for ; Thu, 10 Oct 2024 02:21:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=FoFguIbg; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1728526876; bh=n/JKTNU2va/lirDDPcckg6m/ugbhNRHtANM0xDYPESg=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=FoFguIbgSum9ifWKLpBdXTyn0SjrhL0Lzc34svnfzq7h+X+ZXw8SaYxRu08OwCZWgfS+0JWB/vobwb2X7g50gkcMULxd3eqDcIgbw2GLQqUFx/dHwTp5NqAaVbZwtyQCQ+EBYafujEGdmCF2y1G98jsRJG5fj05POfpzx1+TnmmvYbOxQ2HSAlLDKKBYhlgiIT8uY1kHliKeWVmSMSHwAzvJMaPPqsx1hnJfx/wEt3Xkjj1aVXbqZGjaBdVsqT8IrwkS+ZMtLdem1rRfk4b26srN533NYYPWGSYixJPjORtetVefVB4CyV6muHUHlZJdsJa277jhcC9qhOfz9+E2Mg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1728526876; bh=dAmc0oGcAfsVUXAoTqhDAUH/z5q1yGt9qVadabsboba=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=CggV1SMPAwePfXfPvcdKlO/IFzvFN/TCDpqF7EoqypUMjjo2p17+7MnYxvaNsCIU5crGYq0ayZbDb2Sjoa+W3+ZnGNEZjVS6scBMAjDS95iXXG3Gc7kn5fG0V82HGmi13PLOAMKG9xaPoTbXq6A7l0dxgDVvTcqzsC1M7B9K1Lj+i0xsaICPlmadMw26GcP9RArl2RzRf54vvVgOFNMzobipX+YHJP2P51/EkSi0kiohjrwnLQcXHH4Agwz6C7v6Pb2Lp1opzIFnYcG8QGO5HnubrS8/Kr1fqipSmmW6plvOAWYtEkRl6ycaBsMqlyI2+v1LHMic7dJuOAsXFyqUlg== X-YMail-OSG: RCSgjnAVM1kTtmG9VtCFD7Nr6At6dljwM.pJZ9vSDuYHyrXg5Kkq9OqGKCvVGvH JzJaQUm8FxzNIpNbNFHp9GBgqf_RhhsmO1TlhgdG9efvcRQ_u5t8t7au11TbvSERAiVebgPeKikk HsQAUGjGP2wSLnpfLJnZSVieS7ftj8cEYmvA5Nj0QORQSKLORkv1ITbfszrrmafvqy6fyt_areiT 2XIqTiRsIjxPHO1CeJVcOmLlPCguvYs0lNf_EtNGwyD3SpbbU9oC5WDuGwWben_P5vCvhcn_VbtA EGYuV7ZxOCMKKyJ0JRVEmpDfEbmPu9.NlW.K1t_yApkP1VZ6YSZf3O4XOurqRnlQNAMn5ixy6CRx IyWDVH48ehnb.XSw_ylfDUKSfwCeA30USKrys82LrLHfln4DHww9e4dHadbVqfCkezF9.Z7A_yES n2PCrBwA460ayozGt2kq5RZXVRACMtF0L6urMLB_Zd2Yw7kE7SLna03LsiHovKfIXAwWRZuiQSnu GVkhB4M3Zywkj6yKamq.Chbw7yiSh4Dt_nk7N0QhEIzLqEziyOSdA_K.lXOQnlUCJlJ52luxp89N fTMjTLIN6QH1EspKnNFKacG6j35bI7x0xrqhTw_RWpQFt9922gL.5bdrmHD1lIphNXECwp4H3p46 vGNBuKaznijB_qCV4QuoYw0miMSUrzC5HcW_FY1MIUyoMGReqW0k6ZZTLxWdg2iIICBqVBDeE8Os tnXOn1UjjvldTsTxSCkvhSIXhuz8xgnLuAi_9NNf.3BQf7qzlnTu_AQ5AAfTnck_Co8l5v7gWasI YrJn3_U4f7_x39IoP.MUmxYzsWVZ.RhxCFie3bsqZp1g..X2nTkK08m8PDG0Qham2LJf5Xk.2xi9 utpzeD.Q62Qqw6ebR1.SeZ3R4Ingh7YzPB3cFkC6jUuiZpCEaJCY7an8NGO_EZuB4NmtS6fC7Cjb CWTjl9MDymRvm9.IiG2kpAurJpd7UMdaF1u3nMWKnxVyLTqxqUVXgVP0mGqDOK4XcDiwOXM.q6A0 Y1rDHzjvauGVJymyu8u.Y8zNvUrJCaEd4w8OZlZTBy_gcxWdckluOmiWN1f18CxiHDtAZQBj2691 aCu2gyykj30Gf6nZoYifiZrHv.K4f5R2Gb7zLZlctgLnwPBG826A80ZMUGM0g20bsTv47ZfIhL87 iZ8ajA5QCewbzVKrm2XNemGneSMRfIU640muXNpc6WOPXzSOHHM_hCSH.im69ZSeVGqG3E24A6or N0NBaIVe7lrwzi8I2IBrxh_lmHHujCfNd9VUfIhERBE_tHato5b0e6LObOylf8DRf14Z12JGvfAF 2sBWrYApRBZCZAcr5Ss1JogUvu6rzkgYBEU0ir4WsjZAER7T5Fhg44yJDgXtaCIcGSFx7FvUviGF gM7N6H2d01YdY_THDT8azaPFTVGIgXmM7y4hZi6Y1krbj7iae5hXFzkfHNICumaFqDW1qCp7TJk2 A3e7ci7nv4L1wx5JsEupJ8im0T_1GVgKhPyi.RNRgF7eD33vXDIcxj.IJfzYkiY5jQdHm0NZG.ls sugdCEsJd4rPQ6fuJh8zUUqI4J5dGLk.80G5a1JGzR6qw_.B.yYZTjBhB_7nTVgqVjrBjuHI4Z8A TGFWq1LLe.FIykApCEk9n4hUPnAN2_RaGBcq7ZqZEN6GqksENY_yu2XpgHR.kXla8iW3TE2iGIps kmSAvCg_.9bhDTV_UI7nzOu9mg9jOL0XsHORrStQxzUIebyMkL2xrD9q58etIElKBQP_TQcGS_6R inxULm7EJ1TWN7V3nzOQteoYXsOvcvMOWFClvJPytDGbne1B8RvJz0b7rKJLoeWdGUCzym5giHaH LVi4_O.ACQ12_8JNmPeCjoFIBGZTF4AxFw2k4WXap7V7qhmEM.MKtBiAfoNoeSpsWSw9f7DpcWhA tE7zz5g76MLS2gDuaN35kRci1ANINil4Nmu7I4ejTWxdYcGJ71urB.a_3OgGtg0VCnBe0U.Xpxz3 a7c.tXPXC.VOInZ6U9nK11NWWXS5YAmuI3j9hf6eIjzoKCYv4aWVc5MbJ.MNWq3Obd.yZwOHgBaW 34H0uryUTYORM2DLF_Gizx6izbKat3WYDcuDGpGDWtKHZ.eubLlEqtN8W66.LcL_rXa5yaaE__kF rLiAkProATu_uPX8BkT2jBSLr7GA5ifDG9_PxR7jTfaAb7rozaeMgPUxmFg3r5ew_zXda2d1bjSt 6Xir7gKXZEnYYc3qGrPl4HU2fycL8b2DfD6lJuQ-- X-Sonic-MF: X-Sonic-ID: ceb2e492-4ef6-47fa-928f-52bf0e7db5ac Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Thu, 10 Oct 2024 02:21:16 +0000 Received: by hermes--production-gq1-5d95dc458-sd55t (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7969551c646ab585c17a83abecaba769; Thu, 10 Oct 2024 02:21:13 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: RE: Why is the process gets killed because "a thread waited too long to allocate a page"? Message-Id: Date: Wed, 9 Oct 2024 19:21:02 -0700 To: Yuri , freebsd-hackers X-Mailer: Apple Mail (2.3776.700.51) References: X-Spamd-Result: default: False [-1.46 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.54)[0.539]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MID_RHS_MATCH_FROM(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.31:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.31:from] X-Rspamd-Queue-Id: 4XPD5V4l4Wz46Cv X-Spamd-Bar: - Yuri wrote on Date: Wed, 09 Oct 2024 16:12:50 UTC : > When I tried to build lang/rust in the 14i386 poudriere VM the = compiler=20 > got killed with this message in the kernel log: >=20 >=20 > > Oct 9 05:21:11 yv kernel: pid 35188 (rustc), jid 1129, uid 65534,=20= > was killed: a thread waited too long to allocate a page >=20 >=20 >=20 > The same system has no problem building lang/rust in the 14amd64 VM. >=20 >=20 > What does it mean "waited too long"? Why is the process killed when=20 > something is slow? > Shouldn't it just wait instead? If you want to allow it to potentially wait forever, you can use: sysctl vm.pfault_oom_attempts=3D-1 (or analogous in appropriate *.conf files taht would later be executed). You might end up with deadlock/livelock/. . . if you do so. (I've not analyzed the details.) Details: Looking around, sys/vm/vm_pageout.c has: case VM_OOM_MEM_PF: reason =3D "a thread waited too long to allocate = a page"; break; # grep -r VM_OOM_MEM_PF /usr/main-src/sys/ /usr/main-src/sys/vm/vm_pageout.h:#define VM_OOM_MEM_PF 2 /usr/main-src/sys/vm/vm_fault.c: vm_pageout_oom(VM_OOM_MEM_PF); /usr/main-src/sys/vm/vm_pageout.c: if (shortage =3D=3D VM_OOM_MEM_PF && /usr/main-src/sys/vm/vm_pageout.c: if (shortage =3D=3D VM_OOM_MEM || = shortage =3D=3D VM_OOM_MEM_PF) /usr/main-src/sys/vm/vm_pageout.c: case VM_OOM_MEM_PF: sys/vm/vm_fault.c : (NOTE: official code has its variant of the printf under a "if (bootverbose)" but I locally remove that conditional.) /* * Initiate page fault after timeout. Returns true if caller should * do vm_waitpfault() after the call. */ static bool vm_fault_allocate_oom(struct faultstate *fs) { struct timeval now; =20 vm_fault_unlock_and_deallocate(fs); if (vm_pfault_oom_attempts < 0) return (true); if (!fs->oom_started) { fs->oom_started =3D true; getmicrotime(&fs->oom_start_time); return (true); } =20 getmicrotime(&now); timevalsub(&now, &fs->oom_start_time); if (now.tv_sec < vm_pfault_oom_attempts * vm_pfault_oom_wait) return (true); =20 printf("vm_fault_allocate_oom: proc %d (%s) failed to alloc page = on fault, starting OOM\n", curproc->p_pid, curproc->p_comm); =20 vm_pageout_oom(VM_OOM_MEM_PF); fs->oom_started =3D false; return (false); } This is associated with vm.pfault_oom_attempts and vm.pfault_oom_wait . An old comment in my /boot/loader.conf is: # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes (showing defaults at the time): #vm.pfault_oom_attempts=3D 3 #vm.pfault_oom_wait=3D 10 # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) (Note: the "tradeoffs" is associated with: sys/vm/vm_fault.c: vm_waitpfault(dset, vm_pfault_oom_wait * hz); ) sys/vm/vm_pageout.c : void vm_pageout_oom(int shortage) { const char *reason; struct proc *p, *bigproc; vm_offset_t size, bigsize; struct thread *td; struct vmspace *vm; int now; bool breakout; /* * For OOM requests originating from vm_fault(), there is a high * chance that a single large process faults simultaneously in * several threads. Also, on an active system running many * processes of middle-size, like buildworld, all of them * could fault almost simultaneously as well. * * To avoid killing too many processes, rate-limit OOMs * initiated by vm_fault() time-outs on the waits for free * pages. */ mtx_lock(&vm_oom_ratelim_mtx); now =3D ticks; if (shortage =3D=3D VM_OOM_MEM_PF && (u_int)(now - vm_oom_ratelim_last) < hz * vm_oom_pf_secs) { mtx_unlock(&vm_oom_ratelim_mtx); return; } vm_oom_ratelim_last =3D now; mtx_unlock(&vm_oom_ratelim_mtx); . . . size =3D vmspace_swap_count(vm); if (shortage =3D=3D VM_OOM_MEM || shortage =3D=3D = VM_OOM_MEM_PF) size +=3D vm_pageout_oom_pagecount(vm); . . . Looks like time based retries and giving up after about the specified overall time for that many retries, avoiding potentially waiting forever when 0 <=3D vm.pfault_oom_attempts . =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Oct 10 22:23:53 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XPknG0ndDz5Yksc for ; Thu, 10 Oct 2024 22:24:02 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [IPv6:2a00:1450:4864:20::234]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XPknF0Y6Bz4lhQ for ; Thu, 10 Oct 2024 22:24:01 +0000 (UTC) (envelope-from vadimnuclight@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=O4sp7Zrk; spf=pass (mx1.freebsd.org: domain of vadimnuclight@gmail.com designates 2a00:1450:4864:20::234 as permitted sender) smtp.mailfrom=vadimnuclight@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x234.google.com with SMTP id 38308e7fff4ca-2fb2e21b631so3891781fa.0 for ; Thu, 10 Oct 2024 15:24:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728599038; x=1729203838; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=HO5UFR7Av1lM/tHrEgFid5gJGAKLwVGttB0YlxJXSTg=; b=O4sp7ZrkHHtu0I5kfjs/vcpCSZ/iaYWBrPv31xrItmVbS4M7g2wmtWurwEMbNR+bJi nW2jWN3jrhCXu8nBoseHDOMTY7kF1THil7ZjWTjKhJKNBgkFL+a/NOuXoJM5LfLfppxJ k8Oi9DvLBWKS1MbHfzJWTq2BGPQlBcuyjh2dicmZiTEUIxnldDHRFNXn2C8TzXTHC7Nj XLAoR7neORukFkjSN4DStPnynxPTdBRaWSOg5fXBWaeDmCUUdw0jvRzP3qXDaQda3I3C jXsxAoEh/s+oKukh2cydbfZjCMcIhEwMMtsLcCX+rfOpte+TCizIfnv/UPMbdFxNpeIG 7x0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728599038; x=1729203838; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HO5UFR7Av1lM/tHrEgFid5gJGAKLwVGttB0YlxJXSTg=; b=ofYzWW/O6y6lqCgeplpCtS2+PhNaorfwyT96qaqEZe9IB+/hJCLynktEdEGCBPHvsI 70DFXjMzINp9/8sORCx3OiMRg6szgTa/t/x2fF5w7RHk80txN7f01tjFhKfR/1vktX6Z GwkqeU1uPh7g7pe7yaH3kDzN+5jx+RKa5lpCUc/Dd/8ixCDuqjdOtBZklIVHH3g2uu4i 34pyrQ90YMuzuPaSjK5SnF0YC2iu2B1RE/bXcp3nkepPuUII7xtlHeT99qGU2nz5Dkg8 jNyu4h02ga4iFSPWpFiZP5TAWScJbYqMjD3IegbX3j2/I+F5SphRvALLtIJlinRirEOz 1vpw== X-Gm-Message-State: AOJu0YwVPWJHfnazFMz29qunwtHn6gbtKEa1MIcsPM2A41lDs+bFETAN ZzX9pcPZribTvrtdcX0Hr7klF+xJ32X2GWac+Q1uE1jGxCQ0LWddLhFvM7Xp X-Google-Smtp-Source: AGHT+IF4sITiOShFK+7FjuGBKlL8HLvYW7TF8/0P9LfTSBcKDHinbhP2eppXDvOIamLVfzDbHuvD+w== X-Received: by 2002:a2e:6119:0:b0:2fa:faed:e86b with SMTP id 38308e7fff4ca-2fb30dfab1bmr1482871fa.13.1728599037794; Thu, 10 Oct 2024 15:23:57 -0700 (PDT) Received: from nuclight.lan ([37.204.254.214]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2fb24706343sm3402321fa.77.2024.10.10.15.23.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Oct 2024 15:23:57 -0700 (PDT) Date: Fri, 11 Oct 2024 01:23:53 +0300 From: Vadim Goncharov To: "Gavin D. Howard" Cc: "freebsd-hackers@FreeBSD.org" Subject: Re: BPF64: proposal of platform-independent hardware-friendly backwards-compatible eBPF alternative Message-ID: <20241011012353.59784a96@nuclight.lan> In-Reply-To: References: <20240910040544.125245ad@nuclight.lan> <20240910181711.5d324ac5@nuclight.lan> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd12.4) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::234:from] X-Rspamd-Queue-Id: 4XPknF0Y6Bz4lhQ X-Spamd-Bar: --- On Thu, 12 Sep 2024 06:29:38 +0000 "Gavin D. Howard" wrote: > > > If I understand Turing-completeness correctly, the "no backward > > > jumps but allow recursion and quit on stack overflow" is exactly > > > equivalent to having non-infinite loops. > > > > Sure, but then look at practical usefulness: suppose you have stack > > of 32 frames (current limit of eBPF and BPF64). Then you can have > > only 31 iterations on a loop, loosing ability to call more > > functions from loop body. > > That is true. > > However, I wonder if everyone is going at this the wrong way. More > specifically, I wonder if we are targeting the entirely wrong level. > Maybe verifying bytecode or some other low-level code form is just too > hard. What if it were easier just to provide a higher level language > that had enough restrictions to be just barely not Turing-complete. The problem is, then you need compiler of this language in target environment, e.g. kernel, because you cannot trust that compiler's output was not modified in a circumventing way. > To test that idea, I did a quick experiment. > > I have been working on a language called Yao for the past three years. > One feature I planned for the future was the ability to restrict what > packages and keywords are available at compile time. I had already > put in the plumbing, but I just hadn't implemented it. > > But I decided to quickly implement it as an experiment. > > First, I needed a stack limit: > > https://git.yzena.com/Yzena/Yc/commit/99c822bf7b > > Now, you can run this: > > ``` > $ ./release/yc yao --max-stack-depth=32