From nobody Tue Oct 8 20:17:43 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XNS4q5B6Pz5YLVV for ; Tue, 08 Oct 2024 20:18:03 +0000 (UTC) (envelope-from david@crossfamilyweb.com) Received: from mail.dcrosstech.com (syn-024-097-005-251.biz.spectrum.com [24.97.5.251]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.dcrosstech.com", Issuer "DCrossTech.com LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XNS4p2C7Yz4QyQ for ; Tue, 8 Oct 2024 20:18:01 +0000 (UTC) (envelope-from david@crossfamilyweb.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of david@crossfamilyweb.com designates 24.97.5.251 as permitted sender) smtp.mailfrom=david@crossfamilyweb.com; dmarc=none X-Virus-Scanned: amavisd-new at dcrosstech.com Received: from [10.1.12.130] (d130.office.dcrosstech.com [10.1.12.130]) (authenticated bits=0) by mail.dcrosstech.com (8.15.2/8.15.2) with ESMTPSA id 498KHhUR069087 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Tue, 8 Oct 2024 20:17:45 GMT (envelope-from david@crossfamilyweb.com) X-Authentication-Warning: mail.priv.dcrosstech.com: Host d130.office.dcrosstech.com [10.1.12.130] claimed to be [10.1.12.130] Content-Type: multipart/alternative; boundary="------------pqdn9J0qCc3irc6FbzLakvWV" Message-ID: <553ea3d5-c94e-9c2f-c044-db7986625c74@crossfamilyweb.com> Date: Tue, 8 Oct 2024 16:17:43 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: Review D38047 ... and then there was one.... Content-Language: en-US To: Marek Zarychta Cc: freebsd-hackers@freebsd.org References: <1fd47603-0bf2-4fcf-a556-22335d99e203@plan-b.pwste.edu.pl> From: "David E. Cross" In-Reply-To: X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.80)[-0.804]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ONCE_RECEIVED(0.10)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEFALL_USER(0.00)[david]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:11351, ipnet:24.97.0.0/16, country:US]; RCPT_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[crossfamilyweb.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4XNS4p2C7Yz4QyQ X-Spamd-Bar: --- This is a multi-part message in MIME format. --------------pqdn9J0qCc3irc6FbzLakvWV Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Ok, I looked a bit into this and for the case of 'getent *' it really is not (currently) a fair comparison to speed. For 'getent password' the system currently works as follows, for each datasource in the list fully iterate over EVERY datasource, and 'cache' is a datasource, but so is ldap. What you wind up getting is a list of EVERYTHNG in files, then a list of everything in cache, and then a list of everything in LDAP. (or whatever).   SO every time it will always go back to origin, so caching effectively doesn't matter except to duplicate the data. I remember this when I was doing the initial development and I looked into ways to NOT have it do it but for some reason I didn't think it was possible without a substantial rewrite, I am taking another look to see if that is still true or if there is a way around it. Going on my vague (it has been multiple years now), I think in the GENERAL case it is unavoidable.  The way NSCD typically operates is that looked up values are PUSHED into the cache from the client.  That is the client says 'do you have X'? nscd replies 'no', then the CLIENT falls back, does the lookup, get the value and pushes it into nscd.  nscd additionally has a 'perform_lookups' flag that will have it do the lookup itself and then tell the client the result.  The interaction of this variable behavior is that there is no way to programatically shortcircuit it without libc knowing how nscd is optionally configured.  If libc knew that nscd would perform the lookups itself then it could for getent type calls just return immediately after the cache layer enumeration.  if libc knew that nscd would NOT perform lookups then it could bypass it and do the normal. I guess I could implement it as follows: nscd retruns NS_SUCCESS if it performs its own lookups and then in the case of getent NS_SUCCESS is treated as a return step for the cache layer only (since otherwise getent calls are treated as continue otherwise you'd never enumerate anything after files). and NS_NOTFOUND if it doesn't.. and then the libc layer would treat that as a continue.  .. I think that may do it... I need to refamiliarize myself with that code. In the meantime, checking basic lookups (not enumerations) is a more fair test.  Also keep in mind that without [notfound=return] that misses will always fall back to origin, which is probably what you want with nscd in the default configuration, but not with nscd doing its own lookups. On 10/7/24 11:33, Marek Zarychta wrote: > > W dniu 7.10.2024 o 07:05, David Cross pisze: > >> How many entries are in your ldap structure? I can attempt a replication here > > Hello David, > > I will rather not expose it publicly. Whole LDAP directory contains > few thousand entries - and it was was used for the tests mentioned in > this thread. > > With the filters applied I see below 1k entries, and then lookup with > nsdc running takes: first lookup 0.16s, next lookups 0.09s, while > without nscd it varies from 0.12 to 0.08 - so nscd performs OK. > > I have your patch applied and I am still testing it with > net/nss-pam-ldapd from ports with patch for login classes applied > (it's present in port but not enabled by default). So far it works > without issues. > > -- > Marek Zarychta --------------pqdn9J0qCc3irc6FbzLakvWV Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Ok, I looked a bit into this and for the case of 'getent *' it really is not (currently) a fair comparison to speed.

For 'getent password' the system currently works as follows, for each datasource in the list fully iterate over EVERY datasource, and 'cache' is a datasource, but so is ldap.

What you wind up getting is a list of EVERYTHNG in files, then a list of everything in cache, and then a list of everything in LDAP. (or whatever).   SO every time it will always go back to origin, so caching effectively doesn't matter except to duplicate the data.

I remember this when I was doing the initial development and I looked into ways to NOT have it do it but for some reason I didn't think it was possible without a substantial rewrite, I am taking another look to see if that is still true or if there is a way around it.

Going on my vague (it has been multiple years now), I think in the GENERAL case it is unavoidable.  The way NSCD typically operates is that looked up values are PUSHED into the cache from the client.  That is the client says 'do you have X'? nscd replies 'no', then the CLIENT falls back, does the lookup, get the value and pushes it into nscd.  nscd additionally has a 'perform_lookups' flag that will have it do the lookup itself and then tell the client the result.  The interaction of this variable behavior is that there is no way to programatically shortcircuit it without libc knowing how nscd is optionally configured.  If libc knew that nscd would perform the lookups itself then it could for getent type calls just return immediately after the cache layer enumeration.  if libc knew that nscd would NOT perform lookups then it could bypass it and do the normal.


I guess I could implement it as follows:

nscd retruns NS_SUCCESS if it performs its own lookups and then in the case of getent NS_SUCCESS is treated as a return step for the cache layer only (since otherwise getent calls are treated as continue otherwise you'd never enumerate anything after files). and NS_NOTFOUND if it doesn't.. and then the libc layer would treat that as a continue.  .. I think that may do it... I need to refamiliarize myself with that code.


In the meantime, checking basic lookups (not enumerations) is a more fair test.  Also keep in mind that without [notfound=return] that misses will always fall back to origin, which is probably what you want with nscd in the default configuration, but not with nscd doing its own lookups.

On 10/7/24 11:33, Marek Zarychta wrote:

W dniu 7.10.2024 o 07:05, David Cross pisze: 

How many entries are in your ldap structure?  I can attempt a replication here

Hello David,

I will rather not expose it publicly. Whole LDAP directory contains few thousand entries - and it was was used for the tests mentioned in this thread.

With the filters applied I see below 1k entries, and then lookup with nsdc running takes: first lookup 0.16s, next lookups 0.09s, while without nscd it varies from 0.12 to 0.08 - so nscd performs OK.

I have your patch applied and I am still testing it with net/nss-pam-ldapd from ports with patch for login classes applied (it's present in port but not enabled by default). So far it works without issues.

-- 
Marek Zarychta
--------------pqdn9J0qCc3irc6FbzLakvWV--