From owner-freebsd-current@FreeBSD.ORG Mon Jul 16 19:22:22 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5597816A400 for ; Mon, 16 Jul 2007 19:22:22 +0000 (UTC) (envelope-from freebsd@hansmi.ch) Received: from hansmi.home.forkbomb.ch (hansmi.home.forkbomb.ch [213.144.146.165]) by mx1.freebsd.org (Postfix) with ESMTP id CBF7313C494 for ; Mon, 16 Jul 2007 19:22:16 +0000 (UTC) (envelope-from freebsd@hansmi.ch) Received: (qmail 25320 invoked by uid 1000); 16 Jul 2007 19:02:58 -0000 Date: Mon, 16 Jul 2007 21:02:58 +0200 From: Michael Hanselmann To: freebsd-current@freebsd.org Message-ID: <20070716190258.GA25030@hansmi.ch> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="IiVenqGWf+H9Y6IX" Content-Disposition: inline User-Agent: Mutt/1.5.13 (2006-08-11) Cc: tobias.mueller@hostpoint.ch, m.kooijman@student.utwente.nl, elias.wittwer@hostpoint.ch Subject: [PATCH] Implement getgroupmembership(3) for massive performance gain when using LDAP or Winbind X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2007 19:22:22 -0000 --IiVenqGWf+H9Y6IX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello I was working with a company which plans to migrate its FreeBSD servers from using /etc/{passwd,group} to LDAP. They will have about 45'000 users and as much groups in the directory. Tests showed that any function retrieving the groups a user is member of, for example getgrouplist(3) or initgroups(3), is very slow. In our case, it was about 7 seconds per invocation. Further investigation showed how inefficient these functions are implemented through getgrouplist(3). FreeBSD's implementation loops through all groups and their members to check whether a user is member of it, in which case it adds the group to a list. In our case, this means retrieving 45'000 search results from the LDAP server. Directory services like LDAP or Winbind allow queries to have filters, enabling us to write a much more efficient implementation. The attached patches (nss-getgroupmembership-try9.diff for FreeBSD 6, nss-getgroupmembership-fbsd7-try3.diff for FreeBSD 7) use an nss module's getgroupmembership(3) function if available. Otherwise it uses a fallback which then uses the old algorithm with some modifications. After applying it, getgrouplist(3) takes only a few milliseconds to retrieve all groups of a user. Another patch, attached as bsdnss.diff, is needed for nss_ldap. It applies to the ports/net/nss_ldap/files/bsdnss.c file and exports the required getgroupmembership function. Most of the code there is from NetBSD. The basic idea of getgroupmembership(3) has been taken from [1] and NetBSD, where it's already implemented. Thanks to Matthijs Kooijman for his preliminary work[2], testing with Winbind and support. A need for this code seems to have been around since some time already; the first reference[3] I've found being in March 2006. What do you think about the code? Is it usable? If not, I'm open to provide fixes. If yes, please apply it. Development of these patches is being sponsored by Hostpoint AG, http://www.hostpoint.ch/. They use currently use FreeBSD 6.2 and would like to see this patch being applied there, too. Greets, Michael [1] http://osdir.com/ml/netbsd.devel.userlevel/2004-12/msg00001.html [2] http://lists.freebsd.org/pipermail/freebsd-current/2006-May/063548.html [3] http://lists.freebsd.org/pipermail/freebsd-current/2006-March/061413.html --IiVenqGWf+H9Y6IX Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="bsdnss.diff" --- bsdnss.c.orig Mon May 14 14:25:05 2007 +++ bsdnss.c Mon May 14 15:01:06 2007 @@ -1,9 +1,11 @@ +#include #include #include #include #include #include #include +#include #include extern enum nss_status _nss_ldap_getgrent_r(struct group *, char *, size_t, @@ -34,12 +36,15 @@ extern enum nss_status _nss_ldap_gethost extern enum nss_status _nss_ldap_gethostbyaddr_r (struct in_addr * addr, int len, int type, struct hostent * result, char *buffer, size_t buflen, int *errnop, int *h_errnop); +extern enum nss_status _nss_ldap_initgroups_dyn(const char *, gid_t, long int *, + long int *, gid_t **, long int, int *); NSS_METHOD_PROTOTYPE(__nss_compat_getgrnam_r); NSS_METHOD_PROTOTYPE(__nss_compat_getgrgid_r); NSS_METHOD_PROTOTYPE(__nss_compat_getgrent_r); NSS_METHOD_PROTOTYPE(__nss_compat_setgrent); NSS_METHOD_PROTOTYPE(__nss_compat_endgrent); +static NSS_METHOD_PROTOTYPE(__freebsd_getgroupmembership); NSS_METHOD_PROTOTYPE(__nss_compat_getpwnam_r); NSS_METHOD_PROTOTYPE(__nss_compat_getpwuid_r); @@ -57,6 +62,7 @@ static ns_mtab methods[] = { { NSDB_GROUP, "getgrent_r", __nss_compat_getgrent_r, _nss_ldap_getgrent_r }, { NSDB_GROUP, "setgrent", __nss_compat_setgrent, _nss_ldap_setgrent }, { NSDB_GROUP, "endgrent", __nss_compat_endgrent, _nss_ldap_endgrent }, +{ NSDB_GROUP, "getgroupmembership", __freebsd_getgroupmembership, NULL }, { NSDB_PASSWD, "getpwnam_r", __nss_compat_getpwnam_r, _nss_ldap_getpwnam_r }, { NSDB_PASSWD, "getpwuid_r", __nss_compat_getpwuid_r, _nss_ldap_getpwuid_r }, @@ -155,4 +161,62 @@ int __nss_compat_gethostbyaddr(void *ret status = __nss_compat_result(status,errnop); h_errno = h_errnop; return (status); +} + +static int +__gr_addgid(gid_t gid, gid_t *groups, int maxgrp, int *groupc) +{ + int ret, dupc; + + /* skip duplicates */ + for (dupc = 0; dupc < MIN(maxgrp, *groupc); dupc++) { + if (groups[dupc] == gid) + return 1; + } + + ret = 1; + if (*groupc < maxgrp) /* add this gid */ + groups[*groupc] = gid; + else + ret = 0; + (*groupc)++; + return ret; +} + +static int +__freebsd_getgroupmembership(void *retval, void *mdata, va_list ap) +{ + int err; + enum nss_status s; + int *result = va_arg(ap, int *); + const char *user = va_arg(ap, const char *); + gid_t group = va_arg(ap, gid_t); + gid_t *groups = va_arg(ap, gid_t *); + int limit = va_arg(ap, int); + int *size = va_arg(ap, int*); + gid_t *tmpgroups; + long int lstart, lsize; + int i; + + tmpgroups = malloc(limit * sizeof(gid_t)); + if (tmpgroups == NULL) + return NS_TRYAGAIN; + + /* insert primary membership */ + __gr_addgid(group, groups, limit, size); + + lstart = 0; + lsize = limit; + s = _nss_ldap_initgroups_dyn(user, group, &lstart, &lsize, + &tmpgroups, 0, &err); + if (s == NSS_STATUS_SUCCESS) { + for (i = 0; i < lstart; i++) + if (! __gr_addgid(tmpgroups[i], groups, limit, size)) + *result = -1; + s = NSS_STATUS_NOTFOUND; + } + + free(tmpgroups); + + return __nss_compat_result(s, 0); } --IiVenqGWf+H9Y6IX Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="nss-getgroupmembership-try9.diff" Index: include/nsswitch.h =================================================================== RCS file: /home/ncvs/src/include/nsswitch.h,v retrieving revision 1.3 diff -u -b -B -u -p -r1.3 nsswitch.h --- include/nsswitch.h 17 Apr 2003 14:14:21 -0000 1.3 +++ include/nsswitch.h 14 May 2007 15:18:30 -0000 @@ -68,6 +68,7 @@ #define NSSRC_DNS "dns" /* DNS; IN for hosts, HS for others */ #define NSSRC_NIS "nis" /* YP/NIS */ #define NSSRC_COMPAT "compat" /* passwd,group in YP compat mode */ +#define NSSRC_ALL "*" /* All sources, used for fallbacks */ /* * currently implemented databases @@ -220,6 +221,24 @@ typedef struct _ns_mod { nss_module_unregister_fn unregister; /* called to unload module */ } ns_mod; +/* + * ns_fbtab `method' function signature. + */ +typedef int (*nss_fbmethod)(const ns_src *source, void *_retval, + void *_mdata, va_list _ap); + +/* + * ns_fbtab - `nsswitch fallback table' + * Contains an entry for each source and the appropriate function to + * call. ns_dtabs are used in the nsdispatch() API in order to allow + * the application to override built-in actions. + */ +typedef struct _ns_fbtab { + const char *src; /* Source this entry implements */ + nss_fbmethod method; /* Method to be called */ + void *mdata; /* Data passed to method */ +} ns_fbtab; + #endif /* _NS_PRIVATE */ @@ -230,6 +249,14 @@ extern int nsdispatch(void *, const ns_d const char *, const ns_src [], ...); #ifdef _NS_PRIVATE +extern int _nsdispatch_with_fb(void *, const ns_dtab [], + const ns_fbtab [], const char *, + const char *, const ns_src [], ...); +extern int _nsdispatch_callmethod(int*, const ns_src *, + const char *, const char *, + const ns_dtab [], const ns_fbtab [], + void *, void *, ...); + extern void _nsdbtaddsrc(ns_dbt *, const ns_src *); extern void _nsdbtput(const ns_dbt *); extern void _nsyyerror(const char *); Index: lib/libc/gen/getgrent.c =================================================================== RCS file: /home/ncvs/src/lib/libc/gen/getgrent.c,v retrieving revision 1.32.8.3 diff -u -b -B -u -p -r1.32.8.3 getgrent.c --- lib/libc/gen/getgrent.c 8 Oct 2006 05:45:57 -0000 1.32.8.3 +++ lib/libc/gen/getgrent.c 14 May 2007 15:18:30 -0000 @@ -46,6 +46,7 @@ __FBSDID("$FreeBSD: src/lib/libc/gen/get #include #endif #include +#define _NS_PRIVATE #include #include #include @@ -54,6 +55,7 @@ __FBSDID("$FreeBSD: src/lib/libc/gen/get #include #include #include +#include #include "un-namespace.h" #include "libc_private.h" #include "nss_tls.h" @@ -1070,6 +1069,188 @@ fin: fseeko(st->fp, pos, SEEK_SET); return (rv); #undef set_lookup_type +} + +static int +__gr_addgid(gid_t gid, gid_t *groups, int maxgrp, int *groupc) +{ + int ret, dupc; + + /* skip duplicates */ + for (dupc = 0; dupc < MIN(maxgrp, *groupc); dupc++) { + if (groups[dupc] == gid) + return 1; + } + + ret = 1; + if (*groupc < maxgrp) + /* add gid */ + groups[*groupc] = gid; + else + ret = 0; + + (*groupc)++; + + return ret; +} + +/* + * Fallback function for sources which don't have the getgroupmembership + * method. It uses the old and inefficient method of looping through all groups + * and their members. + */ +static int +getgroupmembership_fallback(const ns_src *source, + void *rv, void *mdata, va_list ap) +{ + static const ns_dtab dtab_getgrent[] = { + { NSSRC_FILES, files_group, (void *)nss_lt_all }, +#ifdef HESIOD + { NSSRC_DNS, dns_group, (void *)nss_lt_all }, +#endif +#ifdef YP + { NSSRC_NIS, nis_group, (void *)nss_lt_all }, +#endif + { NSSRC_COMPAT, compat_group, (void *)nss_lt_all }, + { NULL, NULL, NULL } + }; + + int *retval = va_arg(ap, int *); + const char *uname = va_arg(ap, const char *); + gid_t group = va_arg(ap, gid_t); + gid_t *groups = va_arg(ap, gid_t *); + int limit = va_arg(ap, int); + int *size = va_arg(ap, int *); + + struct group grp; + struct group *grp_p; + int i; + int lrv; + int lresult; + int lretval; + char *lbuf; + size_t lbufsize; + int ret_errno; + + lbuf = malloc(GRP_STORAGE_INITIAL); + if (lbuf == NULL) { + lrv = NS_UNAVAIL; + goto out; + } + lbufsize = GRP_STORAGE_INITIAL; + + /* Rewind */ + lrv = _nsdispatch_callmethod(&lresult, source, + NSDB_GROUP, "setgrent", + NULL, NULL, + &lretval, NULL, NULL); + + /* + * When installing primary group, duplicate it; + * the first element of groups is the effective gid + * and will be overwritten when a setgid file is executed. + */ + __gr_addgid(group, groups, limit, size); + + + *retval = 0; + + /* + * Scan source to find additional groups. + */ + while (1) { + /* We can't use getgrent() here because it wouldn't be safe for + * multithreaded programs. + */ + do { + ret_errno = 0; + grp_p = NULL; + lrv = _nsdispatch_callmethod(&lresult, source, + NSDB_GROUP, "getgrent_r", + dtab_getgrent, NULL, + &grp_p, (void *)nss_lt_all, &grp, lbuf, lbufsize, &ret_errno); + + if (grp_p == NULL && ret_errno == ERANGE) { + free(lbuf); + + if ((lbufsize << 1) > GRP_STORAGE_MAX) { + lbuf = NULL; + errno = ERANGE; + lrv = NS_UNAVAIL; + goto out; + } + + lbufsize <<= 1; + lbuf = malloc(lbufsize); + if (lbuf == NULL) { + lrv = NS_UNAVAIL; + goto out; + } + } + } while (grp_p == NULL && ret_errno == ERANGE); + + if (ret_errno != 0) { + errno = ret_errno; + lrv = NS_UNAVAIL; + goto out; + } + + if (grp_p == NULL) + break; + + /* Loop through group members */ + for (i = 0; grp.gr_mem[i]; i++) { + if (strcmp(grp.gr_mem[i], uname) == 0) { + if (!__gr_addgid(grp.gr_gid, groups, limit, size)) { + *retval = -1; + } + } + } + } + + /* Close database */ + lrv = _nsdispatch_callmethod(&lresult, source, + NSDB_GROUP, "endgrent", + NULL, NULL, + &lretval, NULL, NULL); + + lrv = NS_NOTFOUND; + +out: + free(lbuf); + + return lrv; +} + +/* + * getgroupmembership is about the same as getgrouplist(3), except it has a + * different interface that supports being dispatched by nss. + */ +int +getgroupmembership(const char *uname, gid_t agroup, gid_t *groups, + int maxgrp, int *groupc) +{ + static const ns_fbtab fallback[] = { + { NSSRC_ALL, getgroupmembership_fallback, NULL }, + { NULL, NULL, NULL } + }; + int rv, rerror, retval; + + assert(uname != NULL); + /* groups may be NULL if just sizing when invoked with maxgrp = 0 */ + assert(groupc != NULL); + + *groupc = 0; + + rv = _nsdispatch_with_fb(&retval, NULL, fallback, + NSDB_GROUP, "getgroupmembership", defaultsrc, + &rerror, uname, agroup, groups, maxgrp, groupc); + + /* too many groups found? */ + if (*groupc > maxgrp) + return -1; + else + return 0; } Index: lib/libc/gen/getgrouplist.c =================================================================== RCS file: /home/ncvs/src/lib/libc/gen/getgrouplist.c,v retrieving revision 1.14 diff -u -b -B -u -p -r1.14 getgrouplist.c --- lib/libc/gen/getgrouplist.c 3 May 2005 16:20:03 -0000 1.14 +++ lib/libc/gen/getgrouplist.c 14 May 2007 15:18:30 -0000 @@ -46,46 +46,12 @@ __FBSDID("$FreeBSD: src/lib/libc/gen/get #include #include +extern int +getgroupmembership(const char *uname, gid_t agroup, gid_t *groups, + int maxgrp, int *groupc); + int getgrouplist(const char *uname, gid_t agroup, gid_t *groups, int *grpcnt) { - const struct group *grp; - int i, maxgroups, ngroups, ret; - - ret = 0; - ngroups = 0; - maxgroups = *grpcnt; - /* - * When installing primary group, duplicate it; - * the first element of groups is the effective gid - * and will be overwritten when a setgid file is executed. - */ - groups[ngroups++] = agroup; - if (maxgroups > 1) - groups[ngroups++] = agroup; - /* - * Scan the group file to find additional groups. - */ - setgrent(); - while ((grp = getgrent()) != NULL) { - for (i = 0; i < ngroups; i++) { - if (grp->gr_gid == groups[i]) - goto skip; - } - for (i = 0; grp->gr_mem[i]; i++) { - if (!strcmp(grp->gr_mem[i], uname)) { - if (ngroups >= maxgroups) { - ret = -1; - break; - } - groups[ngroups++] = grp->gr_gid; - break; - } - } -skip: - ; - } - endgrent(); - *grpcnt = ngroups; - return (ret); + return getgroupmembership(uname, agroup, groups, *grpcnt, grpcnt); } Index: lib/libc/net/nsdispatch.c =================================================================== RCS file: /home/ncvs/src/lib/libc/net/nsdispatch.c,v retrieving revision 1.12 diff -u -b -B -u -p -r1.12 nsdispatch.c --- lib/libc/net/nsdispatch.c 1 Apr 2004 19:12:45 -0000 1.12 +++ lib/libc/net/nsdispatch.c 14 May 2007 15:18:30 -0000 @@ -569,19 +572,108 @@ nss_method_lookup(const char *source, co return (NULL); } +/* + * Looks up a fallback method + */ +static nss_fbmethod +nss_fbmethod_lookup(const char *source, const char *method, + const ns_fbtab fallback[], void **mdata) +{ + int i; + + for (i = 0; fallback[i].src != NULL; i++) + if (strcasecmp(source, fallback[i].src) == 0 || + strcasecmp(NSSRC_ALL, fallback[i].src) == 0) { + *mdata = fallback[i].mdata; + return (fallback[i].method); + } -__weak_reference(_nsdispatch, nsdispatch); + *mdata = NULL; + return NULL; +} + +/* + * Calls a function from an nss source. If the function isn't defined, it uses + * the provided fallback table. For a description of the fallback table, see + * _nsdispatch_with_fbv. + */ +static int +_nsdispatch_callmethodv(int* result, const ns_src *source, + const char *database, const char *method_name, + const ns_dtab disp_tab[], const ns_fbtab fallback[], + void *retval, void *mdata, va_list ap) +{ + nss_method method; + nss_fbmethod fbmethod; + *result = NS_NOTFOUND; + + method = nss_method_lookup(source->name, database, + method_name, disp_tab, &mdata); + + if (method == NULL && fallback != NULL) { + /* Try to get fallback function */ + fbmethod = nss_fbmethod_lookup(source->name, + method_name, fallback, &mdata); + } else { + fbmethod = NULL; + } + + if (method != NULL || fbmethod != NULL) { + if (fbmethod != NULL) { + *result = fbmethod(source, retval, mdata, ap); + } else { + *result = method(retval, mdata, ap); + } + + if (*result & (source->flags)) + return NS_ACTION_RETURN; + } + + return NS_ACTION_CONTINUE; +} + +/* + * Wrapper around _nsdispatch_callmethodv + */ int -_nsdispatch(void *retval, const ns_dtab disp_tab[], const char *database, - const char *method_name, const ns_src defaults[], ...) +_nsdispatch_callmethod(int* result, const ns_src *source, + const char *database, const char *method_name, + const ns_dtab disp_tab[], const ns_fbtab fallback[], + void *retval, void *mdata, ...) { va_list ap; + int rv; + + va_start(ap, mdata); + rv = _nsdispatch_callmethodv(result, source, + database, method_name, + disp_tab, fallback, + retval, mdata, ap); + va_end(ap); + + return rv; +} + +/* + * nsdispatch function with fallback functionality. The fallbacks can be used + * to replace unimplemented functions in sources. + * + * A fallback table can contain any source name or the special entry of + * NSSRC_ALL to match all sources. If used, NSSRC_ALL must be the last entry. + * The fallbcak function is then called for each source which doesn't have an + * implementation of the wanted function. + */ +int +_nsdispatch_with_fbv(void *retval, const ns_dtab disp_tab[], + const ns_fbtab fallback[], const char *database, + const char *method_name, const ns_src defaults[], + va_list ap) +{ const ns_dbt *dbt; const ns_src *srclist; - nss_method method; void *mdata; - int isthreaded, serrno, i, result, srclistsize; + int isthreaded, serrno, i, result, srclistsize, rv; isthreaded = __isthreaded; serrno = errno; @@ -608,15 +702,13 @@ _nsdispatch(void *retval, const ns_dtab while (srclist[srclistsize].name != NULL) srclistsize++; } + for (i = 0; i < srclistsize; i++) { - result = NS_NOTFOUND; - method = nss_method_lookup(srclist[i].name, database, - method_name, disp_tab, &mdata); - if (method != NULL) { - va_start(ap, defaults); - result = method(retval, mdata, ap); - va_end(ap); - if (result & (srclist[i].flags)) + rv = _nsdispatch_callmethodv(&result, &srclist[i], + database, method_name, + disp_tab, fallback, + retval, mdata, ap); + if (rv == NS_ACTION_RETURN) { break; } } @@ -626,3 +718,43 @@ fin: errno = serrno; return (result); } + +/* + * Wrapper around _nsdispatch_with_fbv + */ +int +_nsdispatch_with_fb(void *retval, const ns_dtab disp_tab[], + const ns_fbtab fallback[], const char *database, + const char *method_name, const ns_src defaults[], + ...) +{ + va_list ap; + int rv; + + va_start(ap, defaults); + rv = _nsdispatch_with_fbv(retval, disp_tab, fallback, database, + method_name, defaults, ap); + va_end(ap); + + return rv; +} + +/* + * Original nsdispatch function without fallback functionality. + */ +int +_nsdispatch(void *retval, const ns_dtab disp_tab[], const char *database, + const char *method_name, const ns_src defaults[], ...) +{ + va_list ap; + int rv; + + va_start(ap, defaults); + rv = _nsdispatch_with_fbv(retval, disp_tab, NULL, database, + method_name, defaults, ap); + va_end(ap); + + return rv; +} + +__weak_reference(_nsdispatch, nsdispatch); --IiVenqGWf+H9Y6IX Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="nss-getgroupmembership-fbsd7-try3.diff" Index: include/nsswitch.h =================================================================== RCS file: /home/ncvs/src/include/nsswitch.h,v retrieving revision 1.4 diff -u -b -B -u -p -r1.4 nsswitch.h --- include/nsswitch.h 28 Apr 2006 12:03:34 -0000 1.4 +++ include/nsswitch.h 12 Jun 2007 20:49:25 -0000 @@ -69,6 +69,7 @@ #define NSSRC_NIS "nis" /* YP/NIS */ #define NSSRC_COMPAT "compat" /* passwd,group in YP compat mode */ #define NSSRC_CACHE "cache" /* cache daemon */ +#define NSSRC_ALL "*" /* All sources, used for fallbacks */ /* * currently implemented databases @@ -222,6 +223,24 @@ typedef struct _ns_mod { nss_module_unregister_fn unregister; /* called to unload module */ } ns_mod; +/* + * ns_fbtab `method' function signature. + */ +typedef int (*nss_fbmethod)(const ns_src *source, void *_retval, + void *_mdata, va_list _ap); + +/* + * ns_fbtab - `nsswitch fallback table' + * Contains an entry for each source and the appropriate function to + * call. ns_dtabs are used in the nsdispatch() API in order to allow + * the application to override built-in actions. + */ +typedef struct _ns_fbtab { + const char *src; /* Source this entry implements */ + nss_fbmethod method; /* Method to be called */ + void *mdata; /* Data passed to method */ +} ns_fbtab; + #endif /* _NS_PRIVATE */ @@ -232,6 +251,14 @@ extern int nsdispatch(void *, const ns_d const char *, const ns_src [], ...); #ifdef _NS_PRIVATE +extern int _nsdispatch_with_fb(void *, const ns_dtab [], + const ns_fbtab [], const char *, + const char *, const ns_src [], ...); +extern int _nsdispatch_callmethod(int*, const ns_src *, + const char *, const char *, + const ns_dtab [], const ns_fbtab [], + void *, void *, ...); + extern void _nsdbtaddsrc(ns_dbt *, const ns_src *); extern void _nsdbtput(const ns_dbt *); extern void _nsyyerror(const char *); Index: lib/libc/gen/getgrent.c =================================================================== RCS file: /home/ncvs/src/lib/libc/gen/getgrent.c,v retrieving revision 1.36 diff -u -b -B -u -p -r1.36 getgrent.c --- lib/libc/gen/getgrent.c 18 Sep 2006 09:34:48 -0000 1.36 +++ lib/libc/gen/getgrent.c 12 Jun 2007 20:49:25 -0000 @@ -46,6 +46,7 @@ __FBSDID("$FreeBSD: src/lib/libc/gen/get #include #endif #include +#define _NS_PRIVATE #include #include #include @@ -54,6 +55,7 @@ __FBSDID("$FreeBSD: src/lib/libc/gen/get #include #include #include +#include #include "un-namespace.h" #include "libc_private.h" #include "nss_tls.h" @@ -447,17 +449,13 @@ endgrent(void) } -int -getgrent_r(struct group *grp, char *buffer, size_t bufsize, - struct group **result) -{ #ifdef NS_CACHING - static const nss_cache_info cache_info = NS_MP_CACHE_INFO_INITIALIZER( +static const nss_cache_info getgrent_cache_info = NS_MP_CACHE_INFO_INITIALIZER( group, (void *)nss_lt_all, grp_marshal_func, grp_unmarshal_func); #endif - static const ns_dtab dtab[] = { +static const ns_dtab getgrent_dtab[] = { { NSSRC_FILES, files_group, (void *)nss_lt_all }, #ifdef HESIOD { NSSRC_DNS, dns_group, (void *)nss_lt_all }, @@ -467,16 +465,21 @@ getgrent_r(struct group *grp, char *buff #endif { NSSRC_COMPAT, compat_group, (void *)nss_lt_all }, #ifdef NS_CACHING - NS_CACHE_CB(&cache_info) + NS_CACHE_CB(&getgrent_cache_info) #endif { NULL, NULL, NULL } - }; +}; + +int +getgrent_r(struct group *grp, char *buffer, size_t bufsize, + struct group **result) +{ int rv, ret_errno; ret_errno = 0; *result = NULL; - rv = _nsdispatch(result, dtab, NSDB_GROUP, "getgrent_r", defaultsrc, - grp, buffer, bufsize, &ret_errno); + rv = _nsdispatch(result, getgrent_dtab, NSDB_GROUP, "getgrent_r", + defaultsrc, grp, buffer, bufsize, &ret_errno); if (rv == NS_SUCCESS) return (0); else @@ -1346,6 +1349,175 @@ fin: fseeko(st->fp, pos, SEEK_SET); return (rv); #undef set_lookup_type +} + +static int +__gr_addgid(gid_t gid, gid_t *groups, int maxgrp, int *groupc) +{ + int ret, dupc; + + /* skip duplicates */ + for (dupc = 0; dupc < MIN(maxgrp, *groupc); dupc++) { + if (groups[dupc] == gid) + return 1; + } + + ret = 1; + if (*groupc < maxgrp) + /* add gid */ + groups[*groupc] = gid; + else + ret = 0; + + (*groupc)++; + + return ret; +} + +/* + * Fallback function for sources which don't have the getgroupmembership + * method. It uses the old and inefficient method of looping through all groups + * and their members. + */ +static int +getgroupmembership_fallback(const ns_src *source, + void *rv, void *mdata, va_list ap) +{ + int *retval = va_arg(ap, int *); + const char *uname = va_arg(ap, const char *); + gid_t group = va_arg(ap, gid_t); + gid_t *groups = va_arg(ap, gid_t *); + int limit = va_arg(ap, int); + int *size = va_arg(ap, int *); + + struct group grp; + struct group *grp_p; + int i; + int lrv; + int lresult; + int lretval; + char *lbuf; + size_t lbufsize; + int ret_errno; + + lbuf = malloc(GRP_STORAGE_INITIAL); + if (lbuf == NULL) { + lrv = NS_UNAVAIL; + goto out; + } + lbufsize = GRP_STORAGE_INITIAL; + + /* Rewind */ + lrv = _nsdispatch_callmethod(&lresult, source, + NSDB_GROUP, "setgrent", + NULL, NULL, + &lretval, NULL, NULL); + + /* + * When installing primary group, duplicate it; + * the first element of groups is the effective gid + * and will be overwritten when a setgid file is executed. + */ + __gr_addgid(group, groups, limit, size); + + *retval = 0; + + /* + * Scan source to find additional groups. + */ + while (1) { + /* We can't use getgrent() here because it wouldn't be safe for + * multithreaded programs. + */ + do { + ret_errno = 0; + grp_p = NULL; + lrv = _nsdispatch_callmethod(&lresult, source, + NSDB_GROUP, "getgrent_r", + getgrent_dtab, NULL, + &grp_p, (void *)nss_lt_all, &grp, lbuf, lbufsize, &ret_errno); + + if (grp_p == NULL && ret_errno == ERANGE) { + free(lbuf); + + if ((lbufsize << 1) > GRP_STORAGE_MAX) { + lbuf = NULL; + errno = ERANGE; + lrv = NS_UNAVAIL; + goto out; + } + + lbufsize <<= 1; + lbuf = malloc(lbufsize); + if (lbuf == NULL) { + lrv = NS_UNAVAIL; + goto out; + } + } + } while (grp_p == NULL && ret_errno == ERANGE); + + if (ret_errno != 0) { + errno = ret_errno; + lrv = NS_UNAVAIL; + goto out; + } + + if (grp_p == NULL) + break; + + /* Loop through group members */ + for (i = 0; grp.gr_mem[i]; i++) { + if (strcmp(grp.gr_mem[i], uname) == 0) { + if (!__gr_addgid(grp.gr_gid, groups, limit, size)) { + *retval = -1; + } + } + } + } + + /* Close database */ + lrv = _nsdispatch_callmethod(&lresult, source, + NSDB_GROUP, "endgrent", + NULL, NULL, + &lretval, NULL, NULL); + + lrv = NS_NOTFOUND; + +out: + free(lbuf); + + return lrv; +} + +/* + * getgroupmembership is about the same as getgrouplist(3), except it has a + * different interface that supports being dispatched by nss. + */ +int +getgroupmembership(const char *uname, gid_t agroup, gid_t *groups, + int maxgrp, int *groupc) +{ + static const ns_fbtab fallback[] = { + { NSSRC_ALL, getgroupmembership_fallback, NULL }, + { NULL, NULL, NULL } + }; + int rv, rerror, retval; + + assert(uname != NULL); + /* groups may be NULL if just sizing when invoked with maxgrp = 0 */ + assert(groupc != NULL); + + *groupc = 0; + + rv = _nsdispatch_with_fb(&retval, NULL, fallback, + NSDB_GROUP, "getgroupmembership", defaultsrc, + &rerror, uname, agroup, groups, maxgrp, groupc); + + /* too many groups found? */ + if (*groupc > maxgrp) + return -1; + else + return 0; } Index: lib/libc/gen/getgrouplist.c =================================================================== RCS file: /home/ncvs/src/lib/libc/gen/getgrouplist.c,v retrieving revision 1.15 diff -u -b -B -u -p -r1.15 getgrouplist.c --- lib/libc/gen/getgrouplist.c 9 Jan 2007 00:27:53 -0000 1.15 +++ lib/libc/gen/getgrouplist.c 12 Jun 2007 20:49:25 -0000 @@ -41,47 +41,14 @@ __FBSDID("$FreeBSD: src/lib/libc/gen/get #include #include #include +#include + +extern int +getgroupmembership(const char *uname, gid_t agroup, gid_t *groups, + int maxgrp, int *groupc); int getgrouplist(const char *uname, gid_t agroup, gid_t *groups, int *grpcnt) { - const struct group *grp; - int i, maxgroups, ngroups, ret; - - ret = 0; - ngroups = 0; - maxgroups = *grpcnt; - /* - * When installing primary group, duplicate it; - * the first element of groups is the effective gid - * and will be overwritten when a setgid file is executed. - */ - groups[ngroups++] = agroup; - if (maxgroups > 1) - groups[ngroups++] = agroup; - /* - * Scan the group file to find additional groups. - */ - setgrent(); - while ((grp = getgrent()) != NULL) { - for (i = 0; i < ngroups; i++) { - if (grp->gr_gid == groups[i]) - goto skip; - } - for (i = 0; grp->gr_mem[i]; i++) { - if (!strcmp(grp->gr_mem[i], uname)) { - if (ngroups >= maxgroups) { - ret = -1; - break; - } - groups[ngroups++] = grp->gr_gid; - break; - } - } -skip: - ; - } - endgrent(); - *grpcnt = ngroups; - return (ret); + return getgroupmembership(uname, agroup, groups, *grpcnt, grpcnt); } Index: lib/libc/net/nsdispatch.c =================================================================== RCS file: /home/ncvs/src/lib/libc/net/nsdispatch.c,v retrieving revision 1.14 diff -u -b -B -u -p -r1.14 nsdispatch.c --- lib/libc/net/nsdispatch.c 17 May 2007 03:33:23 -0000 1.14 +++ lib/libc/net/nsdispatch.c 12 Jun 2007 20:49:26 -0000 @@ -590,19 +590,147 @@ nss_method_lookup(const char *source, co return (NULL); } +/* + * Looks up a fallback method + */ +static nss_fbmethod +nss_fbmethod_lookup(const char *source, const char *method, + const ns_fbtab fallback[], void **mdata) +{ + int i; -__weak_reference(_nsdispatch, nsdispatch); + for (i = 0; fallback[i].src != NULL; i++) + if (strcasecmp(source, fallback[i].src) == 0 || + strcasecmp(NSSRC_ALL, fallback[i].src) == 0) { + *mdata = fallback[i].mdata; + return (fallback[i].method); + } + + *mdata = NULL; + return NULL; +} + +/* + * Calls a function from an nss source. If the function isn't defined, it uses + * the provided fallback table. For a description of the fallback table, see + * _nsdispatch_with_fbv. + */ +static int +_nsdispatch_callmethod_with_cache_v(int* result, const ns_src *source, + const char *database, const char *method_name, + const ns_dtab disp_tab[], const ns_fbtab fallback[], +#ifdef NS_CACHING + nss_cache_data *cache_data, nss_cache_data **cache_data_p, + int *cache_flag, +#endif + void *retval, void *mdata, va_list ap) +{ + nss_method method; + nss_fbmethod fbmethod; + + *result = NS_NOTFOUND; + + method = nss_method_lookup(source->name, database, + method_name, disp_tab, &mdata); + + if (method == NULL && fallback != NULL) { + /* Try to get fallback function */ + fbmethod = nss_fbmethod_lookup(source->name, + method_name, fallback, &mdata); + } else { + fbmethod = NULL; + } + + if (method != NULL || fbmethod != NULL) { +#ifdef NS_CACHING + if (cache_data != NULL && cache_data_p != NULL && + cache_flag != NULL && + strcmp(source->name, NSSRC_CACHE) == 0 && + nss_cache_cycle_prevention_func == NULL) { +#ifdef NS_STRICT_LIBC_EID_CHECKING + if (issetugid() != 0) + return NS_ACTION_CONTINUE; +#endif + *cache_flag = 1; + + memset(cache_data, 0, sizeof(nss_cache_data)); + cache_data->info = (nss_cache_info const *)mdata; + *cache_data_p = cache_data; + + if (cache_data->info->id_func != NULL) + *result = __nss_common_cache_read(retval, + *cache_data_p, ap); + else if (cache_data->info->marshal_func != NULL) + *result = __nss_mp_cache_read(retval, + *cache_data_p, ap); + else + *result = __nss_mp_cache_end(retval, + *cache_data_p, ap); + } else { + *cache_flag = 0; +#endif + + if (fbmethod != NULL) { + *result = fbmethod(source, retval, mdata, ap); + } else { + *result = method(retval, mdata, ap); + } +#ifdef NS_CACHING + } +#endif + + if (*result & (source->flags)) + return NS_ACTION_RETURN; + } + + return NS_ACTION_CONTINUE; +} + +/* + * Wrapper around _nsdispatch_callmethod_with_cache_v + */ int -_nsdispatch(void *retval, const ns_dtab disp_tab[], const char *database, - const char *method_name, const ns_src defaults[], ...) +_nsdispatch_callmethod(int* result, const ns_src *source, + const char *database, const char *method_name, + const ns_dtab disp_tab[], const ns_fbtab fallback[], + void *retval, void *mdata, ...) { va_list ap; + int rv; + + va_start(ap, mdata); + rv = _nsdispatch_callmethod_with_cache_v(result, source, + database, method_name, + disp_tab, fallback, +#ifdef NS_CACHING + NULL, NULL, NULL, +#endif + retval, mdata, ap); + va_end(ap); + + return rv; +} + +/* + * nsdispatch function with fallback functionality. The fallbacks can be used + * to replace unimplemented functions in sources. + * + * A fallback table can contain any source name or the special entry of + * NSSRC_ALL to match all sources. If used, NSSRC_ALL must be the last entry. + * The fallbcak function is then called for each source which doesn't have an + * implementation of the wanted function. + */ +int +_nsdispatch_with_fbv(void *retval, const ns_dtab disp_tab[], + const ns_fbtab fallback[], const char *database, + const char *method_name, const ns_src defaults[], + va_list ap) +{ const ns_dbt *dbt; const ns_src *srclist; - nss_method method; void *mdata; - int isthreaded, serrno, i, result, srclistsize; + int isthreaded, serrno, i, result, srclistsize, rv; #ifdef NS_CACHING nss_cache_data cache_data; @@ -640,57 +768,22 @@ _nsdispatch(void *retval, const ns_dtab cache_data_p = NULL; cache_flag = 0; #endif - for (i = 0; i < srclistsize; i++) { - result = NS_NOTFOUND; - method = nss_method_lookup(srclist[i].name, database, - method_name, disp_tab, &mdata); - if (method != NULL) { + for (i = 0; i < srclistsize; i++) { + rv = _nsdispatch_callmethod_with_cache_v(&result, &srclist[i], + database, method_name, + disp_tab, fallback, #ifdef NS_CACHING - if (strcmp(srclist[i].name, NSSRC_CACHE) == 0 && - nss_cache_cycle_prevention_func == NULL) { -#ifdef NS_STRICT_LIBC_EID_CHECKING - if (issetugid() != 0) - continue; + &cache_data, &cache_data_p, &cache_flag, #endif - cache_flag = 1; - - memset(&cache_data, 0, sizeof(nss_cache_data)); - cache_data.info = (nss_cache_info const *)mdata; - cache_data_p = &cache_data; - - va_start(ap, defaults); - if (cache_data.info->id_func != NULL) - result = __nss_common_cache_read(retval, - cache_data_p, ap); - else if (cache_data.info->marshal_func != NULL) - result = __nss_mp_cache_read(retval, - cache_data_p, ap); - else - result = __nss_mp_cache_end(retval, - cache_data_p, ap); - va_end(ap); - } else { - cache_flag = 0; - va_start(ap, defaults); - result = method(retval, mdata, ap); - va_end(ap); - } -#else /* NS_CACHING */ - va_start(ap, defaults); - result = method(retval, mdata, ap); - va_end(ap); -#endif /* NS_CACHING */ - - if (result & (srclist[i].flags)) + retval, mdata, ap); + if (rv == NS_ACTION_RETURN) break; } - } #ifdef NS_CACHING if (cache_data_p != NULL && (result & (NS_NOTFOUND | NS_SUCCESS)) && cache_flag == 0) { - va_start(ap, defaults); if (result == NS_SUCCESS) { if (cache_data.info->id_func != NULL) __nss_common_cache_write(retval, cache_data_p, @@ -705,7 +798,6 @@ _nsdispatch(void *retval, const ns_dtab } else __nss_common_cache_write_negative(cache_data_p); } - va_end(ap); } #endif /* NS_CACHING */ @@ -715,3 +807,43 @@ fin: errno = serrno; return (result); } + +/* + * Wrapper around _nsdispatch_with_fbv + */ +int +_nsdispatch_with_fb(void *retval, const ns_dtab disp_tab[], + const ns_fbtab fallback[], const char *database, + const char *method_name, const ns_src defaults[], + ...) +{ + va_list ap; + int rv; + + va_start(ap, defaults); + rv = _nsdispatch_with_fbv(retval, disp_tab, fallback, database, + method_name, defaults, ap); + va_end(ap); + + return rv; +} + +/* + * Original nsdispatch function without fallback functionality. + */ +int +_nsdispatch(void *retval, const ns_dtab disp_tab[], const char *database, + const char *method_name, const ns_src defaults[], ...) +{ + va_list ap; + int rv; + + va_start(ap, defaults); + rv = _nsdispatch_with_fbv(retval, disp_tab, NULL, database, + method_name, defaults, ap); + va_end(ap); + + return rv; +} + +__weak_reference(_nsdispatch, nsdispatch); --IiVenqGWf+H9Y6IX--