From owner-freebsd-bugs Thu Nov 19 10:49:57 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id KAA04660 for freebsd-bugs-outgoing; Thu, 19 Nov 1998 10:49:57 -0800 (PST) (envelope-from owner-freebsd-bugs@FreeBSD.ORG) Received: from freefall.freebsd.org (freefall.FreeBSD.ORG [204.216.27.21]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id KAA04653 for ; Thu, 19 Nov 1998 10:49:56 -0800 (PST) (envelope-from gnats@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.8.8/8.8.5) id KAA05905; Thu, 19 Nov 1998 10:50:01 -0800 (PST) Received: (from nobody@localhost) by hub.freebsd.org (8.8.8/8.8.8) id KAA04559; Thu, 19 Nov 1998 10:48:05 -0800 (PST) (envelope-from nobody) Message-Id: <199811191848.KAA04559@hub.freebsd.org> Date: Thu, 19 Nov 1998 10:48:05 -0800 (PST) From: lem@cantv.net To: freebsd-gnats-submit@FreeBSD.ORG X-Send-Pr-Version: www-1.0 Subject: misc/8764: pwd_mkdb is slow on many users Sender: owner-freebsd-bugs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org >Number: 8764 >Category: misc >Synopsis: pwd_mkdb is slow on many users >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Thu Nov 19 10:50:01 PST 1998 >Last-Modified: >Originator: Luis Munoz >Organization: CANTV Servicios >Release: 2.2.6-RELEASE >Environment: bash-2.01# uname -a FreeBSD rs8s1.datacenter.cha.cantv.net 2.2.6-RELEASE FreeBSD 2.2.6-RELEASE #0: F ri Aug 21 11:05:02 GMT 1998 root@myname.my.domain:/usr/src/sys/compile/CCSER V i386 >Description: On systems with large user counts (>= 50k), the time to rebuild tha password database becomes quite large. One of our test systems clocked 60+ minutes with 150k users. Increasing the buffer space as suggested on one of the freebsd-* lists reduced this time to 44+ minutes, which is roughly a 25% improvement. Probably further increasing cache could reduce the time by 50%. >How-To-Repeat: >Fix: This is a patch for pwd_mkdb on i386 2.2.6-RELEASE. The patch allows two new command-line options that reduce the time required to update the password database. Those are documented in the man page (after patches are applied :) In our text box, PII w256M RAM and RAID disks, with 150k users, time to build a password database went from 60+ minutes to 44+ minutes. Perhaps it could be improved even more... [If the patches do not work properly, please contact me directly to have them sent via email -lem] *** pwd_mkdb.8-orig Thu Nov 19 18:40:36 1998 --- pwd_mkdb.8 Wed Nov 18 22:18:11 1998 *************** *** 39,46 **** --- 39,48 ---- .Nd "generate the password databases" .Sh SYNOPSIS .Nm pwd_mkdb + .Op Fl b Ar base size .Op Fl c .Op Fl p + .Op Fl s Ar cache size .Op Fl d Ar directory .Op Fl u Ar username .Ar file *************** *** 63,74 **** --- 65,89 ---- .Pp The options are as follows: .Bl -tag -width flag + .It Fl b Ar base size + Use a factor to scale up certain parameters in the definition of + the database (see + .Xr hash 3 ) . + For large ammounts of users, this should be set to a rather large + factor, 64 or so, to speed up database creation. If left undefined + it defaults to reasonable values for around 500 users. .It Fl c Check if the password file is in the correct format. Do not change, add, or remove any files. .It Fl p Create a Version 7 style password file and install it into .Pa /etc/passwd . + .It Fl s Ar cache size + The database functions use a cache to speedup the process. This + flag allows the specification of a cache size. It's defined in + kilobytes. A larger cache tipically produces quicker processing + times. 8192 kilobytes provides plenty of space for larger user + databases. .It Fl d Ar directory Store databases into specified destination directory instead of .Pa /etc . *** pwd_mkdb.c-orig Wed Nov 18 21:34:59 1998 --- pwd_mkdb.c Wed Nov 18 22:08:49 1998 *************** *** 100,106 **** DBT data, sdata, key; FILE *fp, *oldfp; sigset_t set; ! int ch, cnt, ypcnt, len, makeold, tfd, yp_enabled = 0; char *p, *t; char buf[MAX(MAXPATHLEN, LINE_MAX * 2)], tbuf[1024]; char sbuf[MAX(MAXPATHLEN, LINE_MAX * 2)]; --- 100,107 ---- DBT data, sdata, key; FILE *fp, *oldfp; sigset_t set; ! int cachesize, basesize, ch, cnt, ypcnt, len, ! makeold, tfd, yp_enabled = 0; char *p, *t; char buf[MAX(MAXPATHLEN, LINE_MAX * 2)], tbuf[1024]; char sbuf[MAX(MAXPATHLEN, LINE_MAX * 2)]; *************** *** 114,121 **** strcpy(prefix, _PATH_PWD); makeold = 0; username = NULL; ! while ((ch = getopt(argc, argv, "cd:pu:v")) != -1) switch(ch) { case 'c': /* verify only */ cflag = 1; break; --- 115,127 ---- strcpy(prefix, _PATH_PWD); makeold = 0; username = NULL; ! basesize = 1; /* Default base size for DB */ ! cachesize = 2048; /* Default cache size for DB */ ! while ((ch = getopt(argc, argv, "s:b:cd:pu:v")) != -1) switch(ch) { + case 'b': /* Set bucket size */ + basesize = atoi(optarg); + break; case 'c': /* verify only */ cflag = 1; break; *************** *** 125,130 **** --- 131,139 ---- case 'p': /* create V7 "file.orig" */ makeold = 1; break; + case 's': /* Set buffer/cache size */ + cachesize = atoi(optarg); + break; case 'u': /* only update this record */ username = optarg; break; *************** *** 139,144 **** --- 148,173 ---- if (argc != 1 || (username && (*username == '+' || *username == '-'))) usage(); + if (basesize <= 0) { + error("Base size must be an integer >= 0"); + usage(); + } + + if (cachesize <= 0) { + error("Cache size must be an integer >= 0"); + usage(); + } + + /* update the openinfo structure */ + + openinfo.bsize = (u_int) 4096 * basesize; + if (openinfo.bsize > 16384) { + openinfo.bsize = 16384; + } + openinfo.ffactor = (u_int) 32 * basesize; + openinfo.nelem = (u_int) 256 * basesize * basesize; + openinfo.cachesize = cachesize * 1024; + /* * This could be changed to allow the user to interrupt. * Probably not worth the effort. *************** *** 577,582 **** { (void)fprintf(stderr, ! "usage: pwd_mkdb [-c] [-p] [-d ] [-u ] file\n"); exit(1); } --- 606,611 ---- { (void)fprintf(stderr, ! "usage: pwd_mkdb [-b ] [-c] [-p] [-d ] [-s ] [-u ] file\n"); exit(1); } >Audit-Trail: >Unformatted: To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message