Date: Thu, 19 Nov 1998 10:48:05 -0800 (PST) From: lem@cantv.net To: freebsd-gnats-submit@FreeBSD.ORG Subject: misc/8764: pwd_mkdb is slow on many users Message-ID: <199811191848.KAA04559@hub.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 8764
>Category: misc
>Synopsis: pwd_mkdb is slow on many users
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: change-request
>Submitter-Id: current-users
>Arrival-Date: Thu Nov 19 10:50:01 PST 1998
>Last-Modified:
>Originator: Luis Munoz
>Organization:
CANTV Servicios
>Release: 2.2.6-RELEASE
>Environment:
bash-2.01# uname -a
FreeBSD rs8s1.datacenter.cha.cantv.net 2.2.6-RELEASE FreeBSD 2.2.6-RELEASE #0: F
ri Aug 21 11:05:02 GMT 1998 root@myname.my.domain:/usr/src/sys/compile/CCSER
V i386
>Description:
On systems with large user counts (>= 50k), the time to rebuild
tha password database becomes quite large. One of our test systems
clocked 60+ minutes with 150k users. Increasing the buffer space
as suggested on one of the freebsd-* lists reduced this time to
44+ minutes, which is roughly a 25% improvement.
Probably further increasing cache could reduce the time by 50%.
>How-To-Repeat:
>Fix:
This is a patch for pwd_mkdb on i386 2.2.6-RELEASE. The patch allows
two new command-line options that reduce the time required to
update the password database. Those are documented in the man page
(after patches are applied :) In our text box, PII w256M RAM
and RAID disks, with 150k users, time to build a password database
went from 60+ minutes to 44+ minutes. Perhaps it could be improved
even more...
[If the patches do not work properly, please contact me directly
to have them sent via email -lem]
*** pwd_mkdb.8-orig Thu Nov 19 18:40:36 1998
--- pwd_mkdb.8 Wed Nov 18 22:18:11 1998
***************
*** 39,46 ****
--- 39,48 ----
.Nd "generate the password databases"
.Sh SYNOPSIS
.Nm pwd_mkdb
+ .Op Fl b Ar base size
.Op Fl c
.Op Fl p
+ .Op Fl s Ar cache size
.Op Fl d Ar directory
.Op Fl u Ar username
.Ar file
***************
*** 63,74 ****
--- 65,89 ----
.Pp
The options are as follows:
.Bl -tag -width flag
+ .It Fl b Ar base size
+ Use a factor to scale up certain parameters in the definition of
+ the database (see
+ .Xr hash 3 ) .
+ For large ammounts of users, this should be set to a rather large
+ factor, 64 or so, to speed up database creation. If left undefined
+ it defaults to reasonable values for around 500 users.
.It Fl c
Check if the password file is in the correct format. Do not
change, add, or remove any files.
.It Fl p
Create a Version 7 style password file and install it into
.Pa /etc/passwd .
+ .It Fl s Ar cache size
+ The database functions use a cache to speedup the process. This
+ flag allows the specification of a cache size. It's defined in
+ kilobytes. A larger cache tipically produces quicker processing
+ times. 8192 kilobytes provides plenty of space for larger user
+ databases.
.It Fl d Ar directory
Store databases into specified destination directory instead of
.Pa /etc .
*** pwd_mkdb.c-orig Wed Nov 18 21:34:59 1998
--- pwd_mkdb.c Wed Nov 18 22:08:49 1998
***************
*** 100,106 ****
DBT data, sdata, key;
FILE *fp, *oldfp;
sigset_t set;
! int ch, cnt, ypcnt, len, makeold, tfd, yp_enabled = 0;
char *p, *t;
char buf[MAX(MAXPATHLEN, LINE_MAX * 2)], tbuf[1024];
char sbuf[MAX(MAXPATHLEN, LINE_MAX * 2)];
--- 100,107 ----
DBT data, sdata, key;
FILE *fp, *oldfp;
sigset_t set;
! int cachesize, basesize, ch, cnt, ypcnt, len,
! makeold, tfd, yp_enabled = 0;
char *p, *t;
char buf[MAX(MAXPATHLEN, LINE_MAX * 2)], tbuf[1024];
char sbuf[MAX(MAXPATHLEN, LINE_MAX * 2)];
***************
*** 114,121 ****
strcpy(prefix, _PATH_PWD);
makeold = 0;
username = NULL;
! while ((ch = getopt(argc, argv, "cd:pu:v")) != -1)
switch(ch) {
case 'c': /* verify only */
cflag = 1;
break;
--- 115,127 ----
strcpy(prefix, _PATH_PWD);
makeold = 0;
username = NULL;
! basesize = 1; /* Default base size for DB */
! cachesize = 2048; /* Default cache size for DB */
! while ((ch = getopt(argc, argv, "s:b:cd:pu:v")) != -1)
switch(ch) {
+ case 'b': /* Set bucket size */
+ basesize = atoi(optarg);
+ break;
case 'c': /* verify only */
cflag = 1;
break;
***************
*** 125,130 ****
--- 131,139 ----
case 'p': /* create V7 "file.orig" */
makeold = 1;
break;
+ case 's': /* Set buffer/cache size */
+ cachesize = atoi(optarg);
+ break;
case 'u': /* only update this record */
username = optarg;
break;
***************
*** 139,144 ****
--- 148,173 ----
if (argc != 1 || (username && (*username == '+' || *username == '-')))
usage();
+ if (basesize <= 0) {
+ error("Base size must be an integer >= 0");
+ usage();
+ }
+
+ if (cachesize <= 0) {
+ error("Cache size must be an integer >= 0");
+ usage();
+ }
+
+ /* update the openinfo structure */
+
+ openinfo.bsize = (u_int) 4096 * basesize;
+ if (openinfo.bsize > 16384) {
+ openinfo.bsize = 16384;
+ }
+ openinfo.ffactor = (u_int) 32 * basesize;
+ openinfo.nelem = (u_int) 256 * basesize * basesize;
+ openinfo.cachesize = cachesize * 1024;
+
/*
* This could be changed to allow the user to interrupt.
* Probably not worth the effort.
***************
*** 577,582 ****
{
(void)fprintf(stderr,
! "usage: pwd_mkdb [-c] [-p] [-d <dest dir>] [-u <local username>] file\n");
exit(1);
}
--- 606,611 ----
{
(void)fprintf(stderr,
! "usage: pwd_mkdb [-b <size>] [-c] [-p] [-d <dest dir>] [-s <cache size>] [-u <local username>] file\n");
exit(1);
}
>Audit-Trail:
>Unformatted:
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199811191848.KAA04559>