Date: Tue, 2 Dec 1997 19:30:31 GMT From: nick@foobar.org To: FreeBSD-gnats-submit@FreeBSD.ORG Cc: nick@foobar.org Subject: bin/5193: Long rebuild times for pwd_mkdb with large master.passwd files Message-ID: <199712021930.TAA02446@iol.ie> Resent-Message-ID: <199712021940.LAA26928@hub.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 5193 >Category: bin >Synopsis: It takes ages to rebuild master.passwd if it's large. >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Class: change-request >Submitter-Id: current-users >Arrival-Date: Tue Dec 2 11:40:01 PST 1997 >Last-Modified: >Originator: Nick Hilliard >Organization: Ireland On-Line >Release: FreeBSD 2.2.5-RELEASE i386 >Environment: FreeBSD 2.2.5-RELEASE with large master.passwd file >Description: By default, FreeBSD doesn't cope well with rebuilding large passwd files. On an unloaded P5/200 with 30000 passwd entries with the default DB cache size of 2Mb, it takes almost 30 minutes to do a complete passwd rebuild. Increasing the cache size dramatically reduces the rebuild time. I've included a patch below for implementing a "-s" option which will specify the cache size in megabytes. This leads to the following results: # pwd_mkdb -p -s 2 /etc/master.passwd 14.772u 41.534s 26:11.28 3.5% 16+5850k 57+97145io 0pf+0w # pwd_mkdb -p -s 4 /etc/master.passwd 13.720u 27.744s 17:45.82 3.8% 16+11227k 8+65377io 0pf+0w # pwd_mkdb -p -s 6 /etc/master.passwd 11.786u 15.848s 9:49.47 4.6% 16+16253k 8+36842io 0pf+0w # pwd_mkdb -p -s 8 /etc/master.passwd 10.224u 7.435s 4:30.88 6.5% 16+20428k 7+17582io 0pf+0w # pwd_mkdb -p -s 10 /etc/master.passwd 9.927u 4.358s 2:34.70 9.2% 17+24852k 9+10910io 0pf+0w # pwd_mkdb -p -s 12 /etc/master.passwd 10.253u 3.449s 1:45.71 12.9% 17+28356k 7+8043io 0pf+0w # pwd_mkdb -p -s 14 /etc/master.passwd 10.814u 2.894s 1:24.57 16.1% 16+30982k 7+6657io 0pf+0w # pwd_mkdb -p -s 16 /etc/master.passwd 12.406u 3.128s 1:13.88 21.0% 16+35270k 7+6153io 0pf+0w # pwd_mkdb -p -s 18 /etc/master.passwd 12.515u 3.148s 1:12.56 21.5% 16+37672k 18+6303io 0pf+0w # pwd_mkdb -p -s 20 /etc/master.passwd 13.733u 3.255s 1:17.34 21.9% 17+42232k 51+6223io 0pf+0w As you can see, the rebuild time decreases assymptotically. Is there any interest in putting this into the default distribution? >How-To-Repeat: Create large master.passwd file, type pwd_mkdb -p /etc/master.passwd, and go on vacation for several weeks while waiting for the code to execute. >Fix: diff -u -r1.1 pwd_mkdb.8 --- pwd_mkdb.8 1997/12/02 18:59:13 1.1 +++ pwd_mkdb.8 1997/12/02 19:14:03 @@ -76,6 +76,14 @@ Only update the record for the specified user. Utilities that operate on a single user can use this option to avoid the overhead of rebuilding the entire database. +.It Fl s Ar cachesize +Specify in megabytes the size of the memory cache used by the +hashing library. On systems with a large user base, a small cache +size can lead to prohibitively long database file rebuild times. +As a rough guide, the memory usage of +.Nm pwd_mkdb +in megabytes will be a little bit more than twice the figure +specified here. The default is 2 megabytes. .El .Pp The two databases differ in that the secure version contains the user's diff -u -r1.1 pwd_mkdb.c --- pwd_mkdb.c 1997/12/02 18:59:13 1.1 +++ pwd_mkdb.c 1997/12/02 18:59:18 @@ -110,7 +110,7 @@ strcpy(prefix, _PATH_PWD); makeold = 0; username = NULL; - while ((ch = getopt(argc, argv, "cd:pu:v")) != -1) + while ((ch = getopt(argc, argv, "cd:ps:u:v")) != -1) switch(ch) { case 'c': /* verify only */ cflag = 1; @@ -123,6 +123,9 @@ break; case 'u': /* only update this record */ username = optarg; + break; + case 's': /* change default cachesize */ + openinfo.cachesize = atoi(optarg) * 1024 * 1024; break; case 'v': /* backward compatible */ break; >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199712021930.TAA02446>