From owner-freebsd-hackers Sun Jun 23 19:25:56 1996 Return-Path: owner-hackers Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id TAA13589 for hackers-outgoing; Sun, 23 Jun 1996 19:25:56 -0700 (PDT) Received: from post.io.org (post.io.org [198.133.36.6]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id TAA13567; Sun, 23 Jun 1996 19:25:37 -0700 (PDT) Received: from zap.io.org (taob@zap.io.org [198.133.36.81]) by post.io.org (8.7.5/8.7.3) with SMTP id WAA06239; Sun, 23 Jun 1996 22:22:33 -0400 (EDT) Date: Sun, 23 Jun 1996 22:23:31 -0400 (EDT) From: Brian Tao Reply-To: Brian Tao To: FREEBSD-HACKERS-L cc: FREEBSD-ISP-L Subject: Increasing openinfo.cachesize in pwd_mkdb (was Re: Incremental [s]pwd.db updates?) In-Reply-To: <199606152001.WAA24925@gvr.win.tue.nl> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-hackers@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk I did some simple benchmarking with a modified pwd_mkdb that allows me to specify the size of the cache. It defaults to 2MB, which is less than optimal if you have more than a few thousand users in /etc/master.passwd (like many ISP's). Server is a lightly loaded P133 with 128MB doing primary DNS and a bit of SMTP/POP3 service. There is typically 80 to 90MB free after the OS, named and various user processes have taken their chunk. The first test is with an actual 14241-line passwd file generated from one of our customer databases (we have a system that creates master.passwd and RADIUS files from a billing database). # time /root/bin/pwd_mkdb -p -d . master.passwd 9.024u 15.201s 4:26.16 9.0% 10+2799k 0+17624io 0pf+0w # time /root/bin/pwd_mkdb -c 3072 -p -d . master.passwd 7.724u 8.949s 2:36.64 10.6% 10+4022k 3+10081io 0pf+0w # time /root/bin/pwd_mkdb -c 4096 -p -d . master.passwd 6.957u 3.041s 0:58.69 17.0% 10+4767k 3+3565io 0pf+0w # time /root/bin/pwd_mkdb -c 8192 -p -d . master.passwd 7.110u 1.645s 0:30.44 28.7% 10+7600k 3+1941io 0pf+0w # ls -l total 14600 -rw------- 1 root wheel 1192261 Jun 23 21:50 master.passwd -rw-r--r-- 1 root wheel 950366 Jun 23 22:04 passwd -rw-r--r-- 1 root wheel 9220096 Jun 23 22:04 pwd.db -rw------- 1 root wheel 9666560 Jun 23 22:05 spwd.db The second test is with an artificially generated 150,000-line passwd file, and allocating 64MB of core to pwd_mkdb: # /root/bin/pwd_mkdb -c 65536 -p -d . big-pass 83.962u 16.826s 4:39.83 36.0% 10+60692k 1753+22470io 0pf+0w # ls -l total 134576 -rw------- 1 root wheel 9300000 Jun 23 21:06 master.passwd -rw-r--r-- 1 root wheel 6750000 Jun 23 22:12 passwd -rw-r--r-- 1 root wheel 70475776 Jun 23 22:12 pwd.db -rw------- 1 root wheel 73740288 Jun 23 22:14 spwd.db With enough memory, I can now rehash a passwd file over 10 times the size in roughly the same amount of time. The trick is to know how much RAM you can dedicate to one process, without causing it to hit swap. The diffs to support the -c switch are quite simple, if anyone is interested in them (maybe even committing them?). I think it would be a perfect compliment to gvr's enhancements to passwd/chpass/chfn/chsh. I still need to edit the man page and patch vipw to pass a cachesize parameter to pwd_mkdb. -- Brian Tao (BT300, taob@io.org, taob@ican.net) Systems and Network Administrator, Internet Canada Corp. "Though this be madness, yet there is method in't"