Date: Mon, 23 Mar 2009 15:44:05 -0700 From: "Anthony Bourov" <ab@addr.com> To: <freebsd-performance@freebsd.org> Subject: nsdispatch performance issue for large group files (libc) Message-ID: <7E0B3E3BC5054DF2A0BC814501B08324@ABPC>
next in thread | raw e-mail | index | archive | help
Regarding performance of: lib/libc/net/nsdispatch.c
When used from: lib/libc/net/getgrent.c (called by initgroups())
I don't normally post here but I wanted to get some feed back on a =
performance issue that I spotted. I run a large number of high-volume =
web hosting servers and noticed on some of the servers a severe decrease =
in Apache's performance when the /etc/group file is large (over 100,000 =
entries in a group file as it is combined across servers).
I did a trace and found the following operation:
stat("/etc/nsswitch.conf", {st_mode=3D052, st_size=3D4503681233059861, =
...}) =3D 0
repeating as many times as there is groups in the group file. I narrowed =
the problem down to where apache calls "initgroups()" before forking =
each process (nothing wrong here). And init groups goes through every =
entry in the group file using getgrent(), which in turn calls nsdispatch =
and which for every single call does "stat" on "/etc/nsswitch.conf" to =
see if it changed.=20
This issue impacts different servers differently, on most of the SCSI =
servers this delays apache startup my maybe a minute, however, on a Dell =
SATA raid the "stat" command was significantly slower and caused =
everything to come to a halt for several minuted every time apache =
starts.
In my opinion this is a very significant performance issue when working =
with large servers. Most programs, including apache, will call =
"initgroups()" for every time they fork, and it the group file is large =
this means as many "stat" requests on the file system as there are =
entries in the group file for every single fork() that the server does.
For myself I just made it never test "stat" on "/etc/nsswitch.conf" =
after the first time since I know that file is never modified. However, =
a better solution would be to somehow let nsdispatch know that it is =
being ran in batch mode and should not keep testing if the file has =
changed. This would effect both "getgrent" and "getpwent".=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7E0B3E3BC5054DF2A0BC814501B08324>