From owner-freebsd-stable@FreeBSD.ORG Tue Sep 13 02:20:30 2005 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9E48816A41F for ; Tue, 13 Sep 2005 02:20:30 +0000 (GMT) (envelope-from nikolai@net24.co.nz) Received: from netmon.net24.net.nz (netmon.net24.net.nz [210.55.4.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 09A6C43D48 for ; Tue, 13 Sep 2005 02:20:29 +0000 (GMT) (envelope-from nikolai@net24.co.nz) Received: from [210.55.30.50] ([210.55.30.50]) by netmon.net24.net.nz (8.11.6/8.11.6) with ESMTP id j8D2KEK67036; Tue, 13 Sep 2005 14:20:17 +1200 (NZST) (envelope-from nikolai@net24.co.nz) Message-ID: <43263764.4030808@net24.co.nz> Date: Tue, 13 Sep 2005 14:20:20 +1200 From: Nikolai Schupbach User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Brooks Davis References: <4326158B.8010502@net24.co.nz> <20050913005144.GA46178@xor.obsecurity.org> <20050913013353.GA13778@odin.ac.hmc.edu> <20050913013725.GA62591@xor.obsecurity.org> <20050913014632.GA17762@odin.ac.hmc.edu> In-Reply-To: <20050913014632.GA17762@odin.ac.hmc.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, Kris Kennaway Subject: Re: Long Format Directory Listing 15x Slower in FreeBSD 5.x X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Sep 2005 02:20:30 -0000 Brooks Davis wrote: >On Mon, Sep 12, 2005 at 09:37:27PM -0400, Kris Kennaway wrote: > > >>On Mon, Sep 12, 2005 at 06:33:53PM -0700, Brooks Davis wrote: >> >> >>>On Mon, Sep 12, 2005 at 08:51:44PM -0400, Kris Kennaway wrote: >>> >>> >>>>On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote: >>>> >>>> >>>>>Hello, >>>>> >>>>>We have been trying to migrate to FreeBSD 5.4 from an older 4.x release >>>>>for one of our busy mail servers. But we have encountered problems with >>>>>directory listings on 5.4. >>>>> >>>>>Our /var/mail directory contains approximately 8,000 files doing a long >>>>>directory listing (ls -l) takes approximately 5 min and during this time >>>>>the CPU is running near 100%, on a FSBD 4 box the same directory >>>>>contents takes less than 20 seconds to list. Even on a directory with >>>>>200 files, each file with a different owner, it will still take at least >>>>>4-5 seconds to list. The problems only seems to occur when the directory >>>>>contains files from many different users. (as in /var/mail). If I chown >>>>>all the files in the /var/mail directory to a single user the directory >>>>>listing is near instant. >>>>> >>>>>It appears it has something to do with 'ls' looking up the id's in the >>>>>password database, because if I instruct ls to display numeric IDs (ls >>>>>-ln), rather than converting to user and group names, the directory with >>>>>8,000 files, with 8,000 different owners will list instantly. >>>>> >>>>>The reason this concerns me so much is we are also having a problem with >>>>>our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while >>>>>operating on small and even 0 byte mailboxes, when there are approx five >>>>>or more concurrent POP3 or IMAP sessions. And I can't help thinking that >>>>>the two problems are related. >>>>> >>>>>Does anyone have any ideas? Has anyone else noticed this problem? If I >>>>>can't resolve it I'm most likely going to revert to using 4.11, but I'd >>>>>really like to know what is going on. (and yes I'm using UFS_DIRHASH) >>>>> >>>>> >>>>Sounds like the real problem is that you have >8000 users on your >>>>machine and lookups are taking a long time. There has been discussion >>>>of this problem and how to solve it on freebsd-questions@ several >>>>times..I think it involves making a minor change to pwd_mkdb and >>>>rebuilding. Please search the archives for that mailing list. >>>> >>>> >>>If you are using NIS and have any compat options in /etc/nsswitch.conf >>>your performance will really suck in situations like this. IIRC, the >>>compat code is worse than O(n^2) if you look up each user and the >>>non-compat code is close to O(n). I'd really like to stop generating >>>nsswitch.conf entries that use compat in 7.0. >>> >>> >>I'm pretty sure there's a problem with the database hash parameters >>used by pwd_mkdb on systems with large numbers of users..this is what >>I was alluding to above. >> >> > >That wouldn't suprised me. For that matter it could be both problems. > >-- Brook > Looks like exact problem is described here: http://www.freebsd.org/cgi/query-pr.cgi?pr=75855 I'm not using NIS, so I will try the change /etc/nsswitch.conf group and passwd lines from 'compat' to 'files' and report back. -- Nikolai.