From owner-freebsd-current@FreeBSD.ORG Sat Jan 1 16:46:56 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F85A1065672; Sat, 1 Jan 2011 16:46:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id AE56C8FC08; Sat, 1 Jan 2011 16:46:55 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p01GkohM029360 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 1 Jan 2011 18:46:50 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p01GknCt076185; Sat, 1 Jan 2011 18:46:49 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p01GknmR076184; Sat, 1 Jan 2011 18:46:49 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 1 Jan 2011 18:46:49 +0200 From: Kostik Belousov To: Beat G?tzi Message-ID: <20110101164649.GY90883@deviant.kiev.zoral.com.ua> References: <4D1F1AE8.5040704@chruetertee.ch> <20110101151008.GA7762@freebsd.org> <4D1F4A48.6080604@chruetertee.ch> <20110101154537.GW90883@deviant.kiev.zoral.com.ua> <4D1F4FB8.3030303@chruetertee.ch> <20110101161254.GX90883@deviant.kiev.zoral.com.ua> <4D1F5992.8030309@chruetertee.ch> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wmVhWgj1OZ04es6B" Content-Disposition: inline In-Reply-To: <4D1F5992.8030309@chruetertee.ch> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Alexander Best , current@freebsd.org Subject: Re: Suddenly slow lstat syscalls on CURRENT from Juli X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Jan 2011 16:46:56 -0000 --wmVhWgj1OZ04es6B Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 01, 2011 at 05:42:58PM +0100, Beat G?tzi wrote: > On 01.01.2011 17:12, Kostik Belousov wrote: > > On Sat, Jan 01, 2011 at 05:00:56PM +0100, Beat G?tzi wrote: > >> On 01.01.2011 16:45, Kostik Belousov wrote: > >>> Check the output of sysctl kern.maxvnodes and vfs.numvnodes. I suspect > >>> they are quite close or equial. If yes, consider increasing maxvnodes. > >>> Another workaround, if you have huge nested directories hierarhy, is > >>> to set vfs.vlru_allow_cache_src to 1. > >> > >> Thanks for the hint. kern.maxvnodes and vfs.numvnodes were equal: > >> # sysctl kern.maxvnodes vfs.numvnodes > >> kern.maxvnodes: 100000 > >> vfs.numvnodes: 100765 > >> > >> I've increased kern.maxvnodes and the problem was gone until > >> vfs.numvnodes reached the value of kern.maxvnodes again: > >> # sysctl kern.maxvnodes vfs.numvnodes > >> kern.maxvnodes: 150000 > >> vfs.numvnodes: 150109 > > The processes should be stuck in "vlruwk" state, that can be > > checked with ps or '^T' on the terminal. >=20 > Yes, there are various processes in "vlruwk" state, >=20 > >> As the directory structure is quite huge on this server I've set > >> vfs.vlru_allow_cache_src to one now. > > Did it helped ? >=20 > No, it doesn't looks like setting vfs.vlru_allow_cache_src helped. The > problem was gone when I increased kern.maxvnodes until vfs.numvnodes > reached that level. I've stopped all running deamons but numvnodes > doesn't decrease. Stopping the daemons would not decrease the count of cached vnodes. What you can do is to call unmount on the filesystems. Supposedly, the filesystems are busy and unmount shall fail, but it will force freed the vnodes that are unused by any process. --wmVhWgj1OZ04es6B Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk0fWnkACgkQC3+MBN1Mb4hnfQCeKuWmedxT+a+27Ez1rmgU8/zh wRgAnA6F8LmqaCJdSDHjoqp+eha/Cfth =ikth -----END PGP SIGNATURE----- --wmVhWgj1OZ04es6B--