From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 6 09:24:41 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D9FFA1065673 for ; Tue, 6 Apr 2010 09:24:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 823C98FC1F for ; Tue, 6 Apr 2010 09:24:38 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o369OT2O074194 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Apr 2010 12:24:29 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o369OT0a042117; Tue, 6 Apr 2010 12:24:29 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o369OTTB042116; Tue, 6 Apr 2010 12:24:29 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 6 Apr 2010 12:24:29 +0300 From: Kostik Belousov To: Petr Salinger Message-ID: <20100406092429.GV2415@deviant.kiev.zoral.com.ua> References: <20100403165238.GO2415@deviant.kiev.zoral.com.ua> <20100403171945.GQ2415@deviant.kiev.zoral.com.ua> <20100403210829.GW2415@deviant.kiev.zoral.com.ua> <20100405185924.GS2415@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="xaJmculcd5PV2weY" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.8 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-hackers@freebsd.org Subject: Re: leak of the vnodes X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Apr 2010 09:24:42 -0000 --xaJmculcd5PV2weY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Apr 06, 2010 at 11:56:28AM +0200, Petr Salinger wrote: > >>I would expect that sum of mnt_nvnodelistsize should be vfs.numvnodes. > >>The sum is at about 3400, but the vfs.numvnodes is at about 38000. > >>Is my expectation correct ? > >Not quite, reclaimed vnode is removed from mp list. >=20 > Are they in some other list ? No. Some code path does not properly release hold count on the vnode, most likely. Vnode is added to the free list when its hold count reaches zero and vnode is not doomed, doomed vnode is just freed. >=20 > >Can you check > >that vmstat -z | grep VNODE output coincides with sysctl vfs.numvnodes ? >=20 > I would say yes: >=20 > kern.maxvnodes: 31000 > kern.minvnodes: 2000 > vfs.freevnodes: 21 > vfs.wantfreevnodes: 2000 > vfs.numvnodes: 31011 > debug.vnlru_nowhere: 261 >=20 > ITEM SIZE LIMIT USED FREE REQUESTS =20 > FAILURES > VNODE: 480, 0, 30999, 777, 248903, = =20 > 0 > VNODEPOLL: 112, 0, 80, 85, 80, = =20 > 0 >=20 > But still: >=20 > 0xffffff0003939be0 /dev/ad0s2a on / (ufs) > mnt_nvnodelistsize =3D 570 >=20 > 0xffffff000393a000 devfs on /dev (devfs) > mnt_nvnodelistsize =3D 41 >=20 > 0xffffff0003939000 /dev/ad0s2d on /opt/sid/build (ufs) > mnt_nvnodelistsize =3D 52 >=20 > There is 30000 vnodes somewhere. Right. The leak is there. >=20 > >Also, can you unmount linprocfs before the test and see whether it > >leaks as before ? > >Another thing to try is set sysctl debug.disablefullpath to 1. >=20 > Done both before this test, still "hangs/leaks": >=20 > vfs.cache.numfullpathfound: 29075 > vfs.cache.numfullpathfail4: 0 > vfs.cache.numfullpathfail2: 0 > vfs.cache.numfullpathfail1: 0 > vfs.cache.numfullpathcalls: 29075 > debug.disablefullpath: 1 Can you try to narrow down the sequence of operations that is needed to reproduce the leak ? If not, you may add vn_printf() + ddb_backtrace to vgonel(), but this definitely cause immense amount of spam on the console. I am not sure how easy would it be to spot the leaking vnodes. --xaJmculcd5PV2weY Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAku6/cwACgkQC3+MBN1Mb4gHWgCcCkLZCD1xOODcLwhDWO7Bp3/4 iDAAoNnK3q/zSRVbaFFx7zmHQ+1OtnFv =CMvx -----END PGP SIGNATURE----- --xaJmculcd5PV2weY--