From owner-freebsd-stable@FreeBSD.ORG Wed May 8 07:31:07 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8B53C5FA; Wed, 8 May 2013 07:31:07 +0000 (UTC) (envelope-from goran.lowkrantz@ismobile.com) Received: from mail.ismobile.com (mail.ismobile.com [IPv6:2a00:f680:101:11::4]) by mx1.freebsd.org (Postfix) with ESMTP id 0C37CAEB; Wed, 8 May 2013 07:31:07 +0000 (UTC) Received: from mail.ismobile.com (localhost [127.0.0.1]) by dkim.mail.ismobile.com (Postfix) with ESMTP id D8E142B5578; Wed, 8 May 2013 07:30:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=ismobile.com; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=selector1; bh=0a4+mqw Bld+MWKhNky6TKgbKK/Q=; b=dsuk3hV5slkAjbah7mS6TsadcknyEcWY2j3zUOz jr4l0FazSMmTuS2P7W5xBOZePVDtdkoYy0otXgm5//hBqH1OCEhL78fPBIkj3No9 o4p8EDYxQbIgt7IuSlBcQCaxv4n8n6M3kw6TrD6R0arO7EAhKVTCaVlEjHF2f9Zc qwRk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=ismobile.com; h=date:from:to :cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=selector1; b=K 4r9iY6VSbizRvmtrVdFSQ5Bez1WsWOAxivzFgnEt2v2AWEl9jWE/PZ9cnfxcWKWu 0SJ4r/yURzOE0Q+z/uryV6rLrTQSvQ5rXJC6umzwpuVfr+EIfhTw3li8VmiIiWn0 OgYC5603bIMS7dg0hYwunTzI1TqKesu0+UevOaHhhI= Received: from [172.16.2.45] (glz-macbookpro.hq.ismobile.com [172.16.2.45]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.ismobile.com (Postfix) with ESMTPSA id 739082B5573; Wed, 8 May 2013 07:30:58 +0000 (UTC) Date: Wed, 08 May 2013 09:30:57 +0200 From: =?UTF-8?Q?G=C3=B6ran_L=C3=B6wkrantz?= To: Dewayne Geraghty , 'Mikolaj Golub' Subject: RE: Nullfs leaks i-nodes Message-ID: <2FBC9C8F12387387C1AEF445@[172.16.2.45]> In-Reply-To: <56EF269F84824D8DB413D289BB8CBE19@as.lan> References: <20130507204149.GA3267@gmail.com> <56EF269F84824D8DB413D289BB8CBE19@as.lan> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Cc: freebsd-stable@freebsd.org, 'Kostik Belousov' X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 May 2013 07:31:07 -0000 --On May 8, 2013 8:35:18 +1000 Dewayne Geraghty=20 wrote: >> -----Original Message----- >> From: owner-freebsd-stable@freebsd.org >> [mailto:owner-freebsd-stable@freebsd.org] On Behalf Of Mikolaj Golub >> Sent: Wednesday, 8 May 2013 6:42 AM >> To: G=C3=B6ran L=C3=B6wkrantz >> Cc: Kostik Belousov; freebsd-stable@freebsd.org >> Subject: Re: Nullfs leaks i-nodes >> >> On Tue, May 07, 2013 at 08:30:06AM +0200, G=C3=B6ran L=C3=B6wkrantz = wrote: >> > I created a PR, kern/178238, on this but would like to know >> if anyone has >> > any ideas or patches? >> > >> > Have updated the system where I see this to FreeBSD >> 9.1-STABLE #0 r250229 >> > and still have the problem. >> >> I am observing an effect that might look like inode leak, which I >> think is due free nullfs vnodes caching, recently added by kib >> (r240285): free inode number does not increase after unlink; but if I >> purge the free vnodes cache (temporary setting vfs.wantfreevnodes to 0 >> and observing vfs.freevnodes decreasing to 0) the inode number grows >> back. >> >> You have only about 1000 inodes available on your underlying fs, while >> vfs.wantfreevnodes I think is much higher, resulting in running out of >> i-nodes. >> >> If it is really your case you can disable caching, mounting nullfs >> with nocache (it looks like caching is not important in your case). >> >> -- >> Mikolaj Golub >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to >> "freebsd-stable-unsubscribe@freebsd.org" >> > > Hi Goran, > > After I included Kib's vnode caching patch the performance on my "port > builder" machine, decreased significantly. The "port builder" is one of > many jails and nullfs is used extensively. I was starving the system of > vnodes. Increasing the kern.maxvnodes, resulted in better performance > than the original system configuration without vnode caching. Thanks Kib > :) > > I don't think you'll run out of vnodes as it is self adjusting (that was > my concern too) > > I changed kern.maxvnode to approx 3 times what it wanted and tuned for my > needs. Try it and keep an eye on: > sysctl vfs.numvnodes vfs.wantfreevnodes vfs.freevnodes > vm.stats.vm.v_vnodepgsout vm.stats.vm.v_vnodepgsin > > Regards, Dewayne > Hi Dewayne, I got a few of those too but I didn't connect them with the FW problem as=20 here there seems to be reclaim pressure. On the FW I get these numbers: vfs.numvnodes: 7500 vfs.wantfreevnodes: 27936 vfs.freevnodes: 5663 vm.stats.vm.v_vnodepgsout: 0 vm.stats.vm.v_vnodepgsin: 4399 while on the jail systems I get something like this: vfs.numvnodes: 51212 vfs.wantfreevnodes: 35668 vfs.freevnodes: 35665 vm.stats.vm.v_vnodepgsout: 5952 vm.stats.vm.v_vnodepgsin: 939563 and as far as I can understand, the fact that vfs.wantfreevnodes and=20 vfs.freevnodes are almost the same suggests that we have a reclaim = pressure. So one fix for small NanoBSD systems would be to lower vfs.wantfreevnodes=20 and I will test that on a virtual machine and see if I can get better=20 reclaim. MVH G=C3=B6ran