From owner-freebsd-current@FreeBSD.ORG Wed Nov 17 12:57:17 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0712E1065694; Wed, 17 Nov 2010 12:57:17 +0000 (UTC) (envelope-from zec@icir.org) Received: from munja.zvne.fer.hr (munja.zvne.fer.hr [161.53.66.248]) by mx1.freebsd.org (Postfix) with ESMTP id 8882F8FC17; Wed, 17 Nov 2010 12:57:16 +0000 (UTC) Received: from sluga.fer.hr ([161.53.66.244]) by munja.zvne.fer.hr with Microsoft SMTPSVC(6.0.3790.4675); Wed, 17 Nov 2010 13:45:11 +0100 Received: from localhost ([161.53.19.8]) by sluga.fer.hr with Microsoft SMTPSVC(6.0.3790.4675); Wed, 17 Nov 2010 13:45:10 +0100 From: Marko Zec To: freebsd-virtualization@freebsd.org Date: Wed, 17 Nov 2010 13:45:06 +0100 User-Agent: KMail/1.9.10 References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> In-Reply-To: <20101117055208.S24596@maildrop.int.zabbadoz.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201011171345.06789.zec@icir.org> X-OriginalArrivalTime: 17 Nov 2010 12:45:11.0037 (UTC) FILETIME=[45BA4AD0:01CB8655] Cc: Thierry Herbelot , "Bjoern A. Zeeb" , freebsd-current Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 12:57:17 -0000 On Wednesday 17 November 2010 06:57:53 Bjoern A. Zeeb wrote: > On Wed, 17 Nov 2010, Thierry Herbelot wrote: > > Hi, > > first of all freebsd-virtualization@ is the better list for this; Cc:ed. > > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying > > problem : there seems to be a memory leak in the kernel, which eventually > > causes a panic. > > > > (yes, we have seen the following message : "WARNING: VIMAGE (virtualized > > network stack) is a highly experimental feature.") > > > > Configuring a network interface in a jail with vnet enabled, then > > removing that jail causes these messages. In dmesg: > > > > [...] > > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > > > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled > > (with attached VIMAGE kernel config file). > > > > The following commands reproduce the bug: > > > > jail -l -u root -c path=/ name=foo persist vnet && > > jexec foo ifconfig lo0 127.0.0.1/8 && > > jail -r foo > > > > Running it too many times exhausts kernel memory and crashes the machine. > > > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN > > -head. > > The problem has been present since day 1 and is still present up to > HEAD. This is about type stability and teardown. So far we are no > able to actually free TCP (and UDP in SVN) states as they might still > be accesses after "free". It's a general problem in the network stack > and has been implemented as a measure to circumvent panics in those > cases. Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. Note that the original VIMAGE prototype on FreeBSD 4.x from 2002 or 2003 used (mostly) unified memory zones, so there's no technical reason why this couldn't work again in FreeBSD current or 8-stable. Cheers, Marko > I am not sure if that's actually what's biting you wrt to memory or > if it's something else. Unfortunately boot logs and kernel configs > don't help much; enable ddb and get show uma and show malloc reports > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail > restarts. I might have some patches for a couple of things but cannot > (yet) help with the additional (duplicate) UMA zones showing up at > each iteration (for the -z case). > > /bz