From owner-freebsd-current@FreeBSD.ORG Wed Nov 17 06:30:08 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DDB2B106566B for ; Wed, 17 Nov 2010 06:30:08 +0000 (UTC) (envelope-from thierry.herbelot@free.fr) Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [212.27.42.6]) by mx1.freebsd.org (Postfix) with ESMTP id C9D9B8FC14 for ; Wed, 17 Nov 2010 06:30:05 +0000 (UTC) Received: from mail.herbelot.nom (unknown [82.227.159.103]) by smtp6-g21.free.fr (Postfix) with ESMTP id 0FDA38224B; Wed, 17 Nov 2010 07:30:00 +0100 (CET) Received: from tulipe.herbelot.nom (tulipe.herbelot.nom [192.168.2.5]) by mail.herbelot.nom (8.14.1/8.14.1) with ESMTP id oAH6TlFe020404; Wed, 17 Nov 2010 07:29:48 +0100 (CET) From: Thierry Herbelot To: freebsd-current@freebsd.org, FreeBSD virtualization mailing list , "Bjoern A. Zeeb" Date: Wed, 17 Nov 2010 07:29:41 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-STABLE; KDE/4.5.3; amd64; ; ) References: <201011170627.28025.thierry.herbelot@free.fr> <20101117055208.S24596@maildrop.int.zabbadoz.net> In-Reply-To: <20101117055208.S24596@maildrop.int.zabbadoz.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: <201011170729.41894.thierry.herbelot@free.fr> Cc: Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 06:30:08 -0000 "Bjoern A. Zeeb" a =E9crit > On Wed, 17 Nov 2010, Thierry Herbelot wrote: >=20 > Hi, >=20 > first of all freebsd-virtualization@ is the better list for this; Cc:ed. (in fact, I did not know where else to send this message : -net, ... ? than= ks=20 for the CC) >=20 > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying > > problem : there seems to be a memory leak in the kernel, which > > eventually causes a panic. > >=20 > > (yes, we have seen the following message : "WARNING: VIMAGE (virtualized > > network stack) is a highly experimental feature.") > >=20 > > Configuring a network interface in a jail with vnet enabled, then > > removing that jail causes these messages. In dmesg: > >=20 > > [...] > > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > >=20 > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled > > (with attached VIMAGE kernel config file). > >=20 > > The following commands reproduce the bug: > >=20 > > jail -l -u root -c path=3D/ name=3Dfoo persist vnet && > > jexec foo ifconfig lo0 127.0.0.1/8 && > > jail -r foo > >=20 > > Running it too many times exhausts kernel memory and crashes the machin= e. > >=20 > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN > > -head. >=20 > The problem has been present since day 1 and is still present up to > HEAD. This is about type stability and teardown. So far we are no > able to actually free TCP (and UDP in SVN) states as they might still > be accesses after "free". It's a general problem in the network stack > and has been implemented as a measure to circumvent panics in those > cases. >=20 > I am not sure if that's actually what's biting you wrt to memory or > if it's something else. Unfortunately boot logs and kernel configs > don't help much; enable ddb and get show uma and show malloc reports > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail > restarts. I might have some patches for a couple of things but cannot > (yet) help with the additional (duplicate) UMA zones showing up at > each iteration (for the -z case). The context for our problem is that VIMAGE jails are mant to be used "in=20 production" for automated tests, thus are likely to be multiple on one=20 physical host, and are likely to be started and stopped according to our=20 needs. We have tried more "classical" virtualization solutions (qemu, kvm, ... ,=20 normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems = to=20 be the correct answer. (for other reasons, an i386 version of the kernel mu= st=20 be used) The point of my original email was to simplify the configuration for a test= =20 setup : any machine with the attached configuration file and the above=20 sequence of commands creating and deleting jails, running in a loop, will=20 eventually panic. Anyway, I will follow up with the logs you mentioned. Cheers (and thanks for the quick feedback) Thierry Herbelot >=20 > /bz