From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 17 06:00:25 2010 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01D4F106566B; Wed, 17 Nov 2010 06:00:09 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id 052508FC17; Wed, 17 Nov 2010 06:00:09 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id 56FEA41C6DB; Wed, 17 Nov 2010 07:00:07 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id Y9KXxZ-ykPnq; Wed, 17 Nov 2010 07:00:06 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id 5C48B41C690; Wed, 17 Nov 2010 07:00:06 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 24A584448F3; Wed, 17 Nov 2010 05:57:53 +0000 (UTC) Date: Wed, 17 Nov 2010 05:57:53 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Thierry Herbelot In-Reply-To: <201011170627.28025.thierry.herbelot@free.fr> Message-ID: <20101117055208.S24596@maildrop.int.zabbadoz.net> References: <201011170627.28025.thierry.herbelot@free.fr> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-current , FreeBSD virtualization mailing list Subject: Re: VIMAGE: Freed UMA keg was not empty X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: FreeBSD virtualization mailing list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 06:00:25 -0000 On Wed, 17 Nov 2010, Thierry Herbelot wrote: Hi, first of all freebsd-virtualization@ is the better list for this; Cc:ed. > We are using FreeBSD + VIMAGE at work, and we have seen an annoying problem : > there seems to be a memory leak in the kernel, which eventually causes a > panic. > > (yes, we have seen the following message : "WARNING: VIMAGE (virtualized > network stack) is a highly experimental feature.") > > Configuring a network interface in a jail with vnet enabled, then removing > that jail causes these messages. In dmesg: > > [...] > Freed UMA keg was not empty (203 items). Lost 1 pages of memory. > Freed UMA keg was not empty (36 items). Lost 2 pages of memory. > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled (with > attached VIMAGE kernel config file). > > The following commands reproduce the bug: > > jail -l -u root -c path=/ name=foo persist vnet && > jexec foo ifconfig lo0 127.0.0.1/8 && > jail -r foo > > Running it too many times exhausts kernel memory and crashes the machine. > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN -head. The problem has been present since day 1 and is still present up to HEAD. This is about type stability and teardown. So far we are no able to actually free TCP (and UDP in SVN) states as they might still be accesses after "free". It's a general problem in the network stack and has been implemented as a measure to circumvent panics in those cases. I am not sure if that's actually what's biting you wrt to memory or if it's something else. Unfortunately boot logs and kernel configs don't help much; enable ddb and get show uma and show malloc reports after the crash (or watch vmstat -z and vmstat -m) after every 50 jail restarts. I might have some patches for a couple of things but cannot (yet) help with the additional (duplicate) UMA zones showing up at each iteration (for the -z case). /bz -- Bjoern A. Zeeb Welcome a new stage of life. Going to jail sucks -- All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html