From owner-freebsd-xen@FreeBSD.ORG Mon Feb 9 10:28:44 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0DFD3CDA; Mon, 9 Feb 2015 10:28:44 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Cybertrust Public SureServer SV CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1CD7BBCB; Mon, 9 Feb 2015 10:28:42 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.09,543,1418083200"; d="scan'208";a="224018044" Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.80) with Microsoft SMTP Server id 14.3.210.2; Mon, 9 Feb 2015 05:28:38 -0500 Message-ID: <54D88BD5.7050703@citrix.com> Date: Mon, 9 Feb 2015 11:28:37 +0100 From: =?windows-1252?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Andrew Thompson , Subject: Re: xenstore memory issue References: In-Reply-To: Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Feb 2015 10:28:44 -0000 Hello, El 09/02/15 a les 10.39, Andrew Thompson ha escrit: > Hi, > > > I have three VMs with Rackspace and one is behaving oddly with xenstore > memory consumption. Here are the kernel versions and vmstat -m results. > > FreeBSD us.e.com 10.0-RELEASE-p9 FreeBSD 10.0-RELEASE-p9 #0: Mon Sep 15 > 14:35:52 UTC 2014 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > > xenbus 16 2K - 86 16,32,64,256 > xenstore 409 4837K - 38424052 16,32,64,128,256 > xen_hvm 2 8K - 2 4096 > xen_intr 25 4K - 25 128 > > > FreeBSD uk.e.com 10.0-RELEASE-p12 FreeBSD 10.0-RELEASE-p12 #0: Tue Nov 4 > 05:07:17 UTC 2014 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > > xenbus 11 2K - 83 16,32,64,256 > xenstore 198 2317K - 43428137 16,32,64,128,256,512 > xen_hvm 2 8K - 2 4096 > xen_intr 24 3K - 24 128 > > > FreeBSD au.e.com 10.0-RELEASE-p12 FreeBSD 10.0-RELEASE-p12 #0: Tue Nov 4 > 05:07:17 UTC 2014 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > > xenbus 11 2K - 83 16,32,64,256 > xenstore 8477 101653K - 55249 16,32,64,128,256,512 > xen_hvm 2 8K - 2 4096 > xen_intr 14 2K - 14 128 > > > As you can see the third VM is using 100MB in xenstore memory and it seems > to be climbing by 1-2MB per hour. Eventually all the processes go in to > pfault state and it grinds to a halt. That's certainly weird, are you doing something different on this VM as compared to the others? Did you hot-add a nic, disk or ballooned memory? Has the VM been saved/restored or migrated? Tracking down this kind of xenstore leaks can be difficult without having a way to reproduce them. > > How should I be debugging this? Is it either a local leak or the Xen host > is to blame? Even if the host is doing something weird we should be able to cope with it, or at least detect it. Roger.