Date: Tue, 10 Feb 2015 16:25:26 +1300 From: Andrew Thompson <thompsa@FreeBSD.org> To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= <roger.pau@citrix.com>, freebsd-xen@freebsd.org Subject: Re: xenstore memory issue Message-ID: <CAFAOGNTqMOF=wK79PzoOV8d8Qh-4CkHPs4rxy9gmfBEbQ9-CfA@mail.gmail.com> In-Reply-To: <54D88BD5.7050703@citrix.com> References: <CAFAOGNTCJsXr0VOUHAtFoPvQO5VWSUvL_VKkxYoaCtrZdC6zUg@mail.gmail.com> <54D88BD5.7050703@citrix.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 9 February 2015 at 23:28, Roger Pau Monn=C3=A9 <roger.pau@citrix.com> wr= ote: > Hello, > > El 09/02/15 a les 10.39, Andrew Thompson ha escrit: > > Hi, > > > > > > I have three VMs with Rackspace and one is behaving oddly with xenstore > > memory consumption. Here are the kernel versions and vmstat -m results. > > > > As you can see the third VM is using 100MB in xenstore memory and it seem= s > > to be climbing by 1-2MB per hour. Eventually all the processes go in to > > pfault state and it grinds to a halt. > > That's certainly weird, are you doing something different on this VM as > compared to the others? Did you hot-add a nic, disk or ballooned memory? > > Has the VM been saved/restored or migrated? > > Tracking down this kind of xenstore leaks can be difficult without > having a way to reproduce them. > > A bit of trial and error with dtrace has narrowed this down. I can cause the leak by just opening /dev/xen/xenstore int main() { open("/dev/xen/xenstore", O_RDWR, 0); } # vmstat -m | grep xenstore; ./open; vmstat -m | grep xenstore xenstore 8739 104797K - 56078 16,32,64,128,256,512 xenstore 8740 104809K - 56079 16,32,64,128,256,512 Using dtrace probes I can see that xs_dev_close is never called. # dtrace -n 'fbt::xs_dev_open:{} fbt::xs_dev_close: {} dtmalloc::xenstore: {}' # ./open CPU FUNCTION 0 -> xs_dev_open 0 | xenstore:malloc 0 <- xs_dev_open This is on 10.0-RELEASE-p12. I get the same result with `/usr/local/bin/xenstore-read domid` but the above c program is enough too. regards, Andrew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFAOGNTqMOF=wK79PzoOV8d8Qh-4CkHPs4rxy9gmfBEbQ9-CfA>