Date: Tue, 10 Feb 2015 22:22:35 +1300 From: Andrew Thompson <thompsa@FreeBSD.org> To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= <roger.pau@citrix.com>, freebsd-xen@freebsd.org Subject: Re: xenstore memory issue Message-ID: <CAFAOGNR6u3O9WhG0tjey-7thKTM8HXweK2vn4TJRLvE7OezF%2BA@mail.gmail.com> In-Reply-To: <CAFAOGNTqMOF=wK79PzoOV8d8Qh-4CkHPs4rxy9gmfBEbQ9-CfA@mail.gmail.com> References: <CAFAOGNTCJsXr0VOUHAtFoPvQO5VWSUvL_VKkxYoaCtrZdC6zUg@mail.gmail.com> <54D88BD5.7050703@citrix.com> <CAFAOGNTqMOF=wK79PzoOV8d8Qh-4CkHPs4rxy9gmfBEbQ9-CfA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10 February 2015 at 16:25, Andrew Thompson <thompsa@freebsd.org> wrote:
> On 9 February 2015 at 23:28, Roger Pau Monn=C3=A9 <roger.pau@citrix.com> =
wrote:
>
>> Hello,
>>
>> El 09/02/15 a les 10.39, Andrew Thompson ha escrit:
>> > Hi,
>> >
>> >
>> > I have three VMs with Rackspace and one is behaving oddly with xenstor=
e
>> > memory consumption. Here are the kernel versions and vmstat -m results=
.
>> >
>>
> > As you can see the third VM is using 100MB in xenstore memory and it
>> seems
>> > to be climbing by 1-2MB per hour. Eventually all the processes go in t=
o
>> > pfault state and it grinds to a halt.
>>
>> That's certainly weird, are you doing something different on this VM as
>> compared to the others? Did you hot-add a nic, disk or ballooned memory?
>>
>> Has the VM been saved/restored or migrated?
>>
>> Tracking down this kind of xenstore leaks can be difficult without
>> having a way to reproduce them.
>>
>>
> A bit of trial and error with dtrace has narrowed this down. I can cause
> the leak by just opening /dev/xen/xenstore
>
> int main() {
> open("/dev/xen/xenstore", O_RDWR, 0);
> }
>
> # vmstat -m | grep xenstore; ./open; vmstat -m | grep xenstore
> xenstore 8739 104797K - 56078 16,32,64,128,256,512
> xenstore 8740 104809K - 56079 16,32,64,128,256,512
>
>
> Using dtrace probes I can see that xs_dev_close is never called.
>
I think I have worked this out. Rackspace use an agent called nova-agent
which keeps and open handle on /dev/xen/xenstore. Since xenstore isnt using
the D_TRACKCLOSE flag it will not call d_close until the last reference is
dropped. Since xenstore expects to malloc/free on open and close this
assumption breaks and will leak memory.
If i stop nova-agent I can see xs_dev_close being called and the memory
freed with testing with xenstore-read. The correct solution seems to be to
set D_TRACKCLOSE if I understand its purpose correctly.
regards,
Andrew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFAOGNR6u3O9WhG0tjey-7thKTM8HXweK2vn4TJRLvE7OezF%2BA>