From owner-freebsd-xen@FreeBSD.ORG  Tue Feb 10 16:22:49 2015
Return-Path: <owner-freebsd-xen@FreeBSD.ORG>
Delivered-To: freebsd-xen@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 235B12CF;
 Tue, 10 Feb 2015 16:22:49 +0000 (UTC)
Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89])
 (using TLSv1 with cipher RC4-SHA (128/128 bits))
 (Client CN "mail.citrix.com",
 Issuer "Cybertrust Public SureServer SV CA" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id E4875224;
 Tue, 10 Feb 2015 16:22:47 +0000 (UTC)
X-IronPort-AV: E=Sophos;i="5.09,551,1418083200"; d="scan'208";a="224399965"
Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.80)
 with Microsoft SMTP Server id 14.3.210.2;
 Tue, 10 Feb 2015 11:22:24 -0500
Message-ID: <54DA303F.9020203@citrix.com>
Date: Tue, 10 Feb 2015 17:22:23 +0100
From: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= <roger.pau@citrix.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9;
 rv:31.0) Gecko/20100101 Thunderbird/31.4.0
MIME-Version: 1.0
To: Andrew Thompson <thompsa@FreeBSD.org>, <freebsd-xen@freebsd.org>
Subject: Re: xenstore memory issue
References: <CAFAOGNTCJsXr0VOUHAtFoPvQO5VWSUvL_VKkxYoaCtrZdC6zUg@mail.gmail.com>	<54D88BD5.7050703@citrix.com>	<CAFAOGNTqMOF=wK79PzoOV8d8Qh-4CkHPs4rxy9gmfBEbQ9-CfA@mail.gmail.com>
 <CAFAOGNR6u3O9WhG0tjey-7thKTM8HXweK2vn4TJRLvE7OezF+A@mail.gmail.com>
In-Reply-To: <CAFAOGNR6u3O9WhG0tjey-7thKTM8HXweK2vn4TJRLvE7OezF+A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit
X-DLP: MIA2
X-BeenThere: freebsd-xen@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Discussion of the freebsd port to xen - implementation and usage
 <freebsd-xen.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-xen>,
 <mailto:freebsd-xen-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-xen/>
List-Post: <mailto:freebsd-xen@freebsd.org>
List-Help: <mailto:freebsd-xen-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-xen>,
 <mailto:freebsd-xen-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Feb 2015 16:22:49 -0000

Hello Andrew,

El 10/02/15 a les 10.22, Andrew Thompson ha escrit:
> On 10 February 2015 at 16:25, Andrew Thompson <thompsa@freebsd.org> wrote:
> 
>> On 9 February 2015 at 23:28, Roger Pau Monné <roger.pau@citrix.com> wrote:
>> A bit of trial and error with dtrace has narrowed this down. I can cause
>> the leak by just opening /dev/xen/xenstore
>>
>> int main() {
>>   open("/dev/xen/xenstore", O_RDWR, 0);
>> }
>>
>> # vmstat -m | grep xenstore; ./open; vmstat -m | grep xenstore
>>      xenstore  8739 104797K       -    56078  16,32,64,128,256,512
>>      xenstore  8740 104809K       -    56079  16,32,64,128,256,512
>>
>>
>> Using dtrace probes I can see that xs_dev_close is never called.
>>
> 
> I think I have worked this out. Rackspace use an agent called nova-agent
> which keeps and open handle on /dev/xen/xenstore. Since xenstore isnt using
> the D_TRACKCLOSE flag it will not call d_close until the last reference is
> dropped.  Since xenstore expects to malloc/free on open and close this
> assumption breaks and will leak memory.
> 
> If i stop nova-agent I can see xs_dev_close being called and the memory
> freed with testing with xenstore-read. The correct solution seems to be to
> set D_TRACKCLOSE if I understand its purpose correctly.

Thanks for doing all this legwork! IMHO the best solution is to switch
xenstore dev to use cdevpriv in order to store each client data. What we
are doing right now (storing client data in dev->si_dvr1) is plain
wrong. I've uploaded two patches (one for HEAD and one for stable/10) so
that you can try it also, please report back whether this fixes your
problem or not:

https://people.freebsd.org/~royger/xenstore_fix/

Roger.