From owner-freebsd-xen@FreeBSD.ORG Fri Feb 13 19:22:56 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04374C57 for ; Fri, 13 Feb 2015 19:22:56 +0000 (UTC) Received: from mail-pd0-f180.google.com (mail-pd0-f180.google.com [209.85.192.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C77B1E2F for ; Fri, 13 Feb 2015 19:22:55 +0000 (UTC) Received: by pdbfp1 with SMTP id fp1so21222617pdb.9 for ; Fri, 13 Feb 2015 11:22:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=3AkxivkbhgnXn1/pau7mkSIYQZ+MggQPOpcs9Opqxz0=; b=Uz8rFHiwFBcgHPJBgjhljo0xPmS2K98Qkua4hu+v46YoG/GehCPP04AAZqt3ocm4Gf tLwvWYW5UonrSBkTqKjy6QsybCvsAUD7QPdarMXyFPD+kAVJWMnjaIJb5F0d57utYPUx 1SG6vyL+w5aaMpqZWdKBiDd9277v3WLR5TwQCbEpyUfZ6xG8hV39zD9eM8rDACDuYAGl hX7q5Ke0KA8QlyTILP2uu5qCB5gJ3SmW2MV2b7x3ge5DvI3QmdlPb7Nqtr2S5l+0UO+K cWsr83B+4XJKkqPNiY/5ys6cl8bhlAsRgzHAoAW7eogt96MrN3d9kuen+b83D3HgVvq+ ++wA== X-Gm-Message-State: ALoCoQlEmlKBqvnomhW826Vegj4gY8AvQdjveoxAguBNmGCsWnQJNsy9DvCCrxyToS0z8expmxh9 MIME-Version: 1.0 X-Received: by 10.66.161.170 with SMTP id xt10mr17952838pab.14.1423854963282; Fri, 13 Feb 2015 11:16:03 -0800 (PST) Sender: andy@fud.org.nz Received: by 10.70.129.238 with HTTP; Fri, 13 Feb 2015 11:16:03 -0800 (PST) In-Reply-To: <54DE26DB.90000@citrix.com> References: <54D88BD5.7050703@citrix.com> <54DA303F.9020203@citrix.com> <54DE26DB.90000@citrix.com> Date: Sat, 14 Feb 2015 08:16:03 +1300 X-Google-Sender-Auth: 7fvujF7sPr4BQzOJIqZ85a9iAEs Message-ID: Subject: Re: xenstore memory issue From: Andrew Thompson To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Feb 2015 19:22:56 -0000 On 14 February 2015 at 05:31, Roger Pau Monn=C3=A9 w= rote: > El 13/02/15 a les 0.46, Andrew Thompson ha escrit: > > On 11 February 2015 at 05:22, Roger Pau Monn=C3=A9 > wrote: > > > >> Hello Andrew, > >> > >> El 10/02/15 a les 10.22, Andrew Thompson ha escrit: > >>> On 10 February 2015 at 16:25, Andrew Thompson > >> wrote: > >>> > >>>> On 9 February 2015 at 23:28, Roger Pau Monn=C3=A9 > >> wrote: > >>>> A bit of trial and error with dtrace has narrowed this down. I can > cause > >>>> the leak by just opening /dev/xen/xenstore > >> > >> Thanks for doing all this legwork! IMHO the best solution is to switch > >> xenstore dev to use cdevpriv in order to store each client data. What = we > >> are doing right now (storing client data in dev->si_dvr1) is plain > >> wrong. I've uploaded two patches (one for HEAD and one for stable/10) = so > >> that you can try it also, please report back whether this fixes your > >> problem or not: > >> > >> https://people.freebsd.org/~royger/xenstore_fix/ > >> > >> > > I have tested this on 10.0 and it does fix the issue. After 36000 > > allocations there are just 6 active at 13K > > > > xenstore 6 13K - 36699 16,32,64,128,256,512 > > > > > > I looked in to why this only affected one of my many VMs as they run > > xe-update-guest-attrs and nova-agent which should have triggered the > leak. > > They are all stuck on xsdread when reading from the xenstore so the > update > > loop has stopped. > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIM= E > > COMMAND > > 0 13250 13249 0 20 0 16552 1968 xsdread S - 5:57.6= 0 > > /usr/local/bin/xenstore-read domid (xenstore) > > Is this with my patch applied? AFAICT xsdread means it's waiting on the > address at dev->si_drv1, which is completely wrong because every time > the device was opened this address changed. With my patch applied it > should no longer be an issue. > > No, I was looking at the other unpatched servers, I didnt realise the two issues were linked. Please commit! cheers, Andrew