From owner-freebsd-xen@FreeBSD.ORG Thu Feb 12 23:46:30 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D014FE9D for ; Thu, 12 Feb 2015 23:46:30 +0000 (UTC) Received: from mail-pd0-f177.google.com (mail-pd0-f177.google.com [209.85.192.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A0CACF08 for ; Thu, 12 Feb 2015 23:46:30 +0000 (UTC) Received: by pdno5 with SMTP id o5so15302817pdn.8 for ; Thu, 12 Feb 2015 15:46:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=810GG/aDHPTV/d4qW50T+RtXoWR/Gl6LpG/xo7CBbec=; b=m97tgzVB4kWFJ/PEs4Ye4E2OpfgNpAoqy61kDRBvpk4/DJS8CUpI5I4UgnS5EXlvoC Zw5mS/JAXj2OjbpjZLDMsDhZLdtJVbWRMmcq6iy5kR2WM2RPDCgnQtRxYYu75KmfK//3 KtuvHcWHb7c92lDQ9xix1OcGKgNDnD+Pz3c8oj/sdjvsdQ5nqESwGIkP0IrXABCLNStg HbHAi+t/qJQTI5hS6jn1tTsjUI5YWa+2wdVwG0xRiAJ1qjYsG7e/uDAAWP3eCn7z2Pji jrXMUkq1Uf4x59Srqk2Sq4Jc50KzBTgoPCm7xmOQ5PqjjVQXz7QuCFemxTlnvQoPzINs QRtg== X-Gm-Message-State: ALoCoQmIRSP6twdE7RXlW3e9G1N88bJ7u2auNCTT7a0Z0FVMdbQsKVXvwoJQ3OJdMKHFQYr8YhIM MIME-Version: 1.0 X-Received: by 10.70.44.132 with SMTP id e4mr10623987pdm.58.1423784782841; Thu, 12 Feb 2015 15:46:22 -0800 (PST) Sender: andy@fud.org.nz Received: by 10.70.129.238 with HTTP; Thu, 12 Feb 2015 15:46:22 -0800 (PST) In-Reply-To: <54DA303F.9020203@citrix.com> References: <54D88BD5.7050703@citrix.com> <54DA303F.9020203@citrix.com> Date: Fri, 13 Feb 2015 12:46:22 +1300 X-Google-Sender-Auth: _58bbvuH7DmI-bYZ1rWmdvLqPiI Message-ID: Subject: Re: xenstore memory issue From: Andrew Thompson To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Feb 2015 23:46:30 -0000 On 11 February 2015 at 05:22, Roger Pau Monn=C3=A9 w= rote: > Hello Andrew, > > El 10/02/15 a les 10.22, Andrew Thompson ha escrit: > > On 10 February 2015 at 16:25, Andrew Thompson > wrote: > > > >> On 9 February 2015 at 23:28, Roger Pau Monn=C3=A9 > wrote: > >> A bit of trial and error with dtrace has narrowed this down. I can cau= se > >> the leak by just opening /dev/xen/xenstore > > Thanks for doing all this legwork! IMHO the best solution is to switch > xenstore dev to use cdevpriv in order to store each client data. What we > are doing right now (storing client data in dev->si_dvr1) is plain > wrong. I've uploaded two patches (one for HEAD and one for stable/10) so > that you can try it also, please report back whether this fixes your > problem or not: > > https://people.freebsd.org/~royger/xenstore_fix/ > > I have tested this on 10.0 and it does fix the issue. After 36000 allocations there are just 6 active at 13K xenstore 6 13K - 36699 16,32,64,128,256,512 I looked in to why this only affected one of my many VMs as they run xe-update-guest-attrs and nova-agent which should have triggered the leak. They are all stuck on xsdread when reading from the xenstore so the update loop has stopped. UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 13250 13249 0 20 0 16552 1968 xsdread S - 5:57.60 /usr/local/bin/xenstore-read domid (xenstore) Is there an issue with the xs_dev_read and xs_queue_reply logic or would the Xen host be timing out? regards, Andrew