From owner-freebsd-xen@FreeBSD.ORG Tue Feb 10 03:25:33 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1553B2CE for ; Tue, 10 Feb 2015 03:25:33 +0000 (UTC) Received: from mail-pa0-f41.google.com (mail-pa0-f41.google.com [209.85.220.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC7FF7BB for ; Tue, 10 Feb 2015 03:25:32 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id kx10so13846917pab.0 for ; Mon, 09 Feb 2015 19:25:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=0tcEZbzYiX3s9ZPznD062ay6C13/GHCWqvDlPKQG0KQ=; b=a23TCTKYuVd0RPOtw7i7bSskwRiKVynYKZ6vS6f/eodZ9fnZ7+Ef6ZRvnqHT7uhQUz V7hbr8h4MDJ4qan3D5q0RVfXpKx0T4hVPwFYb+s7i8tiYCD80/0N1nvv4vlrtcgeXxpS 0n8LOPltcpZHT+/IQFBrLJR0hK6/SWNwvz7EDbRLA3Vc49Be0hR+Ch6LfhkOG4c+5+8w WbzQ8whH7YzB0HuNo2FaxDCSJ88echBwj4tCdkxmaWy6mZop6b4opok2/M/np4VqknWd +a6qxKpMv3jzhAx5oXIzdLNm/YzzrAw5N70YFmlpNdbuiqqvUmEH1xui1DASnBvJ6+/R t1Iw== X-Gm-Message-State: ALoCoQmMywXrX9J793QsWfYcC7RqnQfvcj5g76kCD9N9AfKugIlO4vI4xpzLPOiOW6RXI/fCIF4U MIME-Version: 1.0 X-Received: by 10.68.253.101 with SMTP id zz5mr33858182pbc.50.1423538726530; Mon, 09 Feb 2015 19:25:26 -0800 (PST) Sender: andy@fud.org.nz Received: by 10.70.129.238 with HTTP; Mon, 9 Feb 2015 19:25:26 -0800 (PST) In-Reply-To: <54D88BD5.7050703@citrix.com> References: <54D88BD5.7050703@citrix.com> Date: Tue, 10 Feb 2015 16:25:26 +1300 X-Google-Sender-Auth: xHBtmjT3NeqMyWZ4RRXSrF7CUfE Message-ID: Subject: Re: xenstore memory issue From: Andrew Thompson To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , freebsd-xen@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Feb 2015 03:25:33 -0000 On 9 February 2015 at 23:28, Roger Pau Monn=C3=A9 wr= ote: > Hello, > > El 09/02/15 a les 10.39, Andrew Thompson ha escrit: > > Hi, > > > > > > I have three VMs with Rackspace and one is behaving oddly with xenstore > > memory consumption. Here are the kernel versions and vmstat -m results. > > > > As you can see the third VM is using 100MB in xenstore memory and it seem= s > > to be climbing by 1-2MB per hour. Eventually all the processes go in to > > pfault state and it grinds to a halt. > > That's certainly weird, are you doing something different on this VM as > compared to the others? Did you hot-add a nic, disk or ballooned memory? > > Has the VM been saved/restored or migrated? > > Tracking down this kind of xenstore leaks can be difficult without > having a way to reproduce them. > > A bit of trial and error with dtrace has narrowed this down. I can cause the leak by just opening /dev/xen/xenstore int main() { open("/dev/xen/xenstore", O_RDWR, 0); } # vmstat -m | grep xenstore; ./open; vmstat -m | grep xenstore xenstore 8739 104797K - 56078 16,32,64,128,256,512 xenstore 8740 104809K - 56079 16,32,64,128,256,512 Using dtrace probes I can see that xs_dev_close is never called. # dtrace -n 'fbt::xs_dev_open:{} fbt::xs_dev_close: {} dtmalloc::xenstore: {}' # ./open CPU FUNCTION 0 -> xs_dev_open 0 | xenstore:malloc 0 <- xs_dev_open This is on 10.0-RELEASE-p12. I get the same result with `/usr/local/bin/xenstore-read domid` but the above c program is enough too. regards, Andrew