From owner-freebsd-ppc@freebsd.org Tue May 3 06:24:38 2016 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BEB0AB2BD71; Tue, 3 May 2016 06:24:38 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pa0-x244.google.com (mail-pa0-x244.google.com [IPv6:2607:f8b0:400e:c03::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 90E5E1E41; Tue, 3 May 2016 06:24:38 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pa0-x244.google.com with SMTP id yl2so678346pac.1; Mon, 02 May 2016 23:24:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=0pEk6l6lOKfFPx4ye2AdzlcGLEQd6UxrBrnMWYYVdmM=; b=D3z5dt4TA8fEqe/hDZ6qBywFMXGwrMfs9cMthuscRpT5iHPa36n8nNVnikVczBwsyv yb3ZLhe7CvZp0qNF/WKS4dYgb1hI8goNrpaRlmNHMnQWeB9KNRvMPCSnlUh1UiT88XZr cnyb8f2uxByxLDJ8fGS8oxuvKdvD/9f5zitQda8rlCYuiZP4raoo7oE3blOcbcZS2N/j bQw00Z1C7xzzTRi9tUf+H15uQHJUGxLFFC0jvs1nGiEREhIjQr+x3cEU7AWdX/qYxeOJ mIKVD+hbzaCerOayV9y0CdZDYED4DLx52CaeWp5HTMNVG6pLWW7425j7QxLEpXNCtRh2 szag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=0pEk6l6lOKfFPx4ye2AdzlcGLEQd6UxrBrnMWYYVdmM=; b=ZTksoRQBbVIIGPKkTty6D5XRYZ/u6CvVulIiN+WrueEideWp8aR6R7JpgVdk1uvUDm 5xTuqlhS/cYscqvAM5ImotLftxr7sEP4iRp1tPl90f8Sd33lWlxkXOUZOWAMkErZ+rKK Z2KTvGCcRFbsPaUcU2BCIcUdVZQSK5B8R4XPsRat0XeUDdAl+8Pa8vI0Dmt3wdHFuaST DcBnz/RZtoJ3sKvXAsw2yjyEfNZC52rIMMRcZrvxcAoHUPJsphLv8Dp0hv9S/ZcKxRDD hWgzTwAMgUOElRhNBcIuxVotmaZ4D2Mz2rvH+qttYrqFugqO5F8YMCFl4HKVPhD1fqH+ zqcQ== X-Gm-Message-State: AOPr4FU7OvijzRgvomq5FPTN+VgaNnJWYDuxMsOAKNFIeIh1GNGRW7ByeHd2NYuBOT2JCA== X-Received: by 10.66.90.136 with SMTP id bw8mr1013807pab.52.1462256678263; Mon, 02 May 2016 23:24:38 -0700 (PDT) Received: from raichu ([2604:4080:1102:0:ca60:ff:fe9d:3963]) by smtp.gmail.com with ESMTPSA id c25sm2531533pfd.67.2016.05.02.23.24.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 May 2016 23:24:37 -0700 (PDT) Sender: Mark Johnston Date: Mon, 2 May 2016 23:20:31 -0700 From: Mark Johnston To: Steve Wills Cc: FreeBSD PowerPC ML , freebsd-current@freebsd.org, scottl@FreeBSD.org, Warner Losh Subject: Re: wired memory leak at r298785 Message-ID: <20160503062031.GA2209@raichu> References: <572756DF.1010809@FreeBSD.org> <5727F71E.20101@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5727F71E.20101@FreeBSD.org> User-Agent: Mutt/1.6.0 (2016-04-01) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2016 06:24:38 -0000 On Mon, May 02, 2016 at 08:55:58PM -0400, Steve Wills wrote: > Hi, > > On 05/ 2/16 09:32 AM, Steve Wills wrote: > > Hi, > > > > Just did my monthly update and r298785 seems to be leaking wired memory > > rather rapidly. My system has 8gb of RAM and the amount of wired memory > > just goes up and up continuously. It takes about 12 hours before it > > exhausts all the RAM and sort of locks up (though shutdown still works). > > > > I also made one other change to the system at the same time as updating, > > which was to add another disk and configure it using ZFS. Perhaps this > > is a ZFS on PowerPC64 issue? My amd64 box running the same rev of > > CURRENT doesn't have the issue. > > > > I've rebooted the box and started repeatedly logging the output of > vmstat -m. It seems to show CAM CCB using a lot of memory and growing > rather rapidly. For example, here's a few lines of diff output: > > - CAM CCB 91418 182836K - 187149 2048 > + CAM CCB 447070 894140K - 900292 2048 > > from two samples that are 60 minutes apart. > > The box is isn't terribly busy, it's just running the monitoring daemons > running (snmpd, collectd), whatever web requests are hitting it (very > few if any), this logging process, and my shell, etc. This was causing problems on one of my amd64 systems, so it's not specific to powerpc64. It turns out to be due to r298004: the CCB allocated in cam_periph_devctl_notify() never gets freed. The patch below seems to fix it. It's possible to trace CCB allocations/frees using dtrace, which makes many of these sorts of problems trivial to find. Running # dtrace -n 'dtmalloc::CAM_CCB: {printf("%s", execname); stack();}' and examining the output showed that hald was frequently allocating CCBs at cam_periph_error+0x48f, but never freeing them. This corresponds to the allocation in cam_periph_devctl_notify(). diff --git a/sys/cam/cam_periph.c b/sys/cam/cam_periph.c index 85b2ff9..1f7be4f 100644 --- a/sys/cam/cam_periph.c +++ b/sys/cam/cam_periph.c @@ -1876,6 +1876,7 @@ cam_periph_devctl_notify(union ccb *ccb) if (cgd->ccb_h.status == CAM_REQ_CMP) sbuf_bcat(&sb, cgd->serial_num, cgd->serial_num_len); + xpt_free_ccb((union ccb *)cgd); } sbuf_printf(&sb, "\" "); sbuf_printf(&sb, "cam_status=\"0x%x\" ", ccb->ccb_h.status);