From owner-freebsd-stable@freebsd.org Wed Aug 1 06:48:15 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4CC751053377 for ; Wed, 1 Aug 2018 06:48:15 +0000 (UTC) (envelope-from return-to@edm.icartizan.email) Received: from dmd231.mail78.dmdelivery.com (dmd231.mail78.dmdelivery.com [61.147.78.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F12C83EFC for ; Wed, 1 Aug 2018 06:48:13 +0000 (UTC) (envelope-from return-to@edm.icartizan.email) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=edm.icartizan.email; s=dmddkim; t=1533106092; bh=QsVOe/tGLbQpzgR5HJYZxF5yiNOYxw43Appl5BVxOsA=; h=Date:Message-Id:To:From:Subject:Content-Type; b=Wwl0G7Ati1/Ql8vvSpERn8HhdNBV0uM0tk+I8sHPzJSNpq9cM9diaVnDkbFwr98Mm vB0T6d4qHX7YA3fgrQHk+fnXM5Crb9j6Slxhh9hGyNIw3O4V7JaqK9KHNKo3gAYKCG vUYZDEplsLzLo6BDxrC1izUUCbiuMFZpBgk1o820= Date: Wed, 01 Aug 2018 14:48:12 +0800 MIME-Version: 1.0 Message-Id: To: freebsd-stable@freebsd.org From: "Audio Interface" Reply-To: "Audio Interface" Subject: Digital Music Changer with Competitive Price is Here X-Mailer: DMD-MTA X-Hash: S7Y1.J9ra2hi_b.I1sjC2Njc4n_OrcX.DNvUlFy9zOTEopLMqsQ8vdTcxMyc.5kpthCh4sQ8AAA19 Feedback-ID: edm.icartizan.email:1:149:webpower List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Owner: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 06:48:15 -0000 Logo=0AIf you are unable to see the message below, click here (http://edm.i= cartizan.email/x/?S7Y1.J9ra2hi_T.H1uJ.ka2RhbGxuQUAA92)=0A=0Adigital music c= hanger=0A=0AIt's the second half of 2018. What are you waiting for?=0AH= igh-performance Bluetooth Digital CD Changer with competitive price is in s= tock now.=0AIs these the right products for your market?=0AIf "YES", please= contact us directly by replying to this Email or through the contact infor= mation at the bottom.=0A=0Afactory CD changer port=0A=0AShenzhen Cartizan T= echnology Co., Ltd.=0AAddress: 6F, Boshen Building, Changxing Road, Longgan= g District, Shenzhen, China=0ATel: 0086 755 27888592=C2=A0 =C2=A0 =C2=A0 = =C2=A0E-mail: salesa@icartizan.com=C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0Web: www= .icartizan.com=0AWhatsApp&Wechat: +86 15813713072=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0Skype: happysophia8686=0A=0AUnsubscribe (http://edm.icarti= zan.email/x/plugin/?pName=3Dunsubscribe&MIDRID=3DS7Y1.J9ra2hi_T.H1uJ.ka= 2RhbGxuQUAA92&Z=3D-2132840997) if you no longer wish to receive these e= mails.=0A From owner-freebsd-stable@freebsd.org Wed Aug 1 07:12:15 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A0091057B75 for ; Wed, 1 Aug 2018 07:12:15 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 068AC859D4 for ; Wed, 1 Aug 2018 07:12:15 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: by mailman.ysv.freebsd.org (Postfix) id BED461057B74; Wed, 1 Aug 2018 07:12:14 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C7CC1057B73 for ; Wed, 1 Aug 2018 07:12:14 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2D0B5859D3 for ; Wed, 1 Aug 2018 07:12:14 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from amavis-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 41gPd72H2Tz6MW for ; Wed, 1 Aug 2018 09:12:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1533107527; x=1535699528; bh=VPL 0lKpxdnrQ3A0rKlr8sUi5Ca9zghKrovc81tZ/cKE=; b=W+yqbKhC66WTu1vCg9n msGRtvQ6mNM9auNai6u2imrZL91TjQI/DirthUBNLo9/xm2AI6XCj6hkRI5PEjZM METAuuhaE7UUkoh3XaQ1pj2DncYyQE0gLmH52SMjwkGoXmWkpwIuX7KBrdog4zki J6+IanCFazyCw4QK+TfhUjOk= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026) with LMTP id yAp59YDtOI9I for ; Wed, 1 Aug 2018 09:12:07 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP id 41gPd360zMz6M1 for ; Wed, 1 Aug 2018 09:12:07 +0200 (CEST) Received: from nabiralnik.ijs.si (nabiralnik.ijs.si [IPv6:2001:1470:ff80::80:16]) by mildred.ijs.si (Postfix) with ESMTP id 41gPd33Wvbz12c for ; Wed, 1 Aug 2018 09:12:07 +0200 (CEST) Received: from sleepy.ijs.si (2001:1470:ff80:e001::76) by webmail.ijs.si with HTTP (HTTP/1.1 POST); Wed, 01 Aug 2018 09:12:07 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 01 Aug 2018 09:12:07 +0200 From: Mark Martinec To: stable@freebsd.org Subject: Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64 Organization: Jozef Stefan Institute In-Reply-To: <20180731220948.GA97237@raichu> References: <1a039af7758679ba1085934b4fb81b57@ijs.si> <3e56e4de076111c04c2595068ba71eec@ijs.si> <20180731220948.GA97237@raichu> Message-ID: <2ec91ebeaba54fda5e9437f868d4d590@ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.3.1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 07:12:15 -0000 > On Tue, Jul 31, 2018 at 11:54:29PM +0200, Mark Martinec wrote: >> I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE >> and the situation has not improved. Also turned off all services. >> ZFS is still leaking memory about 30 MB per hour, until the host >> runs out of memory and swap space and crashes, unless I reboot it >> first every four days. >> >> Any advise before I try to get rid of that faulted disk with a pool >> (or downgrade to 10.3, which was stable) ? 2018-08-01 00:09, Mark Johnston wrote: > If you're able to use dtrace, it would be useful to try tracking > allocations with the solaris tag: > > # dtrace -n 'dtmalloc::solaris:malloc {@allocs[stack(), args[3]] = > count()} dtmalloc::solaris:free {@frees[stack(), args[3]] = > count();}' > > Try letting that run for one minute, then kill it and paste the output. > Ideally the host will be as close to idle as possible while still > demonstrating the leak. Good and bad news: The suggested dtrace command bails out: # dtrace -n 'dtmalloc::solaris:malloc {@allocs[stack(), args[3]] = count()} dtmalloc::solaris:free {@frees[stack(), args[3]] = count();}' dtrace: description 'dtmalloc::solaris:malloc ' matched 2 probes Assertion failed: (buf->dtbd_timestamp >= first_timestamp), file /usr/src/cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c, line 3330. Abort trap But I did get one step further, localizing the culprit. I realized that the "solaris" malloc count goes up in sync with the 'telegraf' monitoring service polls, which also has a ZFS plugin which monitors the zfs pool and ARC. This plugin runs 'zpool list -Hp' periodically. So after stopping telegraf (and other remaining services), the 'vmstat -m' shows that InUse count for "solaris" goes up by 552 every time that I run "zpool list -Hp" : # (while true; do zpool list -Hp >/dev/null; vmstat -m | \ fgrep solaris; sleep 1; done) | awk '{print $2-a; a=$2}' 6664427 541 552 552 552 552 552 552 552 552 556 548 552 552 552 552 552 552 552 552 552 # zpool list -Hp floki 68719476736 37354102272 31365374464 - - 49% 54 1.00x ONLINE - stuff - - - - - - - - UNAVAIL - Mark