Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 01 Aug 2018 14:48:12 +0800
From:      "Audio Interface" <cartizan@edm.icartizan.email>
To:        freebsd-stable@freebsd.org
Subject:   Digital Music Changer with Competitive Price is Here
Message-ID:  <S7Y1.J9ra2hi_b.I1sjC2Njc4n_OrQUAA64@edm.icartizan.email>

next in thread | raw e-mail | index | archive | help
Logo=0AIf you are unable to see the message below, click here (http://edm.i=
cartizan.email/x/?S7Y1.J9ra2hi_T.H1uJ.ka2RhbGxuQUAA92)=0A=0Adigital music c=
hanger=0A=0AIt&#39;s the second half of 2018. What are you waiting for?=0AH=
igh-performance Bluetooth Digital CD Changer with competitive price is in s=
tock now.=0AIs these the right products for your market?=0AIf "YES", please=
 contact us directly by replying to this Email or through the contact infor=
mation at the bottom.=0A=0Afactory CD changer port=0A=0AShenzhen Cartizan T=
echnology Co., Ltd.=0AAddress: 6F, Boshen Building, Changxing Road, Longgan=
g District, Shenzhen, China=0ATel: 0086 755 27888592=C2=A0 =C2=A0 =C2=A0 =
=C2=A0E-mail: salesa@icartizan.com=C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0Web: www=
.icartizan.com=0AWhatsApp&Wechat: +86 15813713072=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0Skype: happysophia8686=0A=0AUnsubscribe (http://edm.icarti=
zan.email/x/plugin/?pName=3Dunsubscribe&amp;MIDRID=3DS7Y1.J9ra2hi_T.H1uJ.ka=
2RhbGxuQUAA92&amp;Z=3D-2132840997) if you no longer wish to receive these e=
mails.=0A
From owner-freebsd-stable@freebsd.org  Wed Aug  1 07:12:15 2018
Return-Path: <owner-freebsd-stable@freebsd.org>
Delivered-To: freebsd-stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A0091057B75
 for <freebsd-stable@mailman.ysv.freebsd.org>;
 Wed,  1 Aug 2018 07:12:15 +0000 (UTC)
 (envelope-from Mark.Martinec+freebsd@ijs.si)
Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org
 [IPv6:2001:1900:2254:206a::50:5])
 by mx1.freebsd.org (Postfix) with ESMTP id 068AC859D4
 for <freebsd-stable@freebsd.org>; Wed,  1 Aug 2018 07:12:15 +0000 (UTC)
 (envelope-from Mark.Martinec+freebsd@ijs.si)
Received: by mailman.ysv.freebsd.org (Postfix)
 id BED461057B74; Wed,  1 Aug 2018 07:12:14 +0000 (UTC)
Delivered-To: stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C7CC1057B73
 for <stable@mailman.ysv.freebsd.org>; Wed,  1 Aug 2018 07:12:14 +0000 (UTC)
 (envelope-from Mark.Martinec+freebsd@ijs.si)
Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 2D0B5859D3
 for <stable@freebsd.org>; Wed,  1 Aug 2018 07:12:14 +0000 (UTC)
 (envelope-from Mark.Martinec+freebsd@ijs.si)
Received: from amavis-ori.ijs.si (localhost [IPv6:::1])
 by mail.ijs.si (Postfix) with ESMTP id 41gPd72H2Tz6MW
 for <stable@freebsd.org>; Wed,  1 Aug 2018 09:12:11 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h=
 user-agent:message-id:references:in-reply-to:organization
 :subject:subject:from:from:date:date:content-transfer-encoding
 :content-type:content-type:mime-version:received:received
 :received:received; s=jakla4; t=1533107527; x=1535699528; bh=VPL
 0lKpxdnrQ3A0rKlr8sUi5Ca9zghKrovc81tZ/cKE=; b=W+yqbKhC66WTu1vCg9n
 msGRtvQ6mNM9auNai6u2imrZL91TjQI/DirthUBNLo9/xm2AI6XCj6hkRI5PEjZM
 METAuuhaE7UUkoh3XaQ1pj2DncYyQE0gLmH52SMjwkGoXmWkpwIuX7KBrdog4zki
 J6+IanCFazyCw4QK+TfhUjOk=
X-Virus-Scanned: amavisd-new at ijs.si
Received: from mail.ijs.si ([IPv6:::1])
 by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026)
 with LMTP id yAp59YDtOI9I for <stable@freebsd.org>;
 Wed,  1 Aug 2018 09:12:07 +0200 (CEST)
Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1])
 by mail.ijs.si (Postfix) with ESMTP id 41gPd360zMz6M1
 for <stable@freebsd.org>; Wed,  1 Aug 2018 09:12:07 +0200 (CEST)
Received: from nabiralnik.ijs.si (nabiralnik.ijs.si
 [IPv6:2001:1470:ff80::80:16])
 by mildred.ijs.si (Postfix) with ESMTP id 41gPd33Wvbz12c
 for <stable@freebsd.org>; Wed,  1 Aug 2018 09:12:07 +0200 (CEST)
Received: from sleepy.ijs.si (2001:1470:ff80:e001::76) by webmail.ijs.si
 with HTTP (HTTP/1.1 POST); Wed, 01 Aug 2018 09:12:07 +0200
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Wed, 01 Aug 2018 09:12:07 +0200
From: Mark Martinec <Mark.Martinec+freebsd@ijs.si>
To: stable@freebsd.org
Subject: Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R
 amd64
Organization: Jozef Stefan Institute
In-Reply-To: <20180731220948.GA97237@raichu>
References: <1a039af7758679ba1085934b4fb81b57@ijs.si>
 <3e56e4de076111c04c2595068ba71eec@ijs.si> <20180731220948.GA97237@raichu>
Message-ID: <2ec91ebeaba54fda5e9437f868d4d590@ijs.si>
X-Sender: Mark.Martinec+freebsd@ijs.si
User-Agent: Roundcube Webmail/1.3.1 
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.27
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, 
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>;
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Aug 2018 07:12:15 -0000

> On Tue, Jul 31, 2018 at 11:54:29PM +0200, Mark Martinec wrote:
>> I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE
>> and the situation has not improved. Also turned off all services.
>> ZFS is still leaking memory about 30 MB per hour, until the host
>> runs out of memory and swap space and crashes, unless I reboot it
>> first every four days.
>> 
>> Any advise before I try to get rid of that faulted disk with a pool
>> (or downgrade to 10.3, which was stable) ?

2018-08-01 00:09, Mark Johnston wrote:
> If you're able to use dtrace, it would be useful to try tracking
> allocations with the solaris tag:
> 
> # dtrace -n 'dtmalloc::solaris:malloc {@allocs[stack(), args[3]] =
>   count()} dtmalloc::solaris:free {@frees[stack(), args[3]] = 
> count();}'
> 
> Try letting that run for one minute, then kill it and paste the output.
> Ideally the host will be as close to idle as possible while still
> demonstrating the leak.

Good and bad news:

The suggested dtrace command bails out:

# dtrace -n 'dtmalloc::solaris:malloc {@allocs[stack(), args[3]] = 
count()} dtmalloc::solaris:free {@frees[stack(), args[3]] = count();}'
dtrace: description 'dtmalloc::solaris:malloc ' matched 2 probes
Assertion failed: (buf->dtbd_timestamp >= first_timestamp), file 
/usr/src/cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c, 
line 3330.
Abort trap

But I did get one step further, localizing the culprit.

I realized that the "solaris" malloc count goes up in sync with
the 'telegraf' monitoring service polls, which also has a ZFS plugin
which monitors the zfs pool and ARC. This plugin runs 'zpool list -Hp'
periodically.

So after stopping telegraf (and other remaining services),
the 'vmstat -m' shows that InUse count for "solaris" goes up by 552
every time that I run "zpool list -Hp" :

# (while true; do zpool list -Hp >/dev/null; vmstat -m | \
     fgrep solaris; sleep 1; done) | awk '{print $2-a; a=$2}'
6664427
541
552
552
552
552
552
552
552
552
556
548
552
552
552
552
552
552
552
552
552

# zpool list -Hp
floki   68719476736     37354102272     31365374464     -       -       
49%     54      1.00x   ONLINE  -
stuff   -       -       -       -       -       -       -       -       
UNAVAIL -


   Mark



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?S7Y1.J9ra2hi_b.I1sjC2Njc4n_OrQUAA64>