From owner-freebsd-stable@freebsd.org  Tue Feb 12 18:29:22 2019
Return-Path: <owner-freebsd-stable@freebsd.org>
Delivered-To: freebsd-stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8484714F0144
 for <freebsd-stable@mailman.ysv.freebsd.org>;
 Tue, 12 Feb 2019 18:29:22 +0000 (UTC) (envelope-from pi@opsec.eu)
Received: from home.opsec.eu (home.opsec.eu [IPv6:2001:14f8:200::1])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id E281D7333C
 for <freebsd-stable@freebsd.org>; Tue, 12 Feb 2019 18:29:21 +0000 (UTC)
 (envelope-from pi@opsec.eu)
Received: from pi by home.opsec.eu with local (Exim 4.92 (FreeBSD))
 (envelope-from <pi@opsec.eu>)
 id 1gtcnz-0000Aq-6h; Tue, 12 Feb 2019 19:29:15 +0100
Date: Tue, 12 Feb 2019 19:29:15 +0100
From: Kurt Jaeger <pi@opsec.eu>
To: Eugene Grosbein <eugen@grosbein.net>
Cc: Garrett Wollman <wollman@bimajority.org>, freebsd-stable@freebsd.org
Subject: Re: 11.2-STABLE kernel wired memory leak
Message-ID: <20190212182915.GM2748@home.opsec.eu>
References: <d8c7abc0-3ba1-40e4-22b1-1b30d28ced14@grosbein.net>
 <201902121757.x1CHve0h056876@hergotha.csail.mit.edu>
 <7125e053-5adf-929a-bde6-a64fceae2aaa@grosbein.net>
 <8a9361fd-4701-4e33-33f0-e4800af7637c@grosbein.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <8a9361fd-4701-4e33-33f0-e4800af7637c@grosbein.net>
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, 
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Feb 2019 18:29:22 -0000

Hi!

> > Use following command to see how much memory is wasted in your case:
> > 
> > vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
> 
> Oops, small correction:
> 
> vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head

On a 8 GB 11.2p8 box doing mostly routing:

  42.3047 abd_chunk
        40 zio_buf_131072
     31.75 zio_data_buf_131072
   19.8901 swblk
   12.9224 RADIX NODE
   11.7344 zio_buf_16384
   10.0664 zio_data_buf_12288
   9.84375 zio_data_buf_40960
     9.375 zio_data_buf_81920
   7.96875 zio_data_buf_98304

So, how do I understand this ? Please note that I use:

vfs.zfs.arc_max=1216348160

-- 
pi@opsec.eu            +49 171 3101372                    One year to go !