From owner-freebsd-stable@freebsd.org Tue Feb 12 19:39:12 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 587DA14CE347 for ; Tue, 12 Feb 2019 19:39:12 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8E8D4774CD for ; Tue, 12 Feb 2019 19:39:11 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id x1CJd0Mq073353 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 12 Feb 2019 20:39:05 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: mike@sentex.net Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x1CJd0ou053404 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 13 Feb 2019 02:39:00 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: 11.2-STABLE kernel wired memory leak To: Mike Tancsa References: <20190212163446.GA29847@raichu> <4ab1331f-80e3-b856-b402-9985e73618bc@grosbein.net> <672edbe6-ac42-f21e-ad8a-7a5f5d4c4e3c@sentex.net> Cc: FreeBSD stable From: Eugene Grosbein Message-ID: Date: Wed, 13 Feb 2019 02:38:53 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <672edbe6-ac42-f21e-ad8a-7a5f5d4c4e3c@sentex.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 8E8D4774CD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.94 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MX_INVALID(0.50)[cached]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; IP_SCORE(-1.37)[ip: (-2.20), ipnet: 2a01:4f8::/29(-2.39), asn: 24940(-2.23), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Feb 2019 19:39:12 -0000 13.02.2019 2:18, Mike Tancsa wrote: > On an nfs server, serving a few large files, my 32G box is showing > > vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", > $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head > 11014.3 abd_chunk > 2090.5 zio_data_buf_131072 > 1142.67 mbuf_jumbo_page > 1134.25 zio_buf_131072 > 355.28 mbuf_jumbo_9k > 233.42 zio_cache > 163.099 arc_buf_hdr_t_full > 130.738 128 > 97.2812 zio_buf_16384 > 96.5099 UMA Slabs > > > CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > Mem: 1348K Active, 98M Inact, 3316K Laundry, 30G Wired, 1022M Free > ARC: 11G Total, 7025M MFU, 3580M MRU, 11M Anon, 78M Header, 681M Other > 9328M Compressed, 28G Uncompressed, 3.05:1 Ratio > Swap: 64G Total, 13M Used, 64G Free > > > Right now its OK, but prior to limiting ARC, I had an issue with memory > and the disk thrashing due to swapping > > pid 643 (devd), uid 0, was killed: out of swap space mbuf* numbers represent memory being wasted IMHO. In contrast to ZFS memory that can contain useful cached data, contents of freed mbufs cannot be re-used, right? Some amount of "free" mbufs in the zone's just fine to serve bursts of traffic without need to grow the zone but your numbers are way too big IMHO. Do you have memory pressure here? Growth of swap usage?