From owner-freebsd-stable@freebsd.org Tue Feb 12 18:18:11 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C584114EFA14 for ; Tue, 12 Feb 2019 18:18:10 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x135.google.com (mail-it1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9718072779; Tue, 12 Feb 2019 18:18:09 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x135.google.com with SMTP id i2so9934068ite.5; Tue, 12 Feb 2019 10:18:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=wlVgQwIv3PyPELFXwQWEFnRYPugM4Vk9H0YCeCJlvus=; b=jDXE++00hovMQwhhr1Y3DL46TQ++8KA+m1IJf3Gily/IzoycpzL5aD9Dt3GmFYG1Ep Djry7MWdKvU4UXdB8t+4DAJiBkikE7s/QU0L5PGRJsoN53Wf/vq2R27gdYvmydY9ed+r QkmaM9o4XKhcsgw9XngFDM3UpXs2Pk63o0Gu77bsGWjlHNTzoBnbWytnAspBIJACfQCn sJwyTLs+yG33M6CZGN2pPvPm4PiQ+PL+drzAJ6Aei+vr7XU5pueJF4bgX7oOpktOARVT u6A9YuEtwiDv6X2ab5Z8G1OeT9s5tKE1+j4yvE7ZJyV4yXj3jyOit9ECBg0OovxxVhSZ XYGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=wlVgQwIv3PyPELFXwQWEFnRYPugM4Vk9H0YCeCJlvus=; b=j2R4H2IqMSVD7gSDi9/hx9d9sXmYcqI4mE/pQstWG82zblWRwq+veznbkV3QFazRJI +TbPxTBp/fmB/IpSNr1P6goLSzeuE3+45ghHnE7h8DVYJSCYtL0rj6DPzIhQII3nil8I WCzJ0rErzbJdJcYzRYjqKUThJY3QJLG6AB5s6KWsBRstLRzj4Q2vVRVODwju1cnt2DD9 INov+tLzUfOksJfu0fiDJk/YOuaST12psMyiTyAX34dDUyRIejMbvPmdwO2+T6dsw6JZ RY1xTknzlxGOoEHPAAnK7+PTg5kMSa6EwuhNFrHWHpESFBmdNzgIbzMPdjHb41NYnOmx N6Dg== X-Gm-Message-State: AHQUAub+8ET2Nhz1CsE1STXQZSj44RrVoAZgdb1wTpYR4nLmK5bvDs47 h0k9aw0lyrsS54hDHP43M1y5+8pZ X-Google-Smtp-Source: AHgI3IZDYXfYUnpJ9/II1IdZ5ckFwsbw+J1t3xwhR94AIX3NEColhFpFEe8e7ps6z/WbqGUX3cAPQA== X-Received: by 2002:a05:660c:30e:: with SMTP id u14mr100677itj.13.1549995488749; Tue, 12 Feb 2019 10:18:08 -0800 (PST) Received: from raichu (toroon0560w-lp140-01-69-159-36-102.dsl.bell.ca. [69.159.36.102]) by smtp.gmail.com with ESMTPSA id h8sm8469ith.2.2019.02.12.10.18.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Feb 2019 10:18:08 -0800 (PST) Sender: Mark Johnston Date: Tue, 12 Feb 2019 13:18:05 -0500 From: Mark Johnston To: Eugene Grosbein Cc: FreeBSD stable , kib@freebsd.org Subject: Re: 11.2-STABLE kernel wired memory leak Message-ID: <20190212181805.GB29847@raichu> References: <20190212163446.GA29847@raichu> <4ab1331f-80e3-b856-b402-9985e73618bc@grosbein.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ab1331f-80e3-b856-b402-9985e73618bc@grosbein.net> User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: 9718072779 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=jDXE++00; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::135 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-5.28 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[5.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.94)[-0.944,0]; IP_SCORE(-2.62)[ip: (-8.56), ipnet: 2607:f8b0::/32(-2.52), asn: 15169(-1.97), country: US(-0.07)]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Feb 2019 18:18:11 -0000 On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote: > 12.02.2019 23:34, Mark Johnston wrote: > > > I suspect that the "leaked" memory is simply being used to cache UMA > > items. Note that the values in the FREE column of vmstat -z output are > > quite large. The cached items are reclaimed only when the page daemon > > wakes up to reclaim memory; if there are no memory shortages, large > > amounts of memory may accumulate in UMA caches. In this case, the sum > > of the product of columns 2 and 5 gives a total of roughly 4GB cached. > > > >> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt > >> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt > > It seems page daemon is broken somehow as it did not reclaim several gigs of wired memory > despite of long period of vm thrashing: Depending on the system's workload, it is possible for the caches to grow quite quickly after a reclaim. If you are able to run kgdb on the kernel, you can find the time of the last reclaim by comparing the values of lowmem_uptime and time_uptime. > $ sed 's/:/,/' vmstat-z.txt | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head > 1892 abd_chunk > 454.629 dnode_t > 351.35 zio_buf_512 > 228.391 zio_buf_16384 > 173.968 dmu_buf_impl_t > 130.25 zio_data_buf_131072 > 93.6887 VNODE > 81.6978 arc_buf_hdr_t_full > 74.9368 256 > 57.4102 4096 >