From owner-freebsd-stable@freebsd.org Tue Feb 12 18:43:00 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81AD114F0C66 for ; Tue, 12 Feb 2019 18:43:00 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x141.google.com (mail-it1-x141.google.com [IPv6:2607:f8b0:4864:20::141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9685F747A4 for ; Tue, 12 Feb 2019 18:42:59 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x141.google.com with SMTP id a6so10173830itl.4 for ; Tue, 12 Feb 2019 10:42:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=f+AL2DQeyA2CwUFSwRtJocZPe80+6XymJjV2DEPQWJI=; b=UK6yb9s2N+ZNIttH1vc9HtxDWVb4wUlVGnRMDVJX0WvcP8u4KGyEqqOeCwktHxcMHw tX6ObyKNetsbj0+xXq5DolXcjZPZcmhDRWTFhtavQuv80to3t4siqDfLbVwGbp1GfJJt HaeXfD9keOkN5OyAvSlAeiE9w9jsxI3M1kIe33hQQQDlzrGCZxATsjzhQ4e+8Ga1MScr Fj3IsbZAnGEFhiPa+CAq1Qqe6caLcHyeDUfxc1Mu/Ut63ryYuFPlaWnYmnEp13T5nppM ruGcc4NtiA516JO9FSmipZAU/PwgJLBspXc2WcVurGiyKFQMc4bAA7FAiGYOE71FLD+W 17Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=f+AL2DQeyA2CwUFSwRtJocZPe80+6XymJjV2DEPQWJI=; b=Qt9WK6e1hRyKA98MvSXhUPY6ijILggb5FiCSacu558xBBf50dPkc5frfjsNoXtlcUS bT29HT3u7bxSBkmyfsS2AhIod+LcCvHOGXIWmZR9snLoRvfdiAnWGlb7zpkVWeqTUhE1 pC65ocnHme4BsYBeTwrzFI8453Tcl1A3eU+yEGKTYJR3+tADxI6isQXz07SBneurn9uf af99DXQ2c8fORP1D69Dt1wTtQizYg1o3buOR8faQtfTMCoNB+hPMBEiC3O98Ge+fxx7x GF79oAtPjfGYHeAcvSxI4a6CTalxWB3Ks05bOCY+xgMlgu5a2fnjr6znrJLW7m8/mqN+ iSlQ== X-Gm-Message-State: AHQUAuaHCYQibCDi7B5AdBTN9qAQUG5+Ugxsegq73PrgODZmUYrPipAP TuBnhxBr5YWJsggmbiK9khpdBevC X-Google-Smtp-Source: AHgI3IaZeNmZ+jCrPFp70oKpTQ2TWJEfYlQNplzhfz0MGgSX/mMEsn9si9WfnFVls2lx/7Y0ZXNVzw== X-Received: by 2002:a02:5184:: with SMTP id s126mr2608420jaa.12.1549996978959; Tue, 12 Feb 2019 10:42:58 -0800 (PST) Received: from raichu (toroon0560w-lp140-01-69-159-36-102.dsl.bell.ca. [69.159.36.102]) by smtp.gmail.com with ESMTPSA id h123sm1723587itb.36.2019.02.12.10.42.57 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Feb 2019 10:42:58 -0800 (PST) Sender: Mark Johnston Date: Tue, 12 Feb 2019 13:42:56 -0500 From: Mark Johnston To: Eugene Grosbein Cc: FreeBSD stable Subject: Re: 11.2-STABLE kernel wired memory leak Message-ID: <20190212184256.GC29847@raichu> References: <20190212163446.GA29847@raichu> <4ab1331f-80e3-b856-b402-9985e73618bc@grosbein.net> <20190212181805.GB29847@raichu> <973d9dc6-8dd4-8cdf-b279-f5c9483da884@grosbein.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <973d9dc6-8dd4-8cdf-b279-f5c9483da884@grosbein.net> User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: 9685F747A4 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UK6yb9s2; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::141 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.38 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.89)[-0.885,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.67)[-0.671,0]; NEURAL_HAM_LONG(-0.99)[-0.992,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.4.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.12)[ip: (3.93), ipnet: 2607:f8b0::/32(-2.51), asn: 15169(-1.97), country: US(-0.07)]; MID_RHS_NOT_FQDN(0.50)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Feb 2019 18:43:00 -0000 On Wed, Feb 13, 2019 at 01:40:06AM +0700, Eugene Grosbein wrote: > 13.02.2019 1:18, Mark Johnston wrote: > > > On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote: > >> 12.02.2019 23:34, Mark Johnston wrote: > >> > >>> I suspect that the "leaked" memory is simply being used to cache UMA > >>> items. Note that the values in the FREE column of vmstat -z output are > >>> quite large. The cached items are reclaimed only when the page daemon > >>> wakes up to reclaim memory; if there are no memory shortages, large > >>> amounts of memory may accumulate in UMA caches. In this case, the sum > >>> of the product of columns 2 and 5 gives a total of roughly 4GB cached. > >>> > >>>> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt > >>>> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt > >> > >> It seems page daemon is broken somehow as it did not reclaim several gigs of wired memory > >> despite of long period of vm thrashing: > > > > Depending on the system's workload, it is possible for the caches to > > grow quite quickly after a reclaim. If you are able to run kgdb on the > > kernel, you can find the time of the last reclaim by comparing the > > values of lowmem_uptime and time_uptime. > > Yes, I have debugger compiled into running kernel and have console access. > What commands should I use? I meant kgdb(1). If you can run that, try: (kgdb) p time_uptime (kgdb) p lowmem_uptime If you are willing to drop the system into DDB, do so and run db> x time_uptime db> x lowmem_uptime