From owner-svn-src-head@freebsd.org Tue Sep 3 22:01:13 2019 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 489FED9ACB; Tue, 3 Sep 2019 22:01:13 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46NLWC47Jxz4RJM; Tue, 3 Sep 2019 22:01:11 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-io1-xd31.google.com with SMTP id b10so39646975ioj.2; Tue, 03 Sep 2019 15:01:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=m2308RF/w9o0xbkVN8BZhIyFHrJra43Z9VyAMOQ1vhw=; b=H8zNQs95XwSk3GOMtAp7rE2+7pOiorLxoUUNK9DyKV0MXv7Iux15G/klMvVRed5Wff KzPLVJU7lB1TDKXxztqLMDLMQlVo+6jWeJ0kqI37OlREKuInzu3Om66Dk0QdOfL0Yreb 0PuxbJnzGAq/gx68xq5gIWC0cWyMgnIp8zEA+/BZwSPUMEPEHV/xn59OtlEXnNpGOJAT aTLLxk3UXT/Kwf8lXjJi5p7fEoYjgSxv3u9ST34vNN0sVp/O99f8sjw0aCStpUdEYnhO fXF17oZoxvDWscP10jemrRlv7o7TKmsuxDwz/8SQaz1kIWbE7sNRL29nKQNh8uQCtH0H c2xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=m2308RF/w9o0xbkVN8BZhIyFHrJra43Z9VyAMOQ1vhw=; b=SFOBkfB2MxF4+ndZoFEFPcGu539rXr+bPaOlOQv3ttFejqVHeCtNxFY6kr8aa9ii1c xt8d7xyNoLKHLWHhx/hLq49PW3nqmHraC8oRYJfsynzHg7hJpdv7qDzStoYUyJPhMOe/ /GvcAKw5Zq/AZOqLMJKHQkqlkGehpCY2RM2GL3nCPG2Scd6sN9WHdgBMVnC0aXI2av0z oN+qfD6MtQOG9F/uMMKsgBUC901FTsOom1baKcnT2wx72fAOe7inIGqeBadGYD+9lgYh YKk5qsqSPdZ8I0cdaJjlRCICZDbPGfRD3IwOz3VUPKg3ShuHhKEpM3U20D6x8h10+6Tp eeJw== X-Gm-Message-State: APjAAAVXaR9mh4aCgbP+V2u3OfqHuiFgnDEw1jfXYvvGL11kzzJXbfP6 R6TxxMqovbs13FGnWhDRliq0rE9P X-Google-Smtp-Source: APXvYqzInxMPnPmcJwIuZLLt6cyp6U6J2MEpP0ohJCpKgjhmioXBClhGhnipTaBcN9kLfFCh0Qo2vg== X-Received: by 2002:a6b:6a07:: with SMTP id x7mr13418219iog.168.1567548070026; Tue, 03 Sep 2019 15:01:10 -0700 (PDT) Received: from raichu (toroon0560w-lp140-03-184-148-66-213.dsl.bell.ca. [184.148.66.213]) by smtp.gmail.com with ESMTPSA id p9sm17245422ios.1.2019.09.03.15.01.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Sep 2019 15:01:08 -0700 (PDT) Sender: Mark Johnston Date: Tue, 3 Sep 2019 18:01:06 -0400 From: Mark Johnston To: Slawa Olhovchenkov , Andriy Gapon Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r351673 - in head: lib/libmemstat share/man/man9 sys/cddl/compat/opensolaris/kern sys/kern sys/vm Message-ID: <20190903220106.GB26733@raichu> References: <201909012222.x81MMh0F022462@repo.freebsd.org> <79c74018-1329-ee69-3480-e2f99821fa93@FreeBSD.org> <20190903161427.GA38096@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190903161427.GA38096@zxy.spb.ru> User-Agent: Mutt/1.12.1 (2019-06-15) X-Rspamd-Queue-Id: 46NLWC47Jxz4RJM X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=H8zNQs95; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::d31 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-4.76 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; RCVD_IN_DNSWL_NONE(0.00)[1.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.08)[ip: (-5.29), ipnet: 2607:f8b0::/32(-2.77), asn: 15169(-2.28), country: US(-0.05)]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[213.66.148.184.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Sep 2019 22:01:13 -0000 On Tue, Sep 03, 2019 at 07:14:27PM +0300, Slawa Olhovchenkov wrote: > On Tue, Sep 03, 2019 at 10:02:59AM +0300, Andriy Gapon wrote: > > > On 02/09/2019 01:22, Mark Johnston wrote: > > > Author: markj > > > Date: Sun Sep 1 22:22:43 2019 > > > New Revision: 351673 > > > URL: https://svnweb.freebsd.org/changeset/base/351673 > > > > > > Log: > > > Extend uma_reclaim() to permit different reclamation targets. > > > > > > The page daemon periodically invokes uma_reclaim() to reclaim cached > > > items from each zone when the system is under memory pressure. This > > > is important since the size of these caches is unbounded by default. > > > However it also results in bursts of high latency when allocating from > > > heavily used zones as threads miss in the per-CPU caches and must > > > access the keg in order to allocate new items. > > > > > > With r340405 we maintain an estimate of each zone's usage of its > > > (per-NUMA domain) cache of full buckets. Start making use of this > > > estimate to avoid reclaiming the entire cache when under memory > > > pressure. In particular, introduce TRIM, DRAIN and DRAIN_CPU > > > verbs for uma_reclaim() and uma_zone_reclaim(). When trimming, only > > > items in excess of the estimate are reclaimed. Draining a zone > > > reclaims all of the cached full buckets (the previous behaviour of > > > uma_reclaim()), and may further drain the per-CPU caches in extreme > > > cases. > > > > > > Now, when under memory pressure, the page daemon will trim zones > > > rather than draining them. As a result, heavily used zones do not incur > > > bursts of bucket cache misses following reclamation, but large, unused > > > caches will be reclaimed as before. > > > > Mark, > > > > have you considered running UMA_RECLAIM_TRIM periodically, even without a memory > > pressure? > > I think that with such a periodic trimming there will be less need to invoke > > vm_lowmem(). Slawa and I talked about this in the past. His complaint is that a large cache can take a significant amount of time to trim, and it manifests as a spike of CPU usage and contention on the zone lock. In particular, keg_drain() iterates over the list of free slabs with the keg lock held, and if the many items were freed to the keg while trimming/draining, the list can be quite long. This can have effects outside the zone, for example if we are reclaiming items from zones used by other UMA zones, like the bucket or slab zones. Reclaiming cached items when there is no demand for free pages seems wrong to me. We historically had similar problems with the page daemon, which last year was changed to perform smaller reclamations at a greater frequency. I suspect a better approach for UMA would be to similarly increase reclaim frequency and reduce the number of items freed in one go. > > Also, I think that we would be able to retire (or re-purpose) lowmem_period. > > E.g., the trimming would be done every lowmem_period, but vm_lowmem() would not > > be throttled. Some of the vm_lowmem eventhandlers probably shouldn't be called each time the page daemon scans the inactive queue (every 0.1s under memory pressure). ufsdirhash_lowmem and mb_reclaim in particular don't seem like they need to be invoked very frequently. We could easily define multiple eventhandlers to differentiate between these cases, though. > > One example of the throttling of vm_lowmem being bad is its interaction with the > > ZFS ARC. When there is a spike in memory usage we want the ARC to adapt as > > quickly as possible. But at present the lowmem_period logic interferes with that. > > Some time ago, I sent Mark a patch that implements this logic, > specialy for ARC and mbuf cooperate. > > Mostly problem I am see at this > work -- very slowly vm_page_free(). May be currenly this is more > speedy... How did you determine this? You are on stable/12 I believe, so r350374 might help if you do not already have it. I guess the vm_page_free() calls are coming from the UMA trimmer?